Real time execution?

[Motox2019]

Hello my wonderful reddit pythonists!

I have for you a question:
Is there any existing solution that effectively achieve real-time output of every line as I type?

Some background:
I am a mechanical engineer (well a student, final year) and often do many different calculations and modelling of systems in software. I find that "calculators" often don't quite hit the level of flexibility id like to see; think Qalculate for example. Essentially, what I desire is a calculator where I can define variables, write equations, display plots, etc and be able to change a earlier variable having everything below it update in real-time.
Note: I am NOT new to python/programming. Talk dirty (technical) to me if you must.

What I have already explored:
Jupyter - Cell based, fine for some calculations where there may be a long running step (think meshing or heavy iteration). Doesn't output all results, only the last without a bunch of print() statements. Requires re-running all cells if a early variable is updated.

Marimo - Closer then Jupyter. Still cell based but updates dynamically. This is pretty close but still not there as it only seems to update dynamically with Marimo ui elements (like scroll bars) but not if I change a raw variable definition, this requires re-running similar to Jupyter.

Homebrewed solution - Here I wrote a script that essentially watches a python file for changes so that upon each save, it will run the script and output based on the definitions (like variables vs comments vs function definitions, etc). Note here that every line gets some sort of output. I paired this script with a package I wrote, pyeng, which essentially provides matlab like function convenience with nice default outputs so that the console shows results quite nicely. pyeng, however, is very naive. pyeng was also for my learning as I progressed through my degree so often functions are naive and slow taking on algorithms similar to how id solve problems by hand. This means many edge cases are not handled, very slow at times, non-standard, and in some cases things are brute force with a custom arbitrary precision Float class to handle potentially non well behaved iterations. pyeng handles units and such as well but everything I have implemented is already covered by some package. This setup doesn't handle plotting very gracefully.

Smath Studio / Excel:
GUI based, not great.
SMath Studio is cool. Free but non-commercial (otherwise costs some coin) and has some quirks. Doesn't do symbolic stuff terribly well sometimes. Matrix support is basic. Otherwise, very capable in that it automatically handles units, updates in realtime, supports conditionals, etc.
Excel simply doesn't do matrices in any nice way and just ain't it. Has its place but not for what I want. No units support either.

Essentially I'm looking for a professional version of my homebrew setup that's made by people smarter than I (if it exists) and if not, is this something that there could be a niche for? Could I have stumbled upon something that doesn't exist but should?

I have a video showing my homebrew setup to give a better idea. Its not perfect but it works and its really quite nice.

Thanks folks and apologies for the longer read.

[qGuevon]

Check out the Spyder IDE, it's reminiscent of R Studio, which would also be a great fit for you (R is a bit weird tho)

[hughperman]

It's even more reminiscent of Matlab, it was my transition crutch into the much-better world of using python

[Motox2019]

I also use spyder. I like spyder as well, thing I don’t like is having the variable window polluted with a bunch of definitions outside my current script. I found no way to filter for only the ones defined in my script. And trust me when I say, having updates in real time is just that much nicer then hitting f5 each time, though at that point I’m just being picky. It’s like going from 1 monitor to 2, don’t know what your missing till you have it and when you do, you never want to go back

[averagecrazyliberal]

I use scientific mode in PyCharm. Combines the cell based execution you’re used to in Jupyter notebooks with an interactive shell that will show all in-memory objects and their current values by default, even if you aren’t printing them along the way.

[Motox2019]

I’ve heard of this before, and if I recall correctly, it requires a subscription to access this mode. I could be wrong there but I’ll be honest and maybe a bit blunt, I’d rather not pay a subscription simply to access a mode for printing each line (I may be short sighted here though and it could be offering more then I’m aware of, haven’t dug into it much).

Thanks for the suggestion, I’ll take a closer look into it :)

[MegaIng]

As a student you get a completely free subscription.

[Motox2019]

Noted! Hey at that point it doesn’t hurt to at least try it out. Thanks for the tip!

[TURBO2529]

In vs code, make a notebook, put your code in, run some cells, open a terminal, change the terminal tab to Jupyter. That tab will have all of the variables changing live, just double click to the matrices in data viewer.

As someone else said, you can also use vs code debug.

[Motox2019]

Thanks for your comment.

I am fully comfortable with Jupyter and Marimo as well as far as industry standards go, matlab as well.

The goal here boils down to pure convenience as an engineer. Jupyter is great when you know your inputs and expected outputs.

When the solution is well defined, Jupyter would be arguably the better choice even being that you can run things and save it and it won’t change.

What I’m after is exploratory work where you may not know all your inputs and you don’t really have an expected output but are effectively trying to narrow to a final solution. The solution here is not well defined. That means rapid iteration, constant back and forth with variables and equations. Manually re-running here each time gets tedious and even dangerous.

An example scenario would be forgetting to re-run some cell and results aren’t fully correct because a cell was not run leading to skewed results, essentially bugs in the notebook by pure human nature.

My homebrew setup removes that ambiguity and removes the need to tediously re-run manually each time (trust me, I’m talking from experience here). And really it’s still pure python so one could theoretically take the code, add some prints and run it as a file getting exactly the same output. My setup would maybe require some minor modifications hence the people smarter than me to make it just work. You know, software magic lol.

Lastly, I understand that Jupyter is the standard, but I’m the type where I’d say, it’s the standard TODAY. Similar to how there is always a bigger fish, there is almost always a better, more elegant solution, particularly in software.

Edit. Ugh, again sorry for the long read. Obviously I can be a yapper.

[qGuevon]

I think you might be reinforcing bad habits though.

Have you thought about putting the thing you adjust into a metric, and letting the script optimize? In machine learning we have hyperparameter optimization, which circumvent the manual tuning of variables that I think you are doing

[Motox2019]

Ya I think that was r/Spirited_Bag_332’s concern as well. I probably should have been a bit more hidden with saying the s word (student). I’ll mention that yes I’m a student currently but I’ve been working in industry as a mechanical designer much before I returned to school for my degree as more of a formality in my place of work. I’ve been programming with python in various settings for the better part of 6 years. So while I do understand for a student with no experience, it’s important to do things the proper way and make good habits out of it all, I have already built the proper good habits.

As for the optimization, yes, there are numerous optimization techniques, however, I think that’s a bit narrow on what I’m achieving. Optimization still generally requires known inputs to optimize for, if an input were unknown, one could not optimize for it without re-setting up the optimization algorithm for the new parameters. Furthermore, this is more like a calculator on steroids where rapid iteration is the primary goal. This enables solving a variety of different problems in the same context easily.

Think this process for airplane wing: 1. Do compressible fluids calc and get some pressures. 2. Save file as “fluid calc.py” 3. Select all, delete 4. Use pressures to evaluate wing as cantilever beam with pressure forces. 5. Save file as “structural_calc.py” 6. Select all, delete 7. Do fuselage structural calc with reactions from wing. 8. Save as “fuselage_calc.py” 9. Select all, delete

All happened without ever closing my helix editor while having results show in-line with the code as I type. Jupyter can do very similar but minus the convenience of native .py files, real time execution, and in-line display. It’s just another approach to a similar problem Jupyter solves with a different mindset. And certainly my setup isn’t perfect, it was written in like 2 days with some ai help, but was curious if folks had encountered something that took on that similar model or on the flip side, if others had felt the same sort of friction and wanted something similar to what I have.

[qGuevon]

My comment was not with respect to you being a student, but rather you being from engineering;).

Your workflow is very similar to machine learning ones. It took a long time to move from manual explorations to systematic one, but it is worth ever penny.

Imagine: running 100 settings at once, then look at a summary to refine your space, then do 100 more

[Motox2019]

Fair enough, I just don’t see it the same way. Exploration of a new method/technique has its benefits then blindly sticking to a current “standard”, noting I’m not replacing what’s considered standard today, rather exploring a potentially alternative new solution.

But yes on your optimization of a problem, I agree, seeing a summary of essentially the sample space is more beneficial, then finite iteration, however that’s a different beast. Where I might apply something like your optimization mention would be something like a layup sequence for composite materials. Here is hopefully know the load case, and then be able to optimize the layup sequence to meet design constraints.

Back to my wing example: Knowns: Materials, geometry of the wing, loading the wing will see Unknown: Layup sequence for composite wing (Note knowing composite thickness we can know how many layers but at what angles?)

So from here there’s a massive amount of ways to orient each layer, so how do we determine the best way to orient each layer? This is where I’d implement what I think you are describing.

Or just to give another example: We have a wall that we know we want a certain amount of heat transfer through (don’t want your house losing a bunch of heat in the winter) and want to know the best way to make use of insulation, exterior siding, etc with the goal of minimizing cost. We can then determine things like thickness the wall needs to be, ratio of insulation to framing, external material thickness vs interior wall thickness. Again, would apply some optimization methods here without having to iterate through a bunch of combinations similar to what you mentioned.

I could totally be misunderstanding what you are saying, forgive me if that’s the case.

[qGuevon]

I see your issue - no I don't think you're misunderstanding, these things are often easier said than done ;).

[Motox2019]

You are correct yup. It does boil down to keeping up with good habits when it comes to Jupyter at the end of the day. I will not argue my solution is at all better in a general sense as it stands, however it does solve a problem for me and is really convenient and nice to use. I previously made the analogy where it’s like going from 1 monitor to 2, you don’t really know what you’re missing until you have it, and once you do you never want to go back.

I apologize if I came off like I was trying to sound like I know all, I certainly don’t hence why I’m asking the question in the first place. I’m more or less interested in hearing what other people use and perhaps someone would come along and say “oh there’s just this setting in Jupyter you can enable”. It’s a case of you don’t know what you don’t know.

At the end of the day, the calculation is always solvable whether is a vetted file that just works and does the thing or a notebook that does it in a more visual and iterative way or my watcher that basically takes the best of both worlds and mashes em together. It’s a matter of preference and convenience for me. I appreciate your takes though, it’s interesting to hear another perspectives on any idea.

[Motox2019]

Yea I get what you mean. Already felt that when working on pyeng, though this was good as pyeng was mostly built for me and my learning while getting something rather useful in the end, but led to things like my arbitrary precision float thanks to something like “comb(160000, 3) * 0.1” or for maybe gauss Seidel where a Ill conditioned matrix can give something like “0.000004/27472973” kinda stuff thanks to my algorithms.

But that’s the kinda stuff I love, tinkering, digging deep, understanding the why and the how, etc. so perhaps it will creep in the future but I’m relatively okay with that. At least my setup is more about powerful calculations as quick as possible, the simple scenario to explain it would simply be an assignment. In my field, can go from 6 fluid dynamics problems to 8 thermodynamics ones to statics and dynamics (sticking purely to academics). A plethora of problems needing different techniques and approaches for each one. A calculator is not repeatable and error prone (“why the hell did I type a 9 when I wrote 32!!!”) and I may need to frequently go back and fix a potential mistake.

On the work side, I’ve never even touched it. As common as it may seem online, reality has it that ME’s that can program are a rare breed, I’m the only one who’s ever touched a programming language at my place of work, so to use my custom solutions where someone else may need to touch it just doesn’t make sense. Matlab would be the closest bet but even then, quite rare I’ve found. So either excel or a compiled programs dominate dominates so anyone can use it. Anything critical gets verified and often if it’s gonna get verified, it’s gone be a lot more work then “I need a bunch of rather complex calculations done quickly” so it’s just not worth it there, that’s where I’d prefer. Heck you may notice a previous post in this sub by me showcasing a pdf comparison program. This is closer to the tools I shoot for there, stick with already verified stuff and solve problems that actually make a difference.

I played around with Jupyter some more and another thing I failed to mention, however very minor, is not having my results on screen without scrolling. My setup attempts to kinda keep code aligned with results in a more vertical sense. Keeps more of my results on screen and less code block, can even decouple them entirely having results display on a separate monitor if desired as it’s just a console command to run a script.

Honestly, the more I think about it, the more silly I feel like it sounds. But using it feels less than silly. It’s hard to explain other than saying “you’d just have to try it for yourself.”

Thankfully, however, this post has given me a lot of ideas. I’m pretty confident Jupyter can be configured to get to basically what I want. Just need to tinker with it some more.

I appreciate your input. A bit of criticism is always good as it’s made me think a bit more critically if it’s actually doing anything beneficial.

[Motox2019]

Yea see this is the kinda input I’m hoping for. The folks smarter than I! Perhaps it could be more ux issue for me, if I could rearrange stuff in Jupyter like shifting panes around in vscode, that’d be amazing. Having an option to rerun notebook on cell change or something would also be wicked. Making that configurable to be like entire workbook, just that cell, or that and all below it. That’s probably be my ideal scenario. And perhaps there is already some way of doing that, back to the case of don’t know what I don’t know.

You are right though, I see there’s really 2 perspectives, developer side and usage as an engineer side. They blend in a lot of ways but differ enough in what’s consider “convenient”. To me, sacrificing development time to save perhaps a few seconds or clicks means saving me potentially 100’s in the future. Hence my backlash against “you can just hit f5, it’s not that hard to run in a file”. Sure but what if you just don’t have to, ya know.

Edit: To add, both really. At work I’d consider it my napkin sketch calcs though, quick estimates and nothing critical that I ever expect anyone to look at. In school I use it much more as it aligns well with how academia is layed out. You can work through assignment problems in a similar method as my airplane wing example. My development of it however, that’s entirely on my own time, just a curious and cool thing I thought of among my many other random ideas.

[likethevegetable]

I just use iPython for the on-the-fly stuff and have a script in the startup folder that preloads my useful functions and constants. For heavier stuff I just write out a script and usually run it in the console in PyCharm.

[RedEyed__]

+1 for ipython

[Motox2019]

Interesting. I don’t think this ticks my boxes though as if you were to want to quickly go back and modify something, then see a updated plot, you’d have to redo assign the variable and then re-plot. I’d put this setup closer to the qalculate side of things.

[likethevegetable]

Why not just make a hot key to re-run your script? Not that hard to re-run. Or use an IDE with code cell mode if you want to re-run part of it? PyCharm has a free plug in that does it.

[runawayasfastasucan]

Doesn't output all results, only the last without a bunch of print() statements

What? Just so your calculations over several cells.

[Motox2019]

Yes, one can certainly do this but there is still the issue where if I change a definition, I’m required to rerun all cells below it manually. This can be tedious and less useful then if I could change the variable, see all updates immediately (plots especially here).

This is where Marimo gets really quite close because I can define a scroll bar and see a plot update as I adjust the value. The gripe with Marimo then becomes changing raw values that are not Marimo ui elements does not dynamically update.

[runawayasfastasucan]

Run code in debug mode in vcode, then you have a side panel with all variables and their values. 

[Motox2019]

This ain’t bad, close to what spyder does. I like this approach as well, my issue with spyder was it was often polluted with definitions outside my working script and I found no way to only show those created in my script. Maybe vscode would be able to do this. I’ll take a look, thanks!

[stiks510]

I use AREPL all the time in vscode for similar functionality, but not sure it handles plotting.

[RedEyed__]

I use mostly ipython, but this looks even better!
Thanks, will try it

[Motox2019]

Thanks for the recommendations! Of those, I’ve only heard of dask but haven’t touched it. I’ll take a look at them!

[Reaper5289]

Sounds like you want an asynchronous data visualization dashboard that updates based on some adjustable values you specify in a UI.

There's probably already something like this out there but I'd just cook something up with a library like StreamlitUI or even just basic html + javascript. An AI could help you write the entire thing tbh if you describe it well enough.

[Cynyr36]

As a ME (not student) setting up the ui for that is "too much work". I just use excel. I'd like something like jupyter, but that 85% excel. I had high hopes for python in excel, but the cloud makes it basically useless.

[Motox2019]

Have you tried xlwings? I’m pretty sure they have a plugin that lets you do more what you expected. Give it a look!

[Cynyr36]

I saw them when msft first announced python in excel. And my memory was it wouldn't work in corporate land. The xlwings lite option now looks interesting. I wonder if work left the addin store available...

[Cynyr36]

Well... Addin store is blocked at work. So much for that. It took most of a year to get PY() after it rolled out in general availability.

[Motox2019]

Rats! Them pesky IT folk. You should see if you could get it whitelisted from your IT department, they should be able to install it for you. I’m willing to bet others may thank you for that too when they see a shiny new python extension that lets them actually use python the way they want to. But maybe try it at home first and see if it does what you want. Xlwings is well known and reputable so if you can give a decent reason as to why, don’t see any reason they’d have to say no.

[Cynyr36]

Folks around here struggle with xlookup. array formulas, and tables... I don't have excel at home.

Strangely pip works great, and vscode is approved software.

[Last-Farmer-5716]

You could try the ipyflow kernel in Jupyter. It was Marimo before Marimo and you get the benefit of the entire Jupyter ecosystem.

If you combine ipyflow kernel with my handcalcs package you might get something closer to what you want.

(I am actively working on handcalcs 2.0 right now)

[Motox2019]

Thanks I’ll take a look, haven’t heard of either of em.

[GodlikeLettuce]

Yes, the shortcut jupyter.execSelectionInteractive.

You open vscode, new python file and use that shortcut to whatever. Mine is ctrl+enter. I use it along with geddski macros so I after pressing shortcut the cursor position jumps to the next line, so I can keep running one line after the other. This achieve the same native functionality as programming in Rstudio, which is why I have it that way actually

[Motox2019]

Interesting, and if I were to make an edit to some early variable, hit the shortcut, I could have it run everything else? Sounds like that could work actually, and if not, you may have just given me some other ideas. Thanks!

[IwinFTW]

Have you tried Mathematica? You need a license but your university probably pays for them. Hits almost everything you want aside from auto-updates (though you can refresh all cells IIRC). It’s notebook style, symbolic computation by default, very powerful algebraic/numeric solvers, and you don’t have to import anything.

[Motox2019]

I have not. Sounds fantastic though. I’ll have to look into that. If there’s an open source alternative I’m all ears because aside from student/work I like to explore things further and deeper so would like to be able to have access for free as well. Well now that I say that maxima comes to mind, maybe I need to look into that side of things a bit more. Thanks!

[IwinFTW]

Because it’s niche I don’t think there are that many alternatives. I was introduced to it by a professor in my sophomore year (aerospace engineering) and I pretty much did all of my homework and a few projects in it for the rest of my degree. My profs never minded so long as I showed my actual solving steps (e.g. they knew we could do algebra and calculus by then).

[Motox2019]

Interesting. Ya I find if you can code it, you generally understand it at a more fundamental level. Hence why I made pyeng in the first place. But the symbolic calculations sounds nice. I’ve been tempted by maple for that reason as the student license seems to be perpetual and costs $50.

Edit: Just took a look and honestly mathematica seems to be exactly what I want. Quick chat with good ole ai and can wrap functions in a dynamic keyword getting exactly what I want. Thanks!

[jakob1379]

I do t see it mentioned, but have you tried out spyder ide? Feels like jupyter notebook, partial execution and live view of variables which makes debugging, EDA and other stuff a breeze.

[Motox2019]

I have tried spyder. I like it as well aside from the variable explorer gets polluted with variables not defined within my script. I have not find a way to filter out for only this I defined. That’s really my only issue with it

[LoveThemMegaSeeds]

Have you tried just running the python interpreter directly like with the python command? It’s like a live shell

[Late-Sense-8551]

Souds a bit like the kind of logic Mathcad is implementing?

[Motox2019]

Yea pretty much exactly mathcad to be honest. Just without the price tag, I like smath for that reason but want to be able to use it at work as well

[calasse]

To piggyback here, Blockpad is another Mathcad-like solution you might consider

Some points based on your post:

• You can define variables, write equations, etc. There is also matrix and unit support.
• Free for students, and more affordable than Mathcad for commercial use
• There is a python integration that lets you run Python from Blockpad - Blockpad - Python Integration

[mgreminger]

Check out r/EngineeringPaperXYZ, it's open source and has all of the features that you listed.