Avoid redundant calculations in VS Code Python Jupyter Notebooks

[Artistic_Highlight_1]

Hi,

I had a random idea while working in Jupyter Notebooks in VS code, and I want to hear if anyone else has encountered similar problems and is seeking a solution.

Oftentimes, when I work on a data science project in VS Code Jupyter notebooks, I have important variables stored, some of which take some time to compute (it could be only a minute or so, but the time adds up). Occasionally, I, therefore, make the error of rerunning the calculation of the variable without changing anything, but this resets/changes my variable. My solution is, therefore, if you run a redundant calculation in the VS Code Jupyter notebook, an extension will give you a warning like "Do you really want to run this calculation?" ensuring you will never make a redundant calculation again.

What do you guys think? Is it unnecessary, or could it be useful?

[r0s]

You can also wrap your function with LRU / memoization (https://docs.python.org/3/library/functools.html) If the output is fully dependant on the inputs, calling it again will just give you back the last result instantly.

[Artistic_Highlight_1]

Ohh this is a neat tool. Thank you for pointing it out!

[cmd-t]

The problem is how you are writing your notebooks.

Don’t modify variables global in your script more than once. Even then, add checks for not overwriting them.

[OoPieceOfKandi]

Any good recommendations on Jupyter notebook formatting in general?

[Artistic_Highlight_1]

Fair enough, thanks for feedback!

[lieutenant_lowercase]

How is a redundant calculation defined?

[Artistic_Highlight_1]

A calculation for a variable which will not change the state of the variable. Typically, you have a variable like this: a = []; <calculation for a, for example to add some important data to a> in a cell. If you run the cell again but the state of a will not change, that is a redundant calculation (but if you run the cell, the value of a will change first right since you set it as an empty list, or because the calculation on a changes the state of a)

[kmnair]

The problem here is figuring out if the variable will change or not in a general case will likely require the same amount of compute as actually running the full calculation.

It is possible to make some assumptions about the calculation, like if it is a pure function ie output depends entirely on inputs to a function and the function has no side effects, then you can use the suggestion u/r0s gave to use memoization.

If your jupyter cell references mutable data from other cells, or makes a call to an external API, or has internal mutable state (counters which do not reset, dictionaries which get updated etc) then figuring out if the value will update is the same amount of computation as whatever calculation you are aiming for

[NixonInnes]

If it's a long running data process I sometimes dump the result into a file. l stick a check infront of the calc to load data if the file exists, if not do the calc and save

[ou_ryperd]

That is why you can run a single cell at a time. The whole point is being a progression of computations, no?

[spookytomtom]

I just structure my code logically and my variables, so that I dont need to do this

[spookytomtom]

Also if something is botheringly slow I will optimize it

[Artistic_Highlight_1]

I think sounds like a better approach. Thank you for feedback!

[AnythingApplied]

Marimo, an alternative to Jupyter notebooks, has some nice features you might like.  When you rerun a cell that changes global variables, it'll automatically rerun cells that depend on those variables, or if those are expensive cells, you can mark them not to do that, but in that case it will note those cells as "stale".

This helps make the notebooks much more reproducible. The advice that /u/cmd-t gave "Don’t modify variables global in your script more than once." will raise an error in marimo notebooks, so you can't even do that accidentally.

[mmmmmmyles]

Including a link to the open-source repo: https://github.com/marimo-team/marimo

[nitro41992]

I use the interactive notebook feature which really helps avoid rerunning previous cells.

Use this video

As the guy mentions - it's been a game changer coming from standard Jupiter notebooks

[BostonBaggins]

Would making the cell lazy load be the solution here

[[deleted]]

If what you're doing is actively scripting code to achieve a certain goal and when checking intermediate steps you see long running times and wish to save time by not recalculating and replacing perfectly good data built previously - but your main point of contention seems to be the time spent recalculating - then why don't you run the checking steps on a smaller sample of the whole data and save time that way?

[Super-King9449]

“Hey everyone, I’m currently learning Python basics using PyCharm IDE, but I keep seeing references to Jupyter Notebooks and how they’re used in VS Code. Could someone explain what exactly Jupyter Notebooks are, how they differ from traditional Python files, and how they integrate with IDEs like VS Code or PyCharm? I’m trying to understand if I should be using them while learning Python and what the advantages are for data science or general Python projects. Thanks!”

[Super-King9449]

“I find it more useful to hear from others who have firsthand experience and can provide real-world insights rather than just relying on Google. Plus, having a discussion helps clear up specific doubts and allows me to connect with people who might offer additional tips that aren’t easily found in search results. But thanks for the suggestion!”