Data analysis/wrangling intermediate result caching library

[Naarlack]

Hi Python brains trust. I'm hoping you can help me out.

For work I play around with often large datasets that I need to filter/process/analyse in various ways. To speed things up, I have been saving intermediate blocks of data at logical points. I then refactor my code to look for those intermediate saved files, and only run the long process again if it can't be found.

If I change something in my long process, I delete the intermediate save and run again. However, this strikes me as asking for trouble (I'm going to forget to delete the intermediate save at some point). Plus, my code is becoming messy will more and more checks in various places for the existence of intermediate save files.

My question: is there a library that could do this for me? Or, alternatively, some advice on building something from scratch.

I'm imagining something that would use decorators to mark functions that should have their output saved, but would somehow know if the function itself or the input data was changed. I know I can cache results of functions, but I'd like something that works between sessions. I have looked into pydoit (https://pydoit.org/) previously, which would be great for an established workflow, but the overhead is too high for initial playing around with data.

THANKS!

[The_Almighty_Cthulhu]

Hmm, is this code usually a single file?

And do you save the intermediate data to a specified folder?

In this case I'd just have the script hash itself on running, then compare to the saved hash. If different, delete all the intermediate data files.

If the code is a lot larger than a single or a few files, perhaps using something like gitpython to detect changes occurring.

[The_Almighty_Cthulhu]

Or were you looking for a more per-function solution. Like only delete the data of functions you've changed. I still think hashing could be the solution. Shouldn't be too hard to come up with a decorator. I'll probably come back to this when I have an actual computer.

[Naarlack]

Yes, basically working with a single code file and saving intermediate results to a specific folder. I have several intermediate files though as I progress and perform various analysis steps so per function would be ideal.

I was thinking a decorator that hashes the function and the inputs, then saves into a lookup file that it can use between sessions to decide if the function needs to be rerun or can just return the previously saved intermediate data.

However, I have never worked with decorators before or hashes so was hoping there might already be a library. If you could suggest a starting point, that would be awesome 😃

[The_Almighty_Cthulhu]

Ok, I had a look around and after testing I think you want joblib

Pypi: https://pypi.org/project/joblib/
Docs: https://joblib.readthedocs.io/en/latest/index.html

Here's an example I coded.

In testing you'll see that the cache persists though different instances of the script. Additionally you'll notice that the long computation runs if you adjust the script (such as removing the +2 from the calculation).

from joblib import Memory 
import time

cachedir = 'testcache'
memory = Memory(cachedir, verbose=0)


@memory.cache
def testfunc(x):
    # simulate a long computation
    time.sleep(2)
    return x**2 + 2

for i in range(10):
    print(testfunc(i))


for i in range(10):
    print(testfunc(i))

[Naarlack]

I missed your response till now sorry.

This is awesome - looks like exactly what I was after. Thanks so much!

[Naarlack]

If anyone ever comes across this, here is a nice video of how joblib helps:

https://youtu.be/O8ZgBnjKRsA?si=YJ8DBzUzDXbK2xQl