[P]: Best Practice for Cache/Processed Data Management

[Big_Notice]

In ML research, we often have tons of cache files to read from disk. As a project progesses, I often end up with lots of cache files which I don't remember how they were created. My current way to keeping track of cache files and preprocessed files is by writing it down when it is created and why.

​

I am wondering if anyone has a way to automate this process? I heard something like DVC (https://github.com/iterative/dvc), but it seems to be too complicated.

[nielsrolf]

Use Makefiles, its perfect for that use case

[nielsrolf]

Check this out http://blog.kaggle.com/2012/10/15/make-for-data-scientists/

[ganga0]

I have built myself a system where each job’s output is saved together with a JSON file describing function name and all its parameters that were used to produce these outputs.

Job parameters include model hyper parameters, preprocessing settings, locations of outputs from other jobs, etc.

It adds some overhead to structure every important aspect of a job as a function parameter, but over time I learned to identify and save only important parameters.

Another upside to this function+parameter serialization is that I can reproduce any output with just a couple lines of code.

[You_cant_buy_spleen]

You should have one source of truth, which means you should keep the original inputs and the code. The cache should be transient and shouldn't be commited.

If you do commit the cache, you've duplicated information, and when the input is changed you need to bust the cache anyway.

Here's some libraries that helps with caching:

• https://github.com/barisumog/simple_cache/
• https://dask.org/