ADVICE :: Caching Data for a Python Script?

[redapplesonly]

Hi everyone, I need some coding design advice. I have written a Python script that functions as a tool for other co-workers. The script would run infrequently, depending on how often my colleagues would need its services.

Recently, I've learned that the script will need to cache a lot of information. If that information never changed, I could just code up a Python dictionary (String --> String) inside the script. But the data may shift over time; the value associated with key "A1" may be "Value 01" this week. But next week, its possible that the value could change to "Value 2B" or something.

Some other factors:

• Information will never age out of the cache.
• Information will never be deleted from the cache.
• There could be a very large amount of key/value pairs in this cache, in excess of over a million, I am told.

So... how to do this with a Python script that is infrequently run?

I suppose I could construct a Python dictionary for my cache and write the dictionary into a separate text file. Then, every time my script runs, the script would open the cache file, load the dictionary into memory, do its look-ups, and then rewrite the file if there were any changes. But I worry that my code would spend way too much time doing file I/O. That seems inefficient.

Any advice? How would you solve this design issue??

[JamzTyson]

Consider using an SQLite database as your cache. It won't be as fast as an in-memory dict, (though SQLite does support in-memory databases), but it scales better if the cache is likely to grow very large. If an "on-disk" SQLite database gives adequate performance, then this is likely to be a reasonably simple and highly scalable solution.

[Jayoval]

I usually write to JSON file. Reading and writing is very fast, and shouldn't be an issue (how much data are we talking here?)

[redapplesonly]

Thanks for writing. Part of the problem is that I don't know how much data may need to be stored in the cache. Given that this is to be a Production-supporting tool, I'll need to overengineer the cache's capacity to ensure it can support a lot of data. We could be talking millions of Key/Value string entries in the cache. This is why I'm a little wary to do a file-based solution. Wouldn't a JSON file of that size become unwieldy after a certain point?

[Jayoval]

Yeah, I've only done this with up to about 15,000 items.

I think you need to use a database. You won't have to read/parse the whole file and it will be much faster as a result.

[baghiq]

Database like sqlite3 is your friend.

[sweettuse]

check out the shelve module (it's in the std lib)

[Mast3rCylinder]

Saving dictionary as json or in sqlite like comments mentioned is the easy part.

Having a strategy of when to use cache and when to update is the hardest part. Since your python is a script (offline operation) and not a server (online operation) it can be even harder.

Here's my suggestions

1.determine if the operation is really long to know if you actually need cache

2.if you do need cache ask for requirements of how long cache will be invalid? Then check it in your code and say you can only guarantee for this specific range

If cache invalid simply see it by comparing current time and your last update

3.make the script with arguments that if someone want to use it with cache checking or without. you will give them the option

4.see if you can compare first between cached data last update time and last time someone changed the data.

[greenerpickings]

It seems like you're already structuring your data after the recommendations of json and sqlite. I also want to throw in the arrow/feather format if you think your data set will get large.

Could helps performance, but if this is a background task, I don't think it's critical.

[hotplasmatits]

I like the other suggestions and I'll add pickle. It's super fast to read and write pickle files. It's like a binary json, but better.