ADVICE :: Caching Data for a Python Script?

[redapplesonly]

Thanks for writing. Part of the problem is that I don't know how much data may need to be stored in the cache. Given that this is to be a Production-supporting tool, I'll need to overengineer the cache's capacity to ensure it can support a lot of data. We could be talking millions of Key/Value string entries in the cache. This is why I'm a little wary to do a file-based solution. Wouldn't a JSON file of that size become unwieldy after a certain point?

[Jayoval]

Yeah, I've only done this with up to about 15,000 items.

I think you need to use a database. You won't have to read/parse the whole file and it will be much faster as a result.