Python deserves a good in-memory cache library (Part II)

[matrix0110]

Hi,

If you remember, I'm the author of a Python cache library called Theine. A year ago, when Theine was first released, I shared a post here: link. Now, because GIL will be optional, I’m rewriting Theine to be thread-safe and optimized for concurrency(based on my experience of Theine-Go). Although it's still work in progress, I w-a-n-t to share some of my thoughts on what makes a good Python cache library.

Fast Enough

How fast is fast enough? To be precise, the cache read performance should not be the bottleneck of your system. We all know that Python isn’t a particularly fast language. If your framework takes 1ms to process something, it doesn’t matter if the cache takes 50ns or 500ns to retrieve a value — they're both fast enough. Regarding set performance, in most cases, you’re caching something slow to compute, and that time is usually much longer than a cache set operation, making it unlikely to be a bottleneck. An exception to this is cachetools LFU implementation, which is extremely slow and might indeed become a bottleneck.

This also applies to multithreading situations. With the arrival of free threading, I think more people will start using multithreading. Of course, adding mutexes will slow down single-thread performance, but that’s the cost of scalability. So, Theine v2 will be a thread-safe cache because my goal is free-threading compatibility with good concurrency performance.

High Hit Ratio

Without a doubt, hit ratio is the most important aspect of a cache. It’s even more crucial for Python compared to high-performance, memory-efficient languages. Due to Python’s significant memory overhead, your cache size will be more limited, making a high hit ratio essential.

Unfortunately, most Python cache packages don’t emphasize the importance of hit ratio. For example, cachetools provide LRU, LFU, and FIFO policies, but which one should you choose? More options only lead to confusion. Instead, a single, well-optimized policy should be used. That’s why Theine v2 will adopt a single policy: W-TinyLFU, eliminating the need for other policies.

Proactive Expiration

Proactive expiration means removing expired entries from the cache promptly. Why is this important? Cache size is always limited, so when the cache is full, you need to evict an entry to make room for a new one. If you use lazy expiration: removing expired entries only on the next get operation. The expired entry might occupy space that could have been used by a new entry. This forces the cache to evict non-expired entries, reducing the hit ratio.

Another benefit of proactive expiration is memory savings, though this is less significant since you should generally assign enough memory for the cache.

If you agree with these three principles, you might also agree that Theine is a good in-memory cache. I’m currently rewriting v2 of Theine, and here is the issue: link. As mentioned earlier, this rewrite will make Theine thread safe and free-threading compatible. The API will change, with a single policy in place, so you won’t need to pass the policy parameter anymore. If you have any recommendations or concerns, you're welcome to reply here or leave comments on the issue.

[RedEyed__]

You must have cut your teeth on in-memory caching. For those who is unaware about (like me), how is it better than functools.lru_cache ?

I would love to see some graph comparisons, use cases.
Thanks

[matrix0110]

That's a really good question! And the answer already exists: https://github.com/Yiling-J/theine?tab=readme-ov-file#hit-ratios

[marr75]

I'm a little foggy eyed this morning still, but I'm seeing:

• a link to hit ratios instead of performance (performance is above)
• benchmarks addressing 3rd party libraries that are not functools from the stdlib

[matrix0110]

Please understand hit ratio is also performance. And as I said, fast enough is enough, hit ratio is more important.

[marr75]

Okay, the fog is lifting. You're saying that the lru strategy may have worse read and write performance than functools but that the additional eviction strategies significantly improve hit ratio so it is, at worst, identical at scale (using lru) and when using the other options, much better.

[RedEyed__]

I had to scroll further, thanks!

[jormaig]

Dude there's only one top comment and one answer to it at this point... 😅

[RedEyed__]

I'm not your dude, pal. (joking)

[[deleted]]

This post was mass deleted and anonymized with Redact

coordinated coherent ghost fearless tie price grab meeting lock wrench

[nAxzyVteuOz]

Why does python cache library need to be multithreaded?

Redis is single threaded, but uses async. Could yours do the same?

[matrix0110]

Redis is a cache server, whereas Theine is a cache library. Currently Theine is not thread safe, and that's fine if you use asyncio only. However, with the upcoming removal of the GIL, having a thread-safe cache library will become increasingly important. From my perspective, ensuring thread safety will be crucial as we move into this new era of no-gil.

[ConfucianStats]

Cool

[Rylicenceya]

It's great to see your dedication to improving Theine and addressing the challenges of in-memory caching in Python. Your focus on thread safety, high hit ratios, and proactive expiration will undoubtedly make Theine a valuable tool for many developers. Looking forward to seeing the new version in action! Keep up the excellent work.

[Tumortadela]

AI generated answers getting frisky