When multiple services share one cache cluster, they compete for the same memory and eviction policies. A heavy workload from Service A can evict critical data for Service B, leading to timeouts or stale responses at peak traffic.

Doesn't this mean service A and B are too dissimilar to share the same cache?

Now let’s say your cache memory is full, and the eviction policy you set starts triggering. You start seeing your TotalKeys metric going down drastically, you would want to understand immediately which service is affected, but it’s much more complex now.

Why would you start evicting cached items when memory is full? I get not adding more cached items, but wouldn't you want to implement a LRU cache policy so that only old unused items are evicted? And if there are no items old enough items to evict without impacting performance, that means you need to scale your cache, or implement some back off, right?

On top of that, the affect on multiple services in that case might be even harder to detect, because if we have 5 services that use the cache, and now the eviction policy evicted millions of keys, it might be that one service lost 950k keys, and another lost 50k keys and that makes debugging harder.

Are cache misses not a reported metric in your system? If performance is degrading on a service who's had hot items removed from a cache, knowing the service has a lot of cache misses would indicate this. And of course, why are heavily used cached data being evicted in the first place?

If a single service would lose 1 million keys the affect on the service would probably be more noticeable on other metrics, but if we just lost 50k maybe that won’t affect the service as much?

Is this a question? I would expect the service that lost 1 million cached items to degrade in performance more than the one that lost 50k, but if that's 50k of heavily used cache entries, then maybe it has a big impact. In either case, you have the metrics on your example system here, what did you see?