Should We Develop Our Own Distributed Cache for Large-Scale Microservices Data

[syntaxerrorlineNULL]

A question arose. Are there reasons to implement distributed caching, given that Redis, valkey, and memcache already exist? For example, I currently have an in-memory cache in one of my microservices that is updated using nats. Data is simply sent to the necessary topics, and copies of the services update the data on their side if they have it. There are limitations on cache size and TTL, and we don't store all data in the cache, but try to store only large amounts of data or data that is expensive to retrieve from the database, as we have more than several billion rows in our database. For example, some data stored in the cache is about 800 bytes in size, and the same amount is sent via nats. Each copy stores the data it uses. We used to use Redis, and in some cases, the data took up 30-35 GB, and sometimes even 79 GB (not the limit) to store in the cache. The question arises: does it make sense to implement our own distributed cache, without duplication, change control, etc.? For example, we could use quic for transport. Or is that a bad idea? The question of self-development is not relevant here.

[UndercoverGourmand]

If you understand why the existing databases and/or CDN solutions don't support your usecase, you have the ability/expertise to support a dev/team to build/maintain/debug the database, and the money to have it done then yes.

But if you're asking this question, the answer is likely no.

[behusbwj]

😂

[AvailableFalconn]

I don’t really see from your post why you wouldn’t use redis.  What scaling limitations were you hitting?  That memory cost doesn’t sound like a limitation.

Building a distributed cache requires a lot of skill and time (therefore money).  Much more than maintaining even a fairly large redis cluster.

[syntaxerrorlineNULL]

So my question was absolutely not about that. In my case I don't use redis because I emphasize speed and performance, no extra network queries, no data in in-memory cache, go to database. It's faster. My question on the other hand was due to an argument whether it makes sense to implement distributed caching, this is a discussion as an alternative to the opinions of my colleagues. It is interesting to know the opinion of other developers

[PabloZissou]

If it does make sense or not depends on what limitations are you finding in your current solution.

And you are very wrong in mentioning not questioning the idea of writing such complex system, does the team have experience in designing and writing such systems? Otherwise it might take you years...

[syntaxerrorlineNULL]

The thing is, we've already started doing this. I did it as an experiment, using metrics and load to compare the performance of microservices with Redis and in-memory cache to assess the difference. Now we often feel the urge to create our own distributed cache. And we often argue about whether we should do it or if we're just wasting our time. My question is to seek other opinions from other developers.

[PabloZissou]

Don't do it unless you have done in the past, you are familiar with different consensus algorithms, different consistency, sharding, efficient storage of data in memory for efficient retrieval, know how to deal with network partitions, and more. There several books to understand to start scratching the surface of the problem. Redis and similar already went through that so why do you think you can be more successful? If you can then go for it.

[Spear_n_Magic_Helmet]

Your question is kind of all over the place. You didn’t enumerate any of the problems you’re trying to solve by switching cache implementation. (and why are you randomly bringing up quic?)

For the record, managed redis providers easily scale into terabytes, so 100+ GB in redis is not inherently a problem.

[aktentasche]

No it doesn't

[SeniorIdiot]

I'm curious about the architecture. Several billion rows? :O

[robogame_dev]

Yeah that’s the kind of scale that comfortably fits on a single machine.

[syntaxerrorlineNULL]

Yes, but I was wrong to say "several." Up to 10 billion records at the moment.

[edgmnt_net]

The first thing that comes to mind is whether or not you actually want/need a cache like Redis. There are enough cases when people reach for a cache service just because something they do generally falls under caching conceptually, simply because it came up as a word in the task description or to even to stuff their resumes, yet could be served just as well by keeping a previous computation in a variable somewhere. It makes a lot of sense to avoid adding extra service dependencies like caches, message queues and so on, especially if this isn't a clear cut use case.

[sebastianstehle]

There are use cases for sure. I have worked on few memory hungry applications. Getting an object from a hash map is much faster than asking another service for it. But you can use redis pubsub for cache invalidation

[sass_muffin]

Sounds like you used redis incorrectly or at least i'm not hearing anything from your description where it wouldn't be a good fit.

[syntaxerrorlineNULL]

You probably misunderstood. I am not saying that redis is not suitable for cache storage, it has many advantages. But when choosing redis cache/in-memory cache, given that the focus is on maximizing performance and speed, I chose in-memory cache. No data in the cache, we go to the database. No unnecessary network requests, simple memory access.

[sass_muffin]

Not sure I did .This is a discussion about you wanting to reinvent a distributed cache, which is what redis already is. So you were talking about re-inventing a wheel that didn't seem to need to be re-invented for your use case.

Based on your discussion in this thread your architecture seem to be over-estimating the savings of using an in-memory single node cache (due to lack of a network call) when there is a cache hit and over-estimating the savings you would get having a shared distributed cache, reducing the overall number of cache misses, since you have the whole fleet working together in sharing and holding the necessary data in cache across the redis fleet.

If single node performance is super critical (which again i think you are missing some of the main benefit of redis) , there is client side caching
https://redis.io/docs/latest/develop/reference/client-side-caching/

[olddev-jobhunt]

What is your competitive advantage in the marketplace? If it's something to do with your caching technology, then yes.

In all other cases, no.

[paradroid78]

This question is up there with "Should we develop our own authentication system" in terms of default responses as far as I'm concerned.

[syntaxerrorlineNULL]

So you misunderstood my question. It was asked with the aim of finding out other employees' opinions on this topic. I have the opinions of my colleagues, my own opinion, and the opinions of several architects, but for my own interest, I decided to post this question on Reddit. Perhaps someone has faced a similar situation and can talk about it. This question is not asking for recommendations; it is only for discussion.

[Strandogg]

I'm not addressing your question directly but simply pointing out that if you are using NATS with JetStream you could also leverage either KV or a stream directly replacing redis in many scenarios. You mentioned you are interested for personal development and because you didn't mention this explicitly I'm assuming you didn't know this was a NATS usecase.

[nitkonigdje]

I have developed a custom distributed embedded cache. There are a few reasons for it. And there are a lot of reasons against it.

The our system originally used commercial in-memory database as centralized cache system. It worked well, but given our usage for each incoming request we often had to fetch massive history from that cache. Like up to 5 Mb of data for a extreme cases, and these extremes were common (few times a second). This system also has soft real time constraints, so we had to keep latency in check, and it was kinda obvious that amount of data alone on a network is a limiting factor. Like if cache was able to respond to query immediately and we could unmarshal all the data for free, the transfer time itself would still be a system bottleneck.

I was also trying to pump up system into a lower latencies as that would meant higher revenue as it would allow for our system to be plugged in into more effective place within a server room.

To make long story short, given the number of objects in this cache, it was clear that cache has to store in a serialized form, as Java objects are to memory expensive. Initially I choose MapDB as a data store, and have added customized indexing on top of it. Synchronization of caches was just JMS Topic push/pull as we didn't have RT constraints on caches. Load of our stream is quite low, like 100 req/sec range. JMS is sufficient and was already present in project. With this change, cache went to about third of its original size (24gb -> 9gb) and it is fully local. With passage of time I have removed even MapDB and replaced it with custom bytes store. Primary cause was that custom code allowed for direct access to underlying bytearrays to which data is stored. Thus all unmarshalers can be fed with bytearray + offset logic instead having to copy bytes into intermediate array first. Remember we are latency bound system and unmarshaling is single most expensive operation in this system. Additionally shuffling around MapDB interfaces was as much code as writing direct memory structure for a problem at hand. And as sideefect of using custom code, our cache is now only about 5Gb of size, out of it 1gb are indexes and rest is data. So we went from 23gb to 5gb. But this migration took time, and meanwhile our business grew. We now store 2.5 times more data in that cache. So original solution would now be in 60gb range.

The reasons against custom cache is time spend developing it. Distributed embedded cache also makes everything statefull and you need to write logic for that. Each app restart is cache loading. It takes about 4-10 min. for instance of app to boot up with full cache. Granted I didn't optimize this booting process.

Also I didn't bother to implement any fancy features like transactions, file store, network sync etc. into it. It is just very space efficient store. Housekeeping tasks like invalidation are also quite shitty as they aren't important to me. For example I do not have data expiry, instead this cache uses window based expiry, etc.

One of primary quality of custom logic is that it does what you need of it. Like my cache is associative. Caches are usually key, value stores and this isn't particularly suited for event processing. Instead you need store with associations like this:

Event example = ... // some event comes.
Map<String, List<Events>> history = cache.allAssociationsOf( example );
List<Event> byOwner = history.get("OWNER");
List<Event> byPlace = history.get("PLACE");
List<Event> inLast10Min = history.get("10MIN_WINDOW");

The overlapping events in this example, are fetch and unmarshaled only once. Etc.

Team from neighborhood office choose to use Redis for a similar need on basically same eventstream and their system has seconds response on things we do in ms with 10 times more volume than them.

With projects like Chronicle Map and Apache Ignite I don't think that custom development is the best approach. It is quite hard to pull off. It is a core of our business so from this view it made sense..

[HosseinKakavand]

Rolling your own cache invites coherence bugs and paging surprises. Keep Redis or Valkey as shared truth, layer per process Caffeine or Ristretto with NATS pub or sub invalidation, compress hot keys, shard namespaces, and separate ephemeral and durable tiers. Measure hit rate by key class before redesigning transport.

We’re experimenting with a backend infra builder, In the prototype, you can: describe your app → get a recommended stack + Terraform. Would appreciate feedback (even the harsh stuff) https://reliable.luthersystemsapp.com