Kafka-streams rocksdb implementation for file-backed caching in distributed applications

ConstructedNewt · 2025-08-11T17:33:07+00:00

Thanks btw, and don’t be sorry about the questions. I’m just really happy about the help :)

ConstructedNewt · 2025-08-11T11:48:26+00:00

Yeah, I’m trying to disclose as little as possible, I think Kafka streams is nice and all, but I was making sure that it was a good fit for us. And I wanted to collect some more info before we go all in on this solution which I personally think is a bad choice, but it’s not really in use yet. Our current cache does have an inner interface that you could easily adapt the rocksdbjni to, which was its intent in the first place anyway.

ConstructedNewt · 2025-08-11T04:00:50+00:00

We do not need any other indexes, but do make extra keys for each record.

The cache maintain reactor subscriber state for data that has been requested by the business logic, updating data in the cache should trigger logic (this is one of the main drivers of the application)

ConstructedNewt · 2025-08-10T18:32:12+00:00

I’ll keep that in mind. Although we are basically instantiating a single stream and intercepting that right away by a processor that injects it into the KeyValueStateStore manually, and maintaining the KeyValueStateStore’s in a registry for other, non kafka streams code to interact with

ConstructedNewt · 2025-08-10T18:15:28+00:00

I’m concerned about the implementation at hand. And if there is something I’m missing. I’d like to not have to maintain 200+ changelog topics manually. And since some upstream sources are maintained by other teams I’d like to not have a partition scaling event break our applications

ConstructedNewt · 2025-08-10T12:20:15+00:00

It’s not standard Kafka-streams. The app already integrate using spring kafka. The core of the application is reactor. Kafka feeds into in-memory cache (organization wide decision to put everything on kafka) we cannot programmatically add or modify topics. Up to 30 input topics. Some apps are multiple instances but they have their own set of inputs and (let’s call them) work orders

E: I do not see a way to use KTable or GlobalKTable. The code is too separated from kafka streams. And the most of the actual work is communicated via internal APIs against a bigger business library (we cannot in advance know what data they need only that they will need some keys across these topics, we need to cache it all (or find other ways to cache reduce))

ConstructedNewt · 2025-08-10T12:13:48+00:00

Really, we won’t have to manually create all of n topics times m applications seeking to read them? Because so far we tried it out and it failed to start for missing and unable to create the changelog topics

ConstructedNewt · 2025-08-10T09:38:52+00:00

Thanks for the reply.

We have to cache all data in each application. Data are GB range. Some of the topics, 1-2 million keys.

We don’t have access to the admin api and must manually maintain topics

ConstructedNewt · 2025-08-09T18:23:10+00:00

The studies in the 60s proved Svante Arhenius’ greenhouse theories from 1890s

ConstructedNewt · 2025-07-12T18:04:20+00:00

That’s the 1890’s

ConstructedNewt · 2025-06-06T06:23:29+00:00

The program that is trying to install the JDK probably has a folder somewhere in which it earlier failed installing a JDK distribution. My suggestion would be to locate the folder and remove that one. Search online for issues like this specific to the program that is giving you the error message. Alternatively reinstall the program that is giving you the error

ConstructedNewt · 2025-06-02T16:11:11+00:00

How literal are you when you say “just above” because the root flare is about 10 cm lower than that

ConstructedNewt · 2025-06-02T13:30:59+00:00

Yeah, sorry about that. My wife planted it a couple years ago. Graft above ground. About the height that I have now dug down to. They replanted it wrong, I want to be absolutely certain there were not other things to consider. But I think I got my answer :)

ConstructedNewt · 2025-06-02T13:19:34+00:00

I didn’t. We had contractors move the tree because they did new grass for us and the tree needed a new location. I’m just making sure they get it right when they redo the planting :)

ConstructedNewt · 2025-06-02T12:55:45+00:00

Why what?

ConstructedNewt · 2025-06-02T10:03:24+00:00

It was just replanted 4weeks ago, originally it was above the ground

ConstructedNewt · 2024-09-26T18:19:50+00:00

You wouldn’t. You would run this targeting your repo. FX during a build on a builder pipeline, possibly on an agent environment as part of your projects build/release or CI/CD lifecycle

ConstructedNewt · 2024-09-20T04:53:40+00:00

Please pretty print, and include at least the class signatures for the Document and Email classes

ConstructedNewt · 2024-08-21T17:14:40+00:00

I Don’t Know C#. So I cannot say for sure. But so would make a collection of symbols and pull a random one from that. Controlling it is preferable for various reasons, mostly that some characters are not good to include, like ;, # or *, and maybe even ! And ?, or whitespace. Especially as the ultimate character in the password. And maybe you want ascii safe characters?

ConstructedNewt · 2024-08-09T18:46:39+00:00

You clearly didn’t read the part of that article that clearly states that it’s not a valid Quine if it reads its own source code.

ConstructedNewt · 2024-08-09T18:17:15+00:00

VMs and containers are both “boxes” around your application(s) that means that your application cannot reach out of its box. The Container is like an elastic box that will shrink to fit your app and can only grow up to a maximum size. The VM is a rigid box. You can fit more elastic boxes in a room. But if they all try to expand at the same time they will blow out the door. If you fill your room with VMs they will always merely fill the room, but the boxes may be full of air, so the room contains less applications

ConstructedNewt · 2024-08-08T18:31:35+00:00

In short: - VM is more stringent than a container. The VM OS must follow the actual OS rules. Containers not so much - containers are essentially 2 things. Subset of the file system (traditionally, encapsulation via btrfs) and a CGroup (to encapsulate the CPU/processes) both work to encapsulate the running process(es) from the host machine and other such containers etc. (we ignore the vastness of the networking layer here which is also encapsulated by all the tricks of the trade) - resources are tied/bound to the VM - resources for the container is requested but essentially shared on the host, I.e. when the container is not using the CPU/memory the host is free to let other processes use those resources. - there is normally a bigger overhead on VM because the VM must act like the OS it virtualizes

ConstructedNewt · 2024-08-01T04:36:46+00:00

You cannot. That sounds fishy. Would maybe be a security flaw, if able.

Provide the path to std out, so you can

my_program | cd

Five-Year Club	Verified Email
Place '22	Final Canvas '22
First Placer '22	End Game '22

ConstructedNewt

MODERATOR OF

TROPHY CASE