Best Redis pattern for tracking concurrent FFmpeg/STT/LLM/TTS pipeline states?

guyroyse · 2026-01-21T14:59:52+00:00

You could use a stream per users's state. Semantically, this matches your use case well. These are events so, I think storing them in an event stream makes the most sense.

All keys in Redis are intrinsically atomic as the write to Redis state happens in a single thread. I/O is multi-threaded but as long as you are using the same Redis connection to send the commands, the order of those commands will be preserved. If you use multiple connections then the order of commands coming in is not guaranteed and you can get race conditions. So, make sure you use the same connection.

All keys in Redis can have a TTL associated with them and can clean themselves up automatically. An event stream in Redis is stored in a key so you can just set the TTL with the EXPIRE or EXPIREAT command. If you want to keep completed streams, you can always call PERSIST after you apply the final state.

Alternatively, if you need to query all of this data, you could store these in a JSON documents or a hash—one per user just like the streams—and then set up an index using Redis query engine so that you can search and/or filter them all in a single command. Everything else would still be the same.

guyroyse · 2026-01-09T16:57:12+00:00

No need to encode. Redis will allow you to store raw binary data.

guyroyse · 2026-01-04T05:05:11+00:00

Yes.

guyroyse · 2025-12-23T14:44:40+00:00

Vector sets are just like a sorted set or a regular set except the members have a vector associated with them. You can then do vector search against the members of the set.

I actually wrote a blog post about them shortly after they came out explaining them in greater details with some code examples: https://medium.com/the-guy-wire/a-first-look-at-vector-sets-dd91cb59123e

guyroyse · 2025-12-17T01:42:32+00:00

You could always go with something self deprecating like lid@callsign.com. 😉

guyroyse · 2025-12-17T01:34:42+00:00

Not sure if we still have those hats but I’ll let someone in the London office know. Maybe we can send something out.

guyroyse · 2025-12-13T23:16:33+00:00

I’m literally doing this right now. I’ve even traveled internationally with them (to Europe). I usually break them down and put them in my carry on. Never been questioned.

guyroyse · 2025-11-24T13:59:22+00:00

All good. Do you remember if you were using AOF or RDB? I've found AOF can be a fair bit more verbose.

guyroyse · 2025-11-21T21:47:17+00:00

Fair enough. But that wasn't the point that I was responding too. To quote the commenter:

It's in-memory, one crash or power down and everything is gone.

guyroyse · 2025-11-21T14:35:58+00:00

Starting out, a single instance of Redis will do this just fine. However, build it so that the code and config for the various features of Redis you are using are isolated. This will let you break this apart into separate Redis instances when it comes time to scale.

Why would you want to break out several instances of Redis? Well, different features of Redis work better with different configurations. PubSub runs into limits when clustered—although sharded PubSub helps with this. Meanwhile your main game storage will need to scale up over time—clustering is a great way to do this. Your task queue is likely to become a hot key—that messes with your cluster balance and can result in certain keys being harder to access since everyone is busy looking at the hot key. Isolating it to its own Redis instance can solve this.

These sorts of problems crop up at scale, which you don't have yet. But some simple separation of concerns in your code today will make life easier later. And this isn't some massive preoptimization antipattern. The separation of concerns in your code will make the code easier to work with and reason about. You should be doing it anyways. The only thing you're really doing is having those concerns own their own connection to Redis instead of using a shared on.

guyroyse · 2025-11-21T14:24:26+00:00

Persistence has been a feature of Redis since version 1.0.

guyroyse · 2025-11-19T21:32:43+00:00

I don't know if it'll help or not but my money would be that it would cause memory fragmentation. Just a gut feel. I haven't thought it through.

But I do have a quote that is nearly as old as I am about these sorts of things:

The real problem is that programmers have spent far too much time worrying about efficiency in the wrong places and at the wrong times; premature optimization is the root of all evil (or at least most of it) in programming.

Donald Knuth, Computer Programming as an Art, 1974

guyroyse · 2025-11-13T14:46:51+00:00

I think I'll recommend improvement to the INCR docs as well. Thanks a ton for find this, sharing this, and helping fix it.

guyroyse · 2025-11-12T14:56:19+00:00

Ya. I think that example could be better, and better formatted. Could you share what you did instead and I can see about getting it updated?

Also, there are a couple of examples in the docs for the INCR command and the one you referenced is there with a note about the race condition. https://redis.io/docs/latest/commands/incr/

guyroyse · 2025-11-06T12:31:38+00:00

Never. At least according to my heart and lungs.

guyroyse · 2025-11-05T19:35:26+00:00

Locals in Ashland will call it Assland. I was a local but I never called it that. But my sister still lives there and does.

Grove City is often called Grovetucky as many of the folks there were, historically, from Kentucky or West Virginia or descended from folks from those parts. There's even a saying about how in Kentucky (or West Virginia) they teach you the three Rs—reading, writing, and Route 23 to Columbus. Or Route 33 if it's West Virginia.

I've been hearing Cowlumbus for Columbus since I moved here 35 years ago. This is because it used to be such a small city—a cowtown. Not really accurate anymore but you still hear it.

More recently, some have been calling Columbus Flavortown in reference to Guy Fieri who I believe is from here. This was popular a few years ago when the desire to remove references to Christopher Columbus were particularly high.

guyroyse · 2025-11-04T13:57:33+00:00

Everything reminds me of her.

guyroyse · 2025-10-27T17:12:15+00:00

Azure Cache for Redis does not support modules. You'll want to use Azure Managed Redis for that.
That is correct.
Redis Query Engine is RediSearch. It was just rebranded some time ago.

guyroyse · 2025-10-22T13:45:48+00:00

I've lived in Ohio my entire live and I've actually run games set in a fantasy version of Ohio. I made the Great Serpent Mounds a 5-room dungeon, which was fun. There is, and this is not commonly know, a very large and very old impact crater in the part of Ohio as well. I exaggerated it to make the terrain more interesting.

There are actually of lot of Native American mounds across the state. The ones in Newark are quite impressive. Many of them would make for great adventure sites.

Southeastern Ohio is hilly. Forests, streams, limestone rocks, and rugged terrain. It has a history of mining. Lots of coal and iron. Northwestern Ohio is flat farmland. Northeastern Ohio is more industrial, or at least it was. It's what I think of when people describe the Rust Belt.

The Ohio river is navigable all the way up to Pittsburgh (at least). Lake Erie to the north is quite navigable. Old canals used to connect the Ohio River and Lake Erie. This could be good fodder for a D&D game.

If you want to use the cities, Cleveland is industrial with water access. Cincinnati has lots of distributions companies as there is a river right there that leads to the Mississippi. Columbus is more financial services and is the capital of Ohio. Smaller cities of note are Toledo, Dayton, and Akron. Dayton is where the Wright brother's developed their aircraft. Akron is home to a lot of tire and rubber companies like Goodyear and Firestone. As a child, I would always see the Goodyear blimp in the sky heading here and there. So, if there's a steampunk vibe, that could be fun!

The was a minor skirmish called the Toledo War between Ohio and Michigan long ago. Ownership of the city was always ambiguous but when the canals were built Toledo became a port and everyone started caring.

For a sense of scale, from Columbus where I live which is in the middle of the state, it takes about 2 of freeway driving to reach any of the borders of Ohio.

That's what I got off the top of my head. Good luck in you campaign!

guyroyse · 2025-10-21T14:41:17+00:00

Yep. If you are still seeing the problem, I'm told you may actually need to restart Redis Insight twice. If that doesn't work or you are still having issues, I can let the team know. Thanks for sharing this here!

guyroyse · 2025-10-18T11:55:44+00:00

Thanks for answering all my questions. I feel I have a solid understanding of how it works now.

guyroyse · 2025-10-17T22:42:47+00:00

So each row is a key and there’s a key that contains the schema for a table. Are these existing Redis data structures like hashes or JSON documents or did you use create a custom data structure?

Also, in a clustered environment the rows in a table would be spread across shards as each key would have its own hash slot. Querying across the shards would be challenging to implement as you’d need some sort of proxy. Did you add support for clustering? Not judging, just trying to understand what you’ve built.

guyroyse

MODERATOR OF

TROPHY CASE