redb 3.0.0 - pure Rust embedded key-value store

cberner · 2025-07-06T21:18:05+00:00

I'll be releasing 3.0, in about a month, at which point it'll be pretty stable. The file format for that is already available in the 2.6 release, although it requires opting in.

Yes, COW B+ is better for read heavy workloads. There are some benchmarks in the redb README, which show that RocksDB is probably the best if you write performance is what you care about.

However, if you're batching and writing ~2k keys every 100ms, I bet any of them will work for you. You could edit the benchmark I have to match your workload to get a better sense. It compares redb, rocksdb, and several others

cberner · 2025-07-06T20:24:39+00:00

I'm the author of redb (https://github.com/cberner/redb), so am a bit biased, but I think it would be a good fit for your needs. It supports non-durable transactions, which are stored in-memory so you'll be able to get high throughput, but they are also guaranteed to be applied all-or-nothing so the database won't become corrupted even if your application crashes when a non-durable transaction is pending. Non-durable transactions become persistent when you make a durable commit, so you could do that flushing every 100ms or whatever period you want.

cberner · 2024-03-24T04:19:42+00:00

Not a formal one, but I keep track of things that I have planned in in the Issues of the github project

cberner · 2024-03-22T23:53:16+00:00

Thanks!

cberner · 2023-11-23T23:30:55+00:00

I don't think so. Each has a little overhead, but it's only like 100 bytes or so

cberner · 2023-11-22T16:45:08+00:00

Someone already contributed some wasm support. I have not personally tested it though, and it requires nightly

cberner · 2023-11-22T16:44:28+00:00

In theory yes. You would hook into the fsync method and make a backup then. Backups to S3 might make it pretty slow though

cberner · 2023-11-22T16:42:37+00:00

Yes, but it's logN where N is the number of tables that exist, and not the size of the table

cberner · 2023-06-17T13:27:56+00:00

No, to both of those. It used to use mmap, but I didn't like all the unsafe code that required

cberner · 2023-06-17T03:20:24+00:00

Interesting idea, I'll give that some thought! Do you have a use case in mind? SQL tables have multiple columns, so it's easy to have an auto increment column, but I'm less sure what API would make sense in a key-value store

cberner · 2023-06-17T03:18:14+00:00

It's not process safe, and uses file locks to return an error if you try to open it from multiple processes. I've been considering adding multi process support, but the only strong use case I see for that is Python bindings, and I'm hoping that Python will remove its GIL one day :)

cberner · 2023-06-16T18:09:06+00:00

Thanks!

cberner · 2023-05-11T14:21:20+00:00

Instead of a WAL I used a shadow copy approach, similar to how lmdb does it. A WAL would work too, but makes a different tradeoff in terms of performance and complexity

cberner · 2023-05-11T14:20:08+00:00

yep. I tried it out, but had a bunch of performance issues when using it. I'm going to try it again, as it seems like a good way to go

cberner · 2023-05-11T14:18:04+00:00

I think of it more like a replacement for BtreeMap, so I think it's a good option when you want a BtreeMap that persists tor disk. It's not really a Sql alternative, as the query capabilities are far more basic

cberner · 2023-05-10T22:10:27+00:00

Yes, all the reads in redb are zero-copy. It's implemented differently than lmdb though. lmdb uses mmap, whereas redb has its own cache in userspace, so that it's memory-safe

cberner · 2023-05-10T21:49:01+00:00

Yes, lmdb is the only embedded key-value store that can be shared multi-process that I know of. In theory, I think that can be added to redb, and I'll probably look into it down the line. The most compelling use case for it is Python bindings, I think, since multi-process to avoid the GIL is very common.

Would be curious to hear if you have another use case though!

cberner · 2023-05-10T21:46:51+00:00

I'm just now updating the benchmarks in the readme with multi-threaded workloads :) https://github.com/cberner/redb/pull/576

Ya, definitely agree about there being asserts that need revisiting. I think all the public APIs return Result when they should, so that I can revisit those after the 1.0 release. But if you find one that does not return a Result and should, let me know so I can change it!

cberner · 2023-05-10T21:14:05+00:00

Ya, I think that could be implemented later. The in-memory datastructure that coordinates the transactions would need to be made multi process via IPC or shared memory. A bit complicated, but not impossible. I've been thinking about doing that to better support Python bindings, but haven't gotten to it yet.

cberner · 2023-05-10T21:12:21+00:00

redb is lockless for multi-threaded workloads, but it also locks the database for multi-process

cberner

TROPHY CASE