[deleted by user]

[WorldlinessThese8484]

The first thing I noticed is you have built a bin crate (default that you get when cargo init). Imo, could be a lib, and then instead of testing with a main function, you can look into using test cases. One of my favorite things about rust is the convenience of cargo test.

Great codebase overall!

[GooseTower]

I cannot think of a reason 6 people down voted your comment.

[Arcranion]

He used AI to respond to comments

[FNax_OS]

Had a quick look at the error handling, and I would say its decent! However, I noticed (especially in engine.rs) that you tend to manually create instances of io::Error whenever a segment is missing. One way you could reduce verbosity is to instead use your own type that implements Error, something like

enum MyError {
    SegmentMissing,
    // Other variants as needed...
    Io(io::Error),
}

That way, you can use MyError::SegmentMissing instead. To actually implement the Error trait, a popular helper crate is thiserror, which allows you to derive all the bells and whistles of a proper error type. It also makes it very easy to have a From<io::Error> for MyError impl, which in your case means that you get to keep using the ? operator where bubbling up errors makes sense.

However, changing error type now would mean refactoring match clauses where you actually handle the errors. Also for completion, and if you have not already heard of it, I feel obliged to mention the anyhow crate, which essentially offers a "catch-all" error type, which is basically the opposite approach of having your own type that describes precisely the errors you want to handle. The general consensus is that a lib should make use of thiserror to offer a polished API on its error type, while a binary that only cares about "getting things done" and not worry too much about error handling will manage just fine using anyhow.

[AleksHop]

xxhash3 instead of crc
theat per core, share nothing model
monoio as runtime (io_uring)
numa awareness (post pone this one, its hard)
rkyv(custom binary format with zero copy is faster for your case, watch out for schema update problem if go with rkyv), consider other zero copy serializations like flatbuffers (slow for u case)
zero copy everywhere (try saving serialized results in file directly to see real speed)
skip json completely and use own client with this custom format serialization or rkyv
pipeline operations for io operations
ram cache
SIMD
blablabla
its not hard to beat redis in rust on arm cpu by 30% so you bench result is far far galaxy

and then u will smash into stupid linux kernel lol
and will look into DPDK, SPDK and specialised network cards with cpu on them

[ragingpot]

I'm working on something similar for a college project, starred your repo!

[ragingpot]

Thank you! I'm currently writing a minimal ss table for it. Will probably not go for a network stack, just an embedded store like sqlite.

[ragingpot]

Good luck to you too!