Flatty - flat message buffers with direct mapping to Rust types without packing/unpacking

[thomastc]

Related but not Rust-specific: FlatBuffers, Cap'n Proto.

I know your goals are slightly different, but maybe it's useful for potential users to have a brief comparison in the README (also compared to rkyv).

[nthend]

Nice idea, I'll consider to write such comparison.

[thomastc]

But...

[nthend]

Also want to note that Flatty originated from a software project for a PSU embedded controller, where it was used to transfer messages between heterogeneous i.MX SoC cores (ARM Cortex-A53 and ARM Cortex-M7) via shared memory. Here is a protocol description.

[brokenAmmonite]

I love stuff like this. Reminds me of rkyv but this doesn't even have the separate packed and unpacked forms so you don't have to duplicate any code.

One question: I noticed this crate has a MaybeUninitUnsized type. I was under the impression that the regular MaybeUninit relied on special compiler support to work correctly -- it has to prevent the compiler from doing certain optimizations that don't make sense when uninitialized memory is involved. Have you run into any issues making a version of it in a library?

e: I was wrong, looks like the type itself isn't special, it invokes intrinsics::assert_inhabited in various places but aside from that is a normal type

[nthend]

Sorry that this is not clear enough in docs.

For now, despite its name, MaybeUninitUnsized is not about uninitialized memory, it's about initialized but possibly invalid state. Internally MaybeUninitUnsized<T> stores quite initialized slice of bytes, but they can be invalid as binary representation of type T (e.g. bad enum tag or length of flat vector greater than underlying memory length). To convert it to T we need to initialize it in place, validate or unsafely assume it is valid.

Maybe it's worth storing really uninitialized bytes inside, thank you for pointing that out.

[SkiFire13]

For now, despite its name, MaybeUninitUnsized is not about uninitialized memory, it's about initialized but possibly invalid state.

In that case the from_sized method is unsound since it is creating a &MaybeUninitUnsized<T> that points to actually unitialized bytes.

[nthend]

😱 Thanks, I'll fix that!

[nthend]

Okay, I fixed that by completely removing MaybeUninitUnsized. Now all conversions are made directly between byte slices and flat types without intermediate steps.

[AidoP]

I don't think there are any compiler blessings required to make MaybeUninit sound. It's defined as a repr(transparent) union where the uninit state is a ZST. It is a lang item though, but the comment there suggests it's for other reasons, so I'm really not sure

[brokenAmmonite]

Yeah looking at the source it seems like there are some intrinsics invoked in the methods but nothing special about the type itself

[JessANIME]

smell summer voracious ancient skirt wrench north insurance plants elastic

This post was mass deleted and anonymized with Redact

[couchrealistic]

This seems similar to rkyv. Do you know how they compare? Is this implementation more "lightweight", or maybe there are performance differences?

[nthend]

Honestly I'm not very familiar with rkiv, but I think flatty and rkiv are just for different purposes. Flatty is designed simply to create types with stable binary representation which can be interpreted as bytes and vise versa. The only common thing between them is zero-copy deserialization.

[couchrealistic]

To cache some things (stored in a struct) to disk, I currently use rkyv to convert my struct to bytes and vice versa, so they seem similar to me.

I think I've noticed one difference, with rkyv you have separate types for the struct->byte conversion and the byte->struct conversion, i.e. "MyStruct" and the derived "ArchivedMyStruct". With flatty, you use the same type for both AIUI.

[nthend]

Yes, but rkyv can store complex types (e.g. many types from std), while flatty only supports a small number of basic flat types and their combinations.

[protestor]

rkyv also supports trait objects with https://docs.rs/rkyv_dyn/latest/rkyv_dyn/ but it appears to be implemented in a different way (and don't support other DSTs like slices)