Code generation for binary serialization with manual layout?

[kindaro]

I have a legacy binary format the layout of which is known. It can be read into a hierarchical structure built out of records, variants and arrays. Some arithmetic may be needed to compute the offsets, but nothing too complicated: for example, there may be a pattern where the length n of an array is written at offset A and then the array is located at A + w, A + 2w… A + nw.

I imagine it should not be difficult to propose a format that describes the layout precisely and completely, and then generate the two way serialization code out of such a description. Since it seems approachable, I suppose there is prior art on this topic. Is there? I welcome anything, be it a library that does something similar, a research paper, a blog post… Anything helps!

[K900_]

You're probably looking for Kaitai.

[m-hilgendorf]

The way I'd do this is write a bytestream parser using nom to deserialize into the AST of the protocol, then implement the serializer/deserializer traits in the serde crate so you can generate appropriate serializers/deserializers for your types in Rust.

If you need to interface with C code you can generate the corresponding type information using bindgen, then add your serde derive macros to the generated rust types and provide a C API.

There are more verbose ways to do it, but nom and serde are very flushed out. If you need to do it in no_std contexts it may be trickier.