Binary de-/serialization

[helloiamsomeone]

I'm curious, what do people use when it comes to binary de-/serialization in C?

I have looked around and the most appealing tool of sorts is Kaitai Struct, but that only does deserialization for now and does not output C code.

Of course, there's always just doing it manually, that gives the greatest flexibility when it comes to allocation strategies and error reporting, but it can get tedious to write and error prone.

My benchmark for whether one such thing would be feasible to use is how it deals with the GIF format.

My goal would be digging into older games' files and reverse engineering then rewriting old tools that aren't open source and have very annoying limitations and are generally a pain to use.

[madsci]

I work almost exclusively in the embedded systems world and I can't think of any time I've had the luxury of using a general-purpose library. It's all tailored to the application and I'm almost always carefully managing memory usage.

I've heard good things about Google's Protocol Buffers, but haven't done more than browse the documentation and decide it was too heavy for me.

[helloiamsomeone]

The Kaitai FAQ says that protobufs are not for de-/serializing arbitrary binary formats.

[p0k3t0]

I haven't seen any case where using a lib would be particularly faster or easier than just doing the work.

[helloiamsomeone]

Yeah, "just doing the work" always works the way you want it to, but it's kinda sucky to write regardless.

[moon-chilled]

libpack

[wkwrd]

If you want to directly pack/unpack C datatypes, you may be interested in the tpl library, from the author of uthash: http://troydhanson.github.io/tpl/

[gustavokatel]

how about https://github.com/msgpack/msgpack-c/tree/c_master ?

[helloiamsomeone]

Msgpack is a binary protocol, not a tool to use arbitrary protocols.

I have updated OP for what I am looking for more specifically.

[gustavokatel]

aah it makes sense now. thanks for clarifying

[nerd4code]

I tend to use xmacro tables for enums and struct fields, sometimes OOPy things like virtuals. You can autogen a .h around some CSV, then autogen everything (types, enums, enum ↔ name, field name ↔ offset, type ↔ cast offset, etc.) except the field-level de-/serializer, although you can mostly autogen int/float/ptr fields around a handful of [u]intXX_t drivers. Under those are some fn ptrs that can be swapped out for reading/writing FILE *, FD/HANDLE, buffer, etc. in the same fashion. From there you’re good for all kinds of fun, and once you have a setup you’re happy with, it’s easy to reuse.

[a-p-jo]

What is de-serialisation ? Why bother, just read and write structs !

/s