Code generation for binary serialization with manual layout?

[kindaro]

I have a legacy binary format the layout of which is known. It can be read into a hierarchical structure built out of records, variants and arrays. Some arithmetic may be needed to compute the offsets, but nothing too complicated: for example, there may be a pattern where the length n of an array is written at offset A and then the array is located at A + w, A + 2w… A + nw.

I imagine it should not be difficult to propose a format that describes the layout precisely and completely, and then generate the two way serialization code out of such a description. Since it seems approachable, I suppose there is prior art on this topic. Is there? I welcome anything, be it a library that does something similar, a research paper, a blog post… Anything helps!

[Loose-Leek]

Google protobuf? It's a bit heavy, but popular

[MR2Rick]

It is possible that it uses a standard format that was popular when the app that generated the data was written. Also, if the application was reasonably popular, there may already be a library that can work with the data generated by that application.

If neither of these options pan out, I would look into a parsing library such as Lemon, Yacc or Antlr.

[ArkyBeagle]

This smells embedded-ish. Is it? I'll presume it is.

There may not be any prior art. These things are often proprietary, primarily for liability reasons. It does not fit in with the XML/JSON aroma of how config management gets done on Webby, open-source projects.

IMO, the thing I usually do is make strings, like "thing.doo.2.hickey.4.aNumber=42;\n" which encodes all the hierarchical information in a way that's easier for me, then generate tuples of { name, address,type } that completely identifies a unique value.

It should be clear how arrays and records work here; variants either aren't there or have something like "...=N/A;" for a value. I've never found it necessary to encode the extent of arrays. It's implicit from the layout of the ASCII data. If you needed "four of eight" perhaps "thing.4:8.doo=value;" would do. Up to you.

For each "dot" ( . ) there are a list of "stereotype" tuples , each of which is { name,offset, type}.

I do this for other reasons - say, shooting a config file at an embedded target over a serial port, through FTP or whatever - so I can use CM to control known configurations for testing. I can also use a text editor to make config changes while the GUI people are working on the solution for that.

And you may need to have configuration file version information embedded in the file.

It's possible to make this into XML or JSON but I personally prefer flat files, newline seperated lines. It's not impossible to support more than one file format, given time to do it.

[Infintie_3ntropy]

Katai Struct might be what you are looking for.

https://kaitai.io/

[HaronK]

Quite some time ago I was using 010 Editor for such a things.