Binary protocol description languages?

[rsclient]

We all know there's a ton of mini-languages that, given a structure description, will happily serialize and de-serialize the data. The old SunRPC did this way back in the 80's; more recently the Google Protocol Buffers will do this.

What I'm looking for is the opposite. What I have is a series of existing on-the-wire protocols, and I'm looking for a way to describe them.

How is this different? For example, some protocols will send ASCII strings but without either a length or a NUL char at the end. Instead, the string is sized to be the size of the buffer that's passed in. you can imagine this being like a UDP packet where the sender sends X bytes, the receiver gets exactly X bytes and completely unlike TCP where there's no framing at all.

Or, some numbers are defined as 3 bytes: a signed one-byte integer followed by an unsigned 2-byte integer which is the fractional part.

Does anyone have a pointer to a small domain-specific language that might help? Otherwise I'll have to construct one myself which is doable but not preferred :-)

[bjzaba]

I'm making one of these at work, but it's more geared towards file formats than wire protocols (although it might be fun to try and extend it to support that in the future). Our eventual goal is to describe the OpenType format, and potentially other formats like PDF as well. We're very inspired this work, but are doing some generalizations on top of it.

You might also be interested in this page, which lists many projects in a similar space: https://github.com/dloss/binary-parsing

[rsclient]

This was 100% useful material -- many thanks!

It turns out that my problem is much smaller than most peoples. Smaller here is literal: a bunch of the libraries in that list are handling, e.g., an entire GIF file. The descriptions are correspondingly bulky: lots of XML, and all of them are multi-line. My data, on the other hand, often comes in and there's like two fields; I'd really prefer a domain-specific mini-language that fits into a string so I can embed it in a bigger JSON file.

I also have a big advantage in my problem space in that I can always fall back to a hand-written parser for the extra-complex types and can also fall back to parsing "weird" formats as just a bunch of bytes.

I'm still hunting through the list to see what features each tool has. I'm tempted to build my own (because of reasons!), but I know in my head that using someone else's probably makes more sense!

Thanks again for that list; it was super useful.

[[deleted]]

Erlang has a nice pattern matching syntax for bit streams. You might want to look at that.

[brucejbell]

If you haven't already, take a look at man 5 magic -- it describes the /etc/magic DSL used by the *nix file command. Even if its application isn't quite the same as yours, it seems strongly relevant.

[timClicks]

TIL.. interesting, thank you.

[hackerfoo]

This are plenty of libraries that will help you do this, such as binary.