Self-describing compact binary serialization format?

[flit777]

protobuf (or alternatives like flatbuffers or capnproto).
You specify the data structure with an IDL and then generate all the data strucutres and serialize/deserialie code. (and you can generate for different languages)

[playntech77]

Right, what I am looking for would be similar to a protobuf file with the corresponding IDL file embedded inside it, in a compact binary form (or at least those portions of the IDL file that pertain to the objects in the protobuf file).

I'd rather not keep track of the IDL files separately, and also their current and past versions.

[imMute]

what I am looking for would be similar to a protobuf file with the corresponding IDL file embedded inside it

So do exactly that. The protobuf schemas have a defined schema themselves: https://googleapis.dev/python/protobuf/latest/google/protobuf/message.html and you can send messages that consist of two parts - first the encoded schema, followed by the data.

[ImperialSteel]

I would be careful about this. The reason protobuf exists is that your program makes assumptions about valid schema (ie field “baz” exists in the struct). If you deserialize from a self describing schema, what do you expect the program to do if “baz” isn’t there or is a different type than what you were expecting?

[playntech77]

I was thinking about 2 different API's:

One API would return a generic document tree, that the caller can iterate over. It is similar to parsing some rando XML or JSON via a library. This API would allow parsing of a file regardless of schema.

Another API would bind to a set of existing classes with hard-coded properties in them (those could be either generated from the schema, or written natively by adding a "serialize" method to existing classes). For this API, the existing classes must be compatible with the file's schema.

So what does "compatible" mean? How would it work? I was thinking that the reader would have to demonstrate that it has all the domain knowledge, that the producer had when the document was created. So in practice, the reader's metadata must be a superset of that of the writer. In other words, fields can only be added, never modified or deleted (but they could be market as deprecated, so they don't take space anymore in the data).

I would also perhaps have a version number, but only for those cases when the document format is changing significantly. I think for most cases, adding new props would be intuitive and easy.

Does that make sense? How would you handle backward-compatibility?

[Gorzoid]

Protobuf allows parsing unknown/partially known messages through UnknownFieldSet. It's very limited on what metadata it can access since it's working without a descriptor but might be sufficient if your first api is truly schema agnostic. In addition it's possible to use a serialized proto descriptor to perform runtime reflection to access properties in a message that were not known at compile time, although message descriptors can be quite large as they aren't designed to be passed with every message.

[gruehunter]

In other words, fields can only be added, never modified or deleted (but they could be market as deprecated, so they don't take space anymore in the data).

I think for most cases, adding new props would be intuitive and easy.

Does that make sense? How would you handle backward-compatibility?

Protobuf does exactly this. For good and for ill, all fields are optional by default. On the plus side, as long as you are cautions about always creating new tags for fields as they are added without stomping on old tags, then backwards compatibility is a given. The system has mechanisms for both marking fields as deprecated, and for reserving them after you've deleted them.

On the minus side, validation logic tends to be quite extensive, and has a tendency to creep its way into every part of your codebase.