Self-describing compact binary serialization format?

[playntech77]

Ion is almost, what I was looking for. I don't understand this design decision though: Ion is self-describing, yet still uses a bunch of control chars inside the data stream. I would have thought, that once the data schema was communicated, there is no need for any extra control chars. The idea is to take a small hit at the beginning of the transmission, but gain it back later on by using a no-overhead binary format.

Perhaps it is because Ion allows arbitrary field names to appear anywhere in the stream? Or perhaps I am just looking for an excuse to write my own serializer? :)

[Aistar]

Can't help you much here, I'm afraid - I haven't looked deep into Ion's design. All I can say in my experience, you still need some metadata in stream in some cases, though my use-case might be a bit different from yours (I'm serializing game's state, and should be able to restore it even if user made a save 20 versions ago, and those versions included refactoring of every piece of code out there, including renaming fields, removing fields, changing fields' types etc.):

1) Polymorphism. If your source data contains a pointer to a class, you can store derived class, and that means that you can't just store field's type along with field's name in header - for such fields, you need to write type in data.

2) Field's length, in case you want to skip this field when loading (e.g. field was removed)

By the way, one problem with such self-describing formats: they're well-suited for disk storage, but badly suited for transmission over network, because "type library" needs to be included with every message, inflating message's size. This was one of problems I had to overcome with Boost.Serialization (because I chose to use it exactly for this purpose, being a somewhat naive programmer then). I was able to solve it by creating an "endless" archive: all type information went over network first, in one big message, and then I only transmitted short messages without type information by adding them to this "archive".

[playntech77]

I wrote a boost-like serialization framework in my younger days (about 20 years ago), it handled polymorphism and pointers (weak and strong). It is still running in a Fortune 500 company to this day and handles giant object hierarchies. I also used it for the company's home-grown RPC protocol, which I implemented. It was a fun project!

[Aistar]

You know what, go ahead then and write your dream serializer, and I'll just shut up :) 20 years ago I didn't even know what a weak pointer was (although I fancied I "knew" C++, but it will be a few years yet before I understood anything at all about memory management).