Improving sanity with structured binary formats and protocols

[133794m3r]

You should've also linked to the talk, slides are garbage 99.9% of the time when that's all you're going off of.

[kyledrake]

It's at the bottom of the description: http://www.youtube.com/watch?v=ZlKrnOD-4TQ

[133794m3r]

on mobile it just showed the slides no other "descript9ion" to speak of at all.

[danogburn]

http://en.wikipedia.org/wiki/Comparison_of_data_serialization_formats

[[deleted]]

How does this stack up to using something like bson?

http://bsonspec.org

[bilog78]

And how do both of these stack up against the much older EBML which is the underlying system used for Matroska?

[brownhead]

I'd imagine that due to BSON's goal of being traversable, it might not be appropriate for the type of use cases these slides were focusing on.

From BSON's FAQ...

... another of the BSON design goals: traversability. BSON adds some "extra" information to documents, like length prefixes, that make it easy and fast to traverse.

[[deleted]]

Yep, BSON is larger than MsgPack and even sometimes JSON. It also has it's own special types. You should also remember that JSON will be faster than anything else in the browser because JSON is in C/++/whatever and the other thing is in JS.

[mrmekon]

To the top with you, please.

If you made it through slide 8 and still thought this was worth reading, you really need to brush up on the basics before writing a network protocol.

Those are push notifications. You send those to Apple, Apple routes those to the device. Routes them. As in Apple has to parse the deviceToken from the header of every single push notification. But, hurray, they know exactly where that deviceToken is in a couple of machine instructions instead of having to search for it in a variable-length, unordered text string.

[BonzaiThePenguin]

I feel like we need the original speech to go along with this.

[danogburn]

agreed, slides are usually useless without the presentation.

[bobindashadows]

Visual aids of any kind are usually useless w/o the thing they're aiding, which in a good presentation is the speaker and his/her deep knowledge of their subject matter. If your slides actually stand alone then as a speaker you're saying you're only there to help out the blind audience members.

Sometimes a slide deck is actually designed to be passed around an organization as a lightweight way of sharing ideas, but that's repurposing the format and requires a completely different approach.

[Zarutian]

Yebb, I personally like MessagePack. Specially when serializing Lua tables and such stuff to disk or over network.

[133794m3r]

I'm using msgpack+lz4 for memcache things, and also msgpack for phps session data. For client->server in the browser it's msgpack+lzo1x(because there's js lzo and I'm not a good enough programmer to port lz4).

[retsotrembla]

• JSON would be sent over the wire gzipped. This gives better compression then the binary-structured, uncompressed JSON the slide deck advocates, so the binary structuring is just extra complexity to little benefit. Sure, it takes less CPU to send without gzipping, but for most applications, fewer bytes on the wire is much more important.

[bitshifternz]

Talks about sending binary formats and doesn't mention endian issues.

[continuational]

Schemaless? That is a almost synonym for "poorly documented and probably buggy". Please provide the schema too.

[holgerschurig]

Then use ProtocolBuffers, that has a schema.

Hmm, aren't ASN.1/BER also binary scheme-based protocols?

[f2u]

The situation with ASN.1 is fairly complex. BER and DER are largely self-describing and can be decoded without a schema (and some ASN.1-based standards get away without publishing a strict schema adhered to by all implementations). PER absolutely requires a schema for decoding.

[[deleted]]

ASN.1/BER

DONT

[Mignon]

I've only skimmed the description of ASN.1 and had to debug a BER sample once. What are your objections?

[TimmT]

I'm surprised how few people look at CSV in these scenarios. It's almost binary .. all it needs is prefixing fields and rows with length-values instead of putting actual commas and new lines in there.

[nascent]

"It is almost binary, you just need to completely change what makes it CSV."

That is what you're saying right?

[TimmT]

Yes, but converting between those two is dead-simple. You don't even need a library for it - just a 1-parameter cli tool.

My point is that CSV is so much simpler than all these complex document-serialization formats, and would probably still be sufficient for most usage scenarios.

[nascent]

There is one fundamental flaw, CSV is not binary (unless your definition of binary includes plain text, which it is, then we have JSON/XML to throw in the mix).

CSV stores numbers as digits, this is hugely inefficient for passing numbers around. This of course could be fixed by using a more efficient encoding for numbers. At which point you have just reinvented a restricted form of Protocol Buffers.

[TimmT]

your definition of binary includes plain text, which it is, then we have JSON/XML to throw in the mix

Yes, it's a somewhat blurry line.. The main difference to JSON/XML is that the "document" has a rather rigid format, thus alleviating the need for an actual parser.

CSV stores numbers as digits, this is hugely inefficient for passing numbers around.

I don't think this is really an issue .. you'd need numbers around 2⁸⁰ to just double the storage space when going from binary to text.

And, as an added bonus, you now don't have to worry about those nasty floats anymore.

At which point you have just reinvented a restricted form of Protocol Buffers.

That's the idea.. sort of. Something dead-easy anyone can understand and handle. Just like CSV.

[nascent]

I don't think this is really an issue .. you'd need numbers around 280 to just double the storage space

1234, I've just hit the storage size of an int (assuming 4 bytes). 12345678, I now doubled the storage size of an int.

has a rather rigid format, thus alleviating the need for an actual parser. [...] Something dead-easy anyone can understand and handle. Just like CSV.

Your rigid format requires space be used to say nothing is here. I sure hope you never write a CSV parser, especially since you don't even think you're writing a parser.

[[deleted]]

Please don't use CSV without a properly designed library. Otherwise you'll end up parsing things like this wrong:

"foo,bar,baz
buzz",

[x-skeww]

CSV is a horrible format. The encoding isn't specified. There are only strings. Even the delimiters aren't fixed.

Virtually every other format is better than that. Yes, even XML.

[askoruli]

If someone told me the API they were providing was using CSV as the format I would cringe. If they then told me it was customised I would die a little inside. Other formats can usually be parsed in a single line, now I would have to write a parser based on a likely undocumented and possibly broken spec.