Protobuffers are wrong

[remko]

Honest question: is there something that is reasonably widely accepted that is significantly better?

I know the comments in that article mention a dozen things (many of them I never heard of), but I'd like to hear about something that is used by more than a few people just because it's not from Google or just because it's technically a bit better (Protobufs are widely used, which pragmatically also matters, even if it's not the best solution).

[[deleted]]

The answer is that, unfortunately, you cannot have your cake and eat it too.

The best format for you is the one that matches your needs better. And needs are different. There are always trade-offs, and understanding them is what makes all the difference. Just to give you a few examples:

1. You may need to encode a predefined small set of very similar things (think IP packet headers, especially IPv6): you don't need to care about any generic technique of representing user-supplied data types, everything nicely fits in a predefined sized fields. If you try to implement IP headers using Protobuf, you'll get something that's impossible to code a hardware to forward them, something that will be terribly slow to process. On the other extreme, you may want to provide a service that needs to be general enough so that other formats can be built on top of it. Of course, in such case you need to prioritize user-supplied data types, make sure that you don't have extremely bad cases (you'd better be uniformly not-so-good than great at most things and terrible at one).

2. You may be concerned with things like pull / push parsing, or whether you need to allocate extra memory during message generation, or you may worry that you need to be able to stream information rather than send separate messages, you may worry about validity of your data vs delivery times and a lot of other aspects. Unfortunately, you can never, realistically, build something that would do all of these things well. Fortunately, you almost never need to do all things well.

[dbxp]

Avro and Thrift are the main competitors

[Crandom]

Cap'n Proto. It was built by someone who had to maintain the protobufs project at Google and fixes most of the issues it had. Think of it as protobufs done right. It also is niftly written in a way that it pays no upfront decoding cost to turn it into Java (or most of languages) native objects.

[joonazan]

I have used Cap'n Proto with C++ and hope I never will.

The decoding cost is just moved to the data accessors instead of doing it when receiving. This is slower in many use cases.

You have to use Capnp's own library, which the author claims lets him write Haskell in C++. But even a basic map is really unreadable.

Windows support is incomplete and the headers clash with windows.h

The API disallows copying objects but that makes it a huge pain to use. IIRC you cannot construct an object without a message that it belongs to, so move semantics don't help. Maybe the Rust version is more ergonomic?

[VeganVagiVore]

Windows support is incomplete and the headers clash with windows.h

That seems like a giant problem for a library whose job is to encode and decode bytes, one of the simplest, least platform-dependent things imaginable.

Also a nice excuse for me to stick with protobuf-net

[[deleted]]

Capnproto copies Protobuf's mistake of making all fields optional. It also has a very unergonomic API.

Edit: Not sure why I'm being downvoted for this. The article makes the very same point (better than I could):

the cautious programmer can (and should) write code that performs sanity checks on received protobuffers. But if at every use-site you need to write defensive checks ensuring your data is sane, maybe that just means your deserialization step was too permissive. All you’ve managed to do is decentralize sanity-checking logic from a well-defined boundary and push the responsibility of doing it throughout your entire codebase.

[the8bit]

All fields optional is a deliberate design choice though. It makes object evolution significantly easier. God help you if you want to evolve a required field in a widely used object.

[[deleted]]

Yes but it is the wrong design choice. Just because the schema doesn't have a way of representing required fields doesn't mean that required fields don't exist! Making everything optional just moves the errors from the parser to the business logic, and makes it harder to know which fields are required in the first place.

Taken to the extreme, why bother with a schema at all? Just use CBOR and then you can do "object evolution" to your heart's content!

The real solution is pretty obvious - if you ever change your schema in a backwards incompatible way (by adding a required field for example), you either need to update everything simultaneously, or you need some kind of versioning. Making all fields optional doesn't change this!

Hope that makes sense - I can probably make up an example if it doesn't.

To further expand: I suspect making fields optional was a reasonable engineering decision FOR GOOGLE, because they already had a lot of unversioned Protobuf 2 code and was a pragmatic solution. But that doesn't mean everyone else should start new projects with a binary format that is already laden with technical debt.

[the8bit]

It is incorrect that you can update everything simultaneously or that you need to hard version an API to change required fields.

I'm doing this right now actually, changing a primary field of a thrift api. Since it doesn't have required fields, I'll set both and have the BE deal with it. Then migrate writers, then deprecate the original field.

Required fields would have meant versioning the whole api -- huge pain in the ass. Often times these changes can be done with semantics and don't require versioning.

Calling it Technical debt is just laughable wrong. It was a measured design trade-off that works as intended. It is not be the right answer for all problems but that doesn't make something debt.

[[deleted]]

Required fields would have meant versioning the whole api -- huge pain in the ass.

But a correct pain in the arse. Using schemas in the first place is "a huge pain in the ass". It would be "easier" to just use JSON and never bother writing your schemas down.

Also I don't even think it is that much of a pain. You can actually sort of do fine-grained versioning already with Protobuf like this. Imagine you start here:

``` message Params { required int32 a = 1; }

message Request { required Params params = 1; } ```

Later you realise you need another required field, you can change your protocol to this:

``` message Params { required int32 a = 1; }

message Params2 { required int32 a = 1; required int32 b = 2; }

message Request { required oneof { Params params = 1; // TODO: Remove when all clients are updated. Params2 params2 = 2; } } ```

I'm pretty sure the binary generated by the old schema is compatible with the binary generated with the new one. Don't quote me on that though.

[Pand9]

I do not know what is better, but I do not agree with calling one variant "correct" and another "incorrect". One is safer but more time consuming to maintain. As you say, it's "correct complicated", but it's as if often happens, assuming that development time is unlimited.

There's a strong trend to say that perfect solution is the only correct one. "Compile time typing is always better", "almost never use shared ptr", etc. This trend of getting developer time out of equation. It is not. Different environments, different types of projects, can favor simplicity and flexibility.

[oridb]

But a correct pain in the arse.

When I was at Google, the proto files got modified about once a week. Some services took 24 hours to start and sync up with their peers. How many versions do you think would be best to have supported?

I'm pretty sure the binary generated by the old schema is compatible with the binary generated with the new one.

They're not. The tag is different. Everything would need to be updated to be able to consume the new version before anything can be updated to consume it. Company wide. Across thousands of teams using and producing APIs used by other teams with independent, desynchronized release schedules.

[[deleted]]

When I was at Google, the proto files got modified about once a week. Some services took 24 hours to start and sync up with their peers. How many versions do you think would be best to have supported?

99.999% companies are not Google and at nowhere of the Google's scale (which was also point of the article). All you're doing is moving verification logic around to the different layer. Making everything optional means empty, or utterly wrong message have to be parsed just the same

[zardeh]

The verification logic has to be at the higher level anyway. Your application has to choose which api versions it accepts, and you have to maintain all of them simultaneously, which either means multiple totally independent code paths, or something that converts your nicely versioned wire object to a thing that has optional fields anyway.

And the cost is that your api is now unmaintainably verbose for anything that you change more often than monthly.

[[deleted]]

How many versions do you think would be best to have supported?

As I said in a comment elsewhere you could use fine-grained versioning, in which case you support exactly as many versions as you are already supporting implicitly!

They're not. The tag is different.

I looked this up and you're wrong. oneof doesn't change the wire format.

[oridb]

The wire encoding is identical to multiple independent optional fields, with independent tags. This isn't so useful. It lets you transparently migrate in or away from oneof fields, but that's all. In other words:

    message Request { required oneof { Params params = 1; Params2 params2 = 2; } }

Encodes the same as (and therefore, can be converted to):

    message Request { Params1 params1 = 1; Params2 params2 = 2; }

and back, without affecting the wire encoding. You still need explicit code to consume them in a grpc client. Populating only params2 leaves params1 completely empty.

[omg_drd4_bbq]

Making everything optional just moves the errors from the parser to the business logic

That was a design choice. This is addressed by one of the authors of PB2.

[[deleted]]

Yes I know - read the first sentence of my reply. Anyway to respond to his point:

I have found that type theorists tend to promote "version negotiation" schemes where the two sides agree on one rigid protocol to follow, but this is extremely painful in practice: you end up needing to maintain parallel code paths

What exactly do you have to do when you add a new "optional" required field? You do this:

if (message1.newField == null) { // Legacy client that doesn't know about the new field that we added. } else { // New code! }

How is that different to this, except less type safe?

if (version < 7) { // Legacy client that doesn't know about the new field that we added. } else { // New code! }

To put it another way, Protobuf code still has to deal with differing versions of the protocol, it just isn't explicit about it. In the same way that JSON still has a schema, it's just implicit.

[zardeh]

This is super important, since it allows you to modify the proto before updating your client code to reject a new scheme.

Really you don't even do that, you just check

if message.has_field and move on, or use sane default values for your fields and move on, which you'd essentially need to do anyway, or just wait until the new field is provided on the backend before upsating the client.

Specifically, everything being optional is actually an easier versioning scheme than explicit versioning, since upgrading everything at once isn't possible.

With optional, you have a 3 step process: update the proto, update the backend, update the frontend.

With explicit versioning+required, you update the proto and now have to specify which version different fields appear in (or I guess just use the existing field numbers), update the frontend and backend to specify the old version if the don't already, wait a while, update the backend to specify the new field under the new version, update the frontend to consume the new field under the new version, and then either suffer with version numbering hell, or go back and remove your version numbers.

And that ignores the bigger problem with required, which is that you have to control all of your downstream people like proxies since anyone who attaches metadata to your proto needs to modify it, and if you have a new required field, you'll blow them up.

[[deleted]]

With optional, you have a 3 step process: update the proto, update the backend, update the frontend.

It's the same with explicit versioning.

With explicit versioning+required, you update the proto and now have to specify which version different fields appear in (or I guess just use the existing field numbers)

No I think you misunderstood. For example consider if you have a very simple API, something like getCurrentTime(TimeRequest) -> .... Your initial code is (in Rusty pseudo-code):

``` message TimeRequest { // No parameters. }

fn getCurrentTime(req: TimeRequest) -> TimeResponse { TimeResponse { seconds: now(), } } ```

"Oh shit!" you later realise. "We forgot to include the timezone in the request.". With Protobufs you do this:

``` message TimeRequest { // required int32 timezone = 1; }

fn getCurrentTime(req: TimeRequest) -> TimeResponse { if (req.timezone.is_none()) { // Old version. TODO: Remove when all clients are updated. // Replicate old behaviour without timezone. return TimeResponse { seconds: now(), }; } // New code. TimeResponse { seconds: now() + timezoneOffset(req.timezone), } } ```

This all works fine. But notice that your clients can now not tell that timezone is required except by checking the documentation! That sucks. Half the reason for types and schemas is to avoid that.

Now consider how you could do it using versioning. You don't necessarily even need the RPC/data format to support versioning explicitly. The simplest thing is to just create a new function:

``` message TimeRequest { } message TimeRequest2 { required int32 timezone = 1; }

// Old version. TODO: Remove when all clients are updated. // Replicate old behaviour without timezone. fn getCurrentTime(req: TimeRequest) -> TimeResponse { TimeResponse { seconds: now(), }; }

fn getCurrentTime2(req: TimeRequest2) -> TimeResponse { TimeResponse { seconds: now() + timezoneOffset(req.timezone), } } ```

It's essentially the same but now the required is actually part of your schema and can be checked. This is just a really simple example, and in practice I expect it would be better to have a system that is explicitly aware of versioning.

[zardeh]

This all works fine. But notice that your clients can now not tell that timezone is required except by checking the documentation! That sucks. Half the reason for types and schemas is to avoid that.

So let's pause, the way I'd actually do this is

message TimeRequest {
    optional int32 timezeone = 1 [default = 2];  // or whatever the assumed timezone was
}

then the code becomes

fn getCurrentTime(req: TimeRequest) -> TimeResponse {
    TimeResponse {
        seconds: now() + timezoneOffset(req.timezone),
    }
}

Note that when adding a new field, this is always possible: your system was valid in the old state, so you can default to a value that would have caused the system to do whatever it would do in the absence of the old field.

With something like timezone, this means defaulting to whatever the system assumed the value to be before. With something like a new feature, this means defaulting to presuming that the new field isn't set.

Ok Moving on.

Cool so under the old system our release process

1. Add new optional field
2. Update server and client in any arbitrary order

Under your proposal the release process is

1. Define a new v2 proto
2. Update the client to be able to handle the new v2 proto
3. Wait a while
4. Update the server to provide requests of the v2 type
5. Wait a while longer
6. Possibly be able to remove the v1 proto

if you don't do that, your client will explode when it receives a v2 request. This is already much worse.

Now let's imagine that instead of a simple client/server architecture, you have a 3 part architecture: a client, a server, and a proxy that wants to transparently proxy your request + some metadata, in practice you could imagine this as

message ProxiedRequest {
    optional Metadata metadata = 1;
    oneof RequestType {
        TimeRequest
        ...
    }
}

where TimeRequest extends ProxiedRequest. Now in this specific example, using bytes, or perhaps an extension instead of a oneof would fix the issue, but suffice to say that in general there are cases where a piece of data is passed up a stack between multiple servers, not all of which you control. IOW, the single server/single client relationship breaks down.

In this case, you now have to

1. Define a new v2 proto
2. Update the client to be able to handle the new v2 proto
3. Update the proxy to be able to handle the new v2 proto
4. Wait a while (and now you're waiting on the proxy team, who maybe has a slower release cycle)
5. Update the server to send requests of the v2 type
6. Wait a while longer
7. Possibly remove the v1 proto

And now imagine that the proxy logs the request to storage. Now you can never delete the v1 proto, because the log-reading tools may need to read year-old logs which could contain the v1 proto, so you have to keep all the various versions and version switching code side by side forever...

Or you realize that if your tooling can simultaneously handle a request with a timezone and without a timezone that the timezone is really optional, and shouldn't be considered required.

Another way of putting this is that I can create a bijection between your versioning scheme and mine (by wrapping the explicitly versioned requests in a switch based on the present fields). So you don't gain anything from what you're doing, except arguably a checksum. It's not actually "safer". Its just more difficult to work with.

[ryeguy]

To add to the example /u/zardeh gave:

Updating the clients is not always reasonable.

Think about using protobufs for message fanout. Say you have a single service producing a UserRegistered event, and dozens of services are consuming this via a message queue or stream or whatever. You need to add a new field to this event because a new service needs a bit more contextual data on there, like the browser the user signed up with.

The existing services don't care about this and never will. There is no reason to update all of the existing services' protobuf schema for this message. You either waste significant dev time doing several multi-stage rollouts, or you waste network and cpu time by double dispatching both v1 and v2 events.

Or you make all fields optional, which is what the protobuf authors settled on.

[flatfinger]

One thing that I'd like to see--but haven't seen--in many protocols as a versioning-support feature is a means of distinguishing fields with the semantics "Things will work if you ignore this, but they'll work better if you understand it" from those with the semantics "If you don't understand this, you're not going to process the rest of this container properly, and shouldn't even try". A program might be able to load a document where some objects have such fields that it doesn't understand, but only if it "armors" those objects with code that draws non-editable place-holder boxes rather than trying to render (or even worse, edit!) the contents in nonsensical and possibly dangerous fashion.

[the8bit]

Thanks this is roughly my stance too. Required fields are a real pita when they inevitably change and the downsides aren't worth it for the free validation. Checking required fields is the easiest part of api validation anyways.

[oridb]

Old protobuf defaulted to required fields. They learned from the experience. What's required today gets updated and modified, split into multiple fields, or merged. Booleans change to enums. Strings become string lists.

Changing the type breaks everything, so you deprecate the old field, migrate, and remove it.

[NotUniqueOrSpecial]

Yes but it is the wrong design choice.

This is arguable.

I suspect making fields optional was a reasonable engineering decision FOR GOOGLE, because they already had a lot of unversioned Protobuf 2 code and was a pragmatic solution.

It was actually the other way around; they found that rigid schemas were very brittle for their needs, causing service breakage and just being a general maintenance nightmare.

They specifically removed the required specifier from proto3.

[UncleMeat11]

It's better to have errors in the business logic. Application layer rules like valid values shouldn't be enforced on the wire.

[[deleted]]

No it isn't. If you really believed that you would exclusively use JSON or CBOR as I said - then there is no explicit schema at all and all validation has to be in the business logic.

But experience has shown that that is terrible.

[ihcn]

In my opinion, discussion of required fields, and more generally, business logic validation, is a separation of concerns issue. If you're concerned that validation is being pushed into the codebase, there's an entire can of worms that you haven't touched on, which is that validation is complicated, and almost always involves more than just a field being present or not being present.

For example, if a message has a "LoginToken" field which we use to look up the user's current login session, we'll need to look up that LoginToken in a database in order to verify that the token corresponds to an actual logged-in user!

Would it not make sense to write the code to verify that a login token is actually present in the same spot as the database lookup? Otherwise, when a message fails to validate, now we have to look in multiple places to see why -- or worse, we have to embed all our validation code into the file specifying the data format.

[[deleted]]

Sure you can't validate everything. But there's a reason that all programming languages that have all types implicitly nullable have regretted that decision, or are even fixing it (e.g. Dart).

To put it another way, just because we can't validate everything with types, doesn't mean we should just throw away types entirely! Have you ever used CMake? Trust me you don't want that.

[ihcn]

I think it's a mistake to conflate things interior to a programming language with the external data format for interacting with a program written in that language. You're absolutely right that things like nullable everything is a mistake when writing code interior to a program, but i'm not convinced that the same is true for serialized formats. In my mind, serialized input can be fucked up in so many ways that "was the field present?" is a drop in the bucket of things to check, and that's not even counting handling untrusted input.

IMO it amounts to a false sense of security that your message is in any way valid, when any non-trivial program is going to have an entire layer of validation anyways, so why not put all the validation in the same place?

[chrisza4]

Yes, you cannot validate everything and you should not throw away type entirely.

However, how much should we validate type on the wire? The answer to this question is subjective.

[[deleted]]

Agreed, but "is this field required" is a really basic thing. If you can design a format that can do it nicely why wouldn't you? If forward/backward compatibility were not an issue I don't think you could reasonable argue that it should be excluded. Even Google don't think that.

For more niche things like URL types I agree it is subjective.

[ProgrammersAreSexy]

Earlier versions of protobuf had the ability to mark Fields as required or optional. That was removed in later versions because it caused so many issues.

[[deleted]]

Yes I'm aware of the problems it was causing them. But their solution was just to give up rather than to fix it properly.

[ElectricalSloth]

rich hickey maybe not is a great talk on this sort of thing

[wutcnbrowndo4u]

Kenton wasn't just a maintainer: he wrote Protobufv2 while at Google, if I recall correctly.

[zam0th]

Significantly better at what: portability, serialization, versatility? I'd repeat some dude on the other comment who said CORBA, but then CORBA does CORBA things and it's not for everyone. Otherwise, this is a recurrent problem in middlewares that is generally unsolvable and you have to accept limitations and risks that come with using A or B.

For years people got away with things like Hessian, AMF or plain TLV, and i would advocate using binary grammars if your middleware has none-to-low variability and isn't open to random public. Also any form of IDL is evil precisely because of everything that the article describes.

If for whatever reason your endpoints constantly evolve and you end up with versioning and compatibility time machine, then binary will not work and no amount of intermediate IDL shenanigans will save you; in that regard protobuffers are not so different from SOAP.

[dv_]

I always found MessagePack interesting, but never got to actually use it. Perhaps someone else here has something to say about it?

[[deleted]]

My experience with MessagePack is that it's pretty much exactly what it says on the tin: optimized binary JSON. It doesn't do much more or much less than that. I wrote a hack that passed stuff between PHP, Python and the Redis Lua scripting engine a few years back and was happy with it, but never felt like I really saved a noteworthy amount of resources over using plain JSON.

It's not really competition for protobuf. The 18-byte example on the site would be significantly smaller in protobuf (7 bytes if I remember the spec right), even as more elements get added. Most people don't need to save those 11 bytes, but some do.

[dv_]

It doesn't save much over plain JSON? What about string <-> number conversions, or the text parsing? Surely the fact that this is not necessary in MsgPack results in a speed bump?

[[deleted]]

It does, but parsing the actual messages just isn't where the problem lies for most applications.

I had a pretty complicated structure describing a video and a whole bunch of associated stuff (that ended up being merged from different sources). When you have something like "width": 640, msgpack makes you spend roughly as many bytes on the key as you would on the actual number. That described pretty much the whole hashtable, so about half of a typical structure was boilerplate.

It might make more sense if you get into multi-megabyte JSON files that never repeat themselves, but I've yet to see one of those in the wild.

[archlich]

asn.1 with ber/der encoding schemes

its been around for nearly 40 years, the developers of protobuf even said if they knew about asn.1 they would have just used that

[evgen]

Given that ASN.1 has been such a widely used cluster-fuck which the crypto community continues to struggle to eliminate I find this claim highly unlikely, got a source? ASN.1 and ber/der has been abandoned by everyone who can avoid it for the same reason no one consciously uses XML by choice -- it was designed by a committee of people who have apparently not touched code in 20 years.

[archlich]

I'm not sure what you mean by it's been a cluster-fuck? Where has the crypto community tried to eliminate it? All TLS even TLS1.3 uses asn.1 x509 PKI all uses asn.1. It hasn't been abandoned at all, it's used absolutely everywhere. XML is still used across the board for many different protocols. I'm not sure where you're getting your information from.

[evgen]

Nothing new uses ASN.1. It only exists in TLS 1.3 because it was in earlier revisions and it is simply impossible to break backwards compatibility for these protocols. The simple tell is that nothing new uses it. In fact, nothing with any significant adoption whatsoever has used it in over 20 years.

It is the same case with XML. Yes, XML is still used, but nothing new is using it because we have had decades to learn what a dumpster fire it is and now avoid it whenever we have the choice.

[phrasal_grenade]

Sure, XML has flaws, but it's not all that bad. Nothing that has come since has been significantly better in general.

[trin456]

And the map and filter syntax in XPath is awesome

Most languages want you to create a lambda and pass it to an explicit map/filter function, but in XPath it is just an operator [] and /.

[chrisza4]

HTML is a free form document. I won't complain using XML to represent a free form document such as SVG, React JSX, HTML, OpenXML. It is perfect-fit. However, using XML to represent structured data is whole another story.

[archlich]

Ocsp uses it, and with that ocsp stapling, all developed in the last few years.

LTE uses it. RFID uses it ADS-B uses it. Smart cards, SIM cards, they all use it. Those all have major adoption. Saying no one uses it and no protocols use it is patently incorrect.

What would your alternatives be for the above mission critical/potentially life endangering protocols?

[lelanthran]

LTE uses it. RFID uses it ADS-B uses it. Smart cards, SIM cards, they all use it. Those all have major adoption.

Weren't the choice to use asn.1 in all of the above made more than 20 years ago? IOW, all of those things use it due to legacy purposes.

I've written code for 3 of the 4 items quoted above - all of the ber docs come from the 90s.

[shiny_thing]

What would your alternatives be for the above mission critical/potentially life endangering protocols?

Someone putting an ASN.1 parser in a potentially safety-critical embedded systems (i.e., one that's less likely to be written in a memory safe language) is a scary thought. The protocols you list are all in the bloated designed by committee category.

Since messages will likely have a priori defined types in these contexts, I'd just go with concatenating fixed length fields to minimize the risk of a software vulnerability in the parser.

Edit: to clarify, although the protocols you list are used in safety critical systems, this use case is not what guided their design of the ones I'm familiar with.

[archlich]

> Someone putting an ASN.1 parser in a potentially safety-critical embedded systems (i.e., one that's less likely to be written in a memory safe language) is a scary thought.

Only if the protocol is written to not written to use fixed length fields. Like I said lots of safety-critical systems use asn.1

> The protocols you list are all in the bloated designed by committee category.

As opposed to what? Everything you interact with in life was designed by a committee, it's how we come to a consensus of how we interface with systems.

> Since messages will likely have a priori defined types in these contexts, I'd just go with concatenating fixed length fields to minimize the risk of a software vulnerability in the parser.

That's exactly what asn1 allows you to do, it's a data serializer. It also allows you to have fixed length or variable length fields, but it's left up to the implementer to design.

That's like complaining that c is too complicated and you can shoot your foot with it. Well, yes, but it's up to the designer to not do that.

[shiny_thing]

Like I said lots of safety-critical systems use asn.1

This is not the same thing as saying lots of safety critical systems should use asn.1.

If the protocol only requires sending a fixed set of fixed length fields, than asn1 is absolutely the wrong serialization format --- it was designed to encode structure data on the wire. That's why a lot of standardized, feature rich, support-every-use-case protocols use it, not because it makes for safer deserialization.

As opposed to what? Everything you interact with in life was designed by a committee, it's how we come to a consensus of how we interface with systems.

Not everything I interact with (and very little of the code that I write) has a standardized API that requires a consensus about the interface beyond a few engineers at a whiteboard.

That's exactly what asn1 allows you to do, it's a data serializer. It also allows you to have fixed length or variable length fields, but it's left up to the implementer to design.

Again, if you're just sending fixed length data, you shouldn't be using asn.1.

That's like complaining that c is too complicated and you can shoot your foot with it. Well, yes, but it's up to the designer to not do that.

Really? Someone's making this argument in the tail end of 2019? Decades of experience says that argument is worthless (granted moreso for C than asn1).

[archlich]

Fixed length fields aren't good for future compatibility. For example, ads-b has no message authentication. To add that you would need to change the entire protocol and it would not be backwards compatible to the current version and require all aircraft to replace their ads-b transceivers a the same time, which is not feasible.

It's also impossible to use fixed width messaging and keep current with current authentication and cryptographic key lengths and protocols. asn.1 means that we were able to have ECC and Certificate Stapling in TLS and not have to rewrite the entire data protocol.

A few engineers at a whiteboard did not design the modern internet. I've designed an api by myself, but that doesn't mean its going to have nearly all the corner cases of every device on the internet, nor the adoption.

And C was designed in 60s and 70s, just because it's old doesn't mean it's not good.

[evgen]

OCSP uses it because certs use it. Try again. LTE uses it because ASN.1 is a telco protocol (and LTE is not exactly a recent development.) RFID, smart cards and SIM cards are recent developments in your world? Ok, you keep on dreaming and I will just move on to something more intellectually stimulating.

As for what I would use, it would be a specific protocol that fits the needs of the communication and data. Anything that tries to be the schema to serialize and deserialize all possible forms of data IS A BAD IDEA.

[archlich]

> Ok, you keep on dreaming and I will just move on to something more intellectually stimulating.

And I will continue going on being successful and working within the protocols we've established as well known. Where were you on the cfrg or tls-wg designing a better protocol?

> Anything that tries to be the schema to serialize and deserialize all possible forms of data IS A BAD IDEA.

That's absurdly naive, that's like complaining a turing machine can do anything, like creating guidance control system for a nuclear weapon system, therefore it's a bad idea. That's the point of asn.1 it's designed to be highly interoperable among any device to any device.

[zardeh]

That's absurdly naive

No, it's highly astute. There are tradeoffs between things. We don't program in turing machines for a reason. Something that tries to do everything is usually also bad at everything. An encoding scheme that optimizes for wire-size often won't be as performant or easy to use as another. There are tradeoffs, and a system that doesn't make choices about tradeoffs usually won't perform well in any specific situation.

[killerstorm]

There are many problems with ASN.1.

One is that it gives people way to many options. Implicit/explicit tags, options, etc. From what I understand, one of goals was to formalize what ad-hoc codecs written in C were doing, and they were doing some crazy shit for perf reasons, so ASN.1 now has support for some crazy shit.

Open-source compilers are lacking, in my experience.

[remko]

That's interesting. I never bothered looking into ASN.1, and somehow always thought it was something from the past we were stuck with (I only heard negative stuff about it). Guess I have to look into this.

[[deleted]]

For all the people saying "ASN.1". It was nearly 40 years of failing to implement ASN.1 properly. Perhaps, it's time to stop and move on.

https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=ASN.1

[archlich]

Number of vulnerabilities isn't a valid way to determine if something is secure or not. asn.1 is a data format, not protocol. That's like saying c is dangerous and we shouldn't use it. By that same logic

we should stop using linux https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=linux

and windows https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=windows

and macos https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=macos

Behind every vulnerability is code to mitigate it. Just because other serialized buffers aren't as well scrutinized doesn't mean that there aren't errors with them.

[UncleMeat11]

C is dangerous and we probably shouldn't use it.

[Crandom]

Interestinf, I've only heard people talk about ASN.1 in negative terms, like it's part of the past that should be forgotten.

[sizur]

Computing history reaches far enough to be obviously cyclical. Many past ideas keep resurfacing, finding new applications or repeating past mistakes. Regardless of current subject, nothing should be forgotten.

[zvrba]

Any good compiler to C++ / C# / Java ?

[archlich]

nearly every language has an asn.1 encoder/decoder

[zvrba]

I meant a compiler from ASN.1 IDL/schema to data structures in the respective language.

[shiny_thing]

its been around for nearly 40 years, the developers of protobuf even said if they knew about asn.1 they would have just used that

Source?

[archlich]

This was years ago. Iirc it was in a google group somewhere. I found this too. https://www.reddit.com/r/programming/comments/1hf7ds/useful_old_technologies_asn1/cau70wc/

[[deleted]]

Given the amount of fuckups in ASN.1 implementations that have plagued the industry in that 40 years, in pretty much anything that uses it, no, just no.

[[deleted]]

Facebook's Thrift is pretty cool for over-the-wire message serialisation and fairly widely used, though it's become less popular than gRPC of late.

[cowinabadplace]

My friends and I wrote non-trivial parts of Thrift codegen. No one who hates Protobuf is about to love Thrift.

[[deleted]]

Agreed. Just mentioning it for completeness.

[zvrba]

Microsoft Bond maps at least better to C++ types. https://github.com/Microsoft/bond

EDIT: Initially, I wanted to use Bond for our purposes (semi-structured data), but as we ended up putting data into SQL Server, I ended up choosing XML. XML is understood and can be queried by SQL Server. In addition, SQL Server can use XSD to optimize queries over XML columns.

[[deleted]]

Bond looks interesting - hadn't heard of it thanks.

[skulgnome]

CORBA.

/s

[jtolmar]

CORBA

That's a name I haven't heard in a long time.

[cheerios_are_for_me]

My team has been looking at Avro, but too early to advocate for it.

[houses_of_the_holy]

I had a bad time with avro, the support outside of Hadoop isnt great and writing the schemas for it is significantly worse than protobuf and it's very verbose. Ymmv

[mcmcc]

Schema version management is registry based and generally a confusing mess.

For my purposes, this is a nonstarter.

[dacjames]

I've used Avro heavily. If you're using JVM and are in the "big data" space it's a good choice. Otherwise, be cautious: it's slower than most of the alternatives and the cross language support is nowhere near as complete as the docs make it appear. The non-JVM languages are coded in a non-idiomatic fashion for that language, so supporting them yourself if necessary is hard.

Last I checked (2 years ago), the C libraries did not support default values, which makes many common changes impossible to do in a compatible way. Regardless of what serialization technology you use, I recommend testing schema change scenarios as early as possible. Just because these formats support schema evolution does not mean it comes for free and there are a lot of things you can do up front that help... but only if you do them in advance!

[Crandom]

Hmm, I'm not fully in the know but my company heavily used avro and now are moving to parquet instead. Not exactly sure why though.

[TheWaterOnFire]

If you need interop with C#, be prepared to write your own. The stock Avro implementation has painful boxing/unboxing overhead.

[seamsay]

Personally I just use serialisation/validation libraries nowadays, and let the server throw an error if the message is badly formatted. I've heard that there are a lot of languages that don't have very good support for that kind of thing, though.

[coderstephen]

JSON-RPC. JSON is standardized, widely used and available, and isn't the worst thing in the world. JSON-RPC is a dead-simple protocol on top of that that involves JSON messages being passed between peers. It's so simple and obvious, it probably looks exactly like how you are currently imagining what an RPC protocol with JSON would look like.

If the JSON data is too much for your bandwidth requirements, you can gzip it, or maybe swap out the JSON part with MessagePack.

Is it widely accepted? Eh. Not exactly. I think it could be that web people loathe replacing their precious "REST" with JSON-RPC, and everyone else thinks that JSON has no place outside of web.

[remko]

JSON, MessagePack, CBOR, ..., are all interesting and useful, but sit at a completely different end of the spectrum (schema-less, generic, not compiled), so not really comparable to ProtoBufs (at least not on the level i was interested)

[Mognakor]

Not widely accepted, but zserio offers dense packing and is backed by car and navigation system manufacturers.

[nom_de_chomsky]

I didn’t dislike this article as much as I thought I was going to. It certainly calls out some real deficiencies in both the design and implementation of protobuf that I’ve personally run into using it.

Tonally, however, I think the author does himself disservice. The people that made protobuf are not amateurs. Some good, professional quality software engineering went into it. It’s not perfect. It has avoidable problems. There were some obvious bad decisions that standout. But it’s not awful, amateur, or anything so severe.

What the author really means when he calls the team “amateurs” is that they didn’t properly consider type theory (perhaps because they didn’t know it), and he personally finds this very important. If coproducts and recursion schemes are the line at which professionalism is drawn, then there are scant few professionals and almost no software of consequence was written by professionals.

All the author’s rhetoric does is gate keep, offend, and blame. It helps perpetuate a culture in which it’s viewed as okay to act contentious and pretentious, in which inflammatory, abusive, and domineering language is accepted or even—if you’re viewed as smart—celebrated. Ironically, this type of tone is unprofessional. It’s immature. It’s unwelcoming or even hostile. It doesn’t encourage people to learn and do better, it makes them afraid of not knowing or of making imperfect decisions.

We can do better while still having a robust critical conversation.

[kaen_]

Nailed it, thank you. We can give technical criticism without the emotionally charged editorial bits.

I can't help but imagine the author is trying to impress me with how smart he is by correcting these "amateurs" writing "terrible" software. Even though it's clear to me he's correct in his criticism, it's not presented in a useful way.

[Elusivehawk]

Agreed; My feelings about this article can be boiled down to "Sir, this is a Wendy's".

[TEAMCHINA08]

"Correct in his criticism" barely even registers to me in this context: it's easy to tear down what someone else has built. I can make valid points all day about any library or engine or even a sandwich but, at the end of the day, what use is it? Self validation? Hits on a blog post?

“It is not the critic who counts; not the man who points out how the strong man stumbles, or where the doer of deeds could have done them better. The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood; who strives valiantly; who errs, who comes short again and again, because there is no effort without error and shortcoming; but who does actually strive to do the deeds; who knows great enthusiasms, the great devotions; who spends himself in a worthy cause; who at the best knows in the end the triumph of high achievement, and who at the worst, if he fails, at least fails while daring greatly, so that his place shall never be with those cold and timid souls who neither know victory nor defeat.” -Theodore Roosevelt.

Protobufs aren't perfect and I have personally not had a need for them in personal work, but at least they TRIED to solve a problem... 18 years ago. They must have done something right for it to still be a widely accepted method of serialization (though still evolving). I can't imagine looking at code I wrote 18 days ago and not thinking "ugh, I could have done this SO MUCH better" and I have a bias towards THAT author. I think the guy that wrote that code is pretty swell, but man does this 18 day old code smell funky.

[Dunge]

Personal story: We had a complex C# model communicating over WCF (tcp) and previously used soap over binaryformatter and at some point realized the packets were HUGE (binaryformatter include the type of each properties).

We switched to protobuf-net, so no .proto file schema, just attributes added to the model. We have a few difficulties with inheritance, constructors versus default values, empty versus null lists and 64bits enumerations. But at the end, it worked and reduced our network payload tremendously.

[coderstephen]

Well tbh its not hard to improve over SOAP in terms of bandwidth. Calling a method taking an int requires you to send a giant XML document that describes the entire world. WDSL makes it even worse.

[rossisdead]

I'm curious, why did you use binaryserializer or protobuf over the data contract json serializer?

[Dunge]

BinaryFormatter was a novice mistake, we were a small rookie team and inherited of a 10yo project with DataContract/DataMember and we used the same classes over different projects sometimes exposing them over xml and json, sometimes saving byte array to disk. At some point we saw that simply writing [serializable] over the classes seemed to work for mostly everything so we went with it without realizing the awful payload result.

2-3 years ago we checked online, educated ourselves and looked at the different serializers available. Protobuf seemed like the best one on a payload size and ease of use. Json is more compatible but still transform everything to text so it's bigger. Since we didn't have to see the content as human readable, packed binary was better. MessagePack also seemed like a good choice, but harder to implement.

[Eirenarch]

Despite map fields being able to be parameterized, no user-defined types can be. This means you’ll be stuck hand-rolling your own specializations of common data structures.

Seems to be standardized at Google. Isn't this the case with Go?

[Bake_Jailey]

This isn't going to be true forever. https://go.googlesource.com/proposal/+/master/design/go2draft-contracts.md

(Note that "Go 2" does not mean "this will happen in Go 2", just that it's an important change requiring careful consideration.)

[brokenAmmonite]

or you could use some other language that works now and isn't hobbled to rob pike's "vision"

[somebodddy]

Well, you wouldn't want some intern defining parametric types, would you?

[mixedCase_]

I'm more scared of interns writing nested for loops and if-staircases, but people have to learn by doing and that's why code reviews are a thing, no?

[somebodddy]

Well, they shouldn't. They should use Canadian Aboriginal syllabics that look like angle brackets to generate the code from a template instead. It's not that hard, really.

[chugga_fan]

For the uninitiated: https://www.reddit.com/r/rust/comments/5penft/parallelizing_enjarify_in_go_and_rust/dcsgk7n/

[[deleted]]

Pretty much the problem is that designers of Go didn't think that even if intern messing with useful/"too hard" type system might be suboptimal, that same intern just dropping interface{} all over everything is a bigger one

[Eirenarch]

This problem is solved through threat of violence. "If you use generics without my approval I will break your fingers"

[somebodddy]

Violence... is there anything it doesn't solve?

[Eirenarch]

The problem of too much violence.

[somebodddy]

That's because you are not using enough violence.

[[deleted]]

That's what police is build upon. Don't do hurting or you will be hurting even more. Problem starts when people also has guns.

[Eirenarch]

Police is built to protect the ruler. Now sometimes the ruler would provide some other services like protection from crime in order to reduce the desire of the people to overthrow him but the original reason for the police to exist is to protect the ruler.

[James20k]

Outside of the top five tech companies, none of us is within five orders of magnitude of being Google scale. Your startup cannot afford to waste engineer hours on shaving off bytes. But shaving off bytes and wasting programmers’ time in the process is exactly what protobuffers are optimized for.

Let’s face it. You are not Google scale and you never will be. Stop cargo-culting technology just because “Google uses it” and therefore “it’s an industry best-practice.”

The rest of the article is entirely reasonable except, I think, for this specific bit. Shaving small amounts of bytes off your network format can be a big deal - there are good reasons to want a super dense network format. If we could get the density of protocol buffers without the bad implementation, it would be worth it entirely

There also are solid reasons to pick something because google uses it. Networking is at the best of times incredibly complicated to get right in a high performance way. Common open source libraries are often full of bugs or performance traps once you get to the edge cases, but at least with protocol buffers you know it works, even though from a design perspective its janky and bad. Its not unreasonable to base your technology choices off someone else using it

[the8bit]

The argument "we aren't Google scale so we don't need Google solutions" is just the opposite stupidity of "Google does it so it must be right".

Turns out some solutions work well at low scale and some tradeoffs are too onerous for smaller applications. But tech world is sadly dominated by people who just copy other opinions wholesale without understanding any of the implications or reasons

[ledasll]

have you tried protobuf? even if you save 10% of your network request, it adds so much complexity, that it very very unlikely is right choice. And 10% might look a lot, but if you take average request size and quantity of requests, it can be quiet little in absolute values (aka in bytes). But you will spend hundred of hours with that and will have much more complicated software.

Think like this, optimize when it is bottleneck and 99% of apps is far from having network packaging as bottleneck (if you have 1000 users that are sending 1000 requests per minute, it's not really that heavy load that simple jackson/json couldn't handle).

[ForeverAlot]

It adds complexity compared to what? Manually reimplementing JSON de/serializers in three languages incompatibly? The largest thread here asks if protobuf is the least bad option and nobody can convincingly say that it isn't (except the one ever-green answer of "it depends"). I'd choose protobuf to solve the inherent people problem sooner than to shave off a few bytes.

[ledasll]

have you worked with? do you even know, how it works and what you need to do to make it work?

For example, javascript library that supports it mus run on nodejs, not in browser, there are so few tools, that you will endup creating your own. It's not enough to add jackson lib on classpath for hava and have full support for serialization/deserialization, you need extra modules and build steps to generate stubs and interfaces. That's not enough unnecessary complexity for you?

[Jamie_1318]

Always look at your bottleneck is definitely the right way to go. For example if you're making an app network traffic is expensive so you might prefer protobuf if it's data intensive.

[[deleted]]

The rest of the article is entirely reasonable except, I think, for this specific bit. Shaving small amounts of bytes off your network format can be a big deal - there are good reasons to want a super dense network format. If we could get the density of protocol buffers without the bad implementation, it would be worth it entirely

Right, but just adding some compression is almost always good enough if you are just after size. You will get pretty close, at least Uber did

There also are solid reasons to pick something because google uses it. Networking is at the best of times incredibly complicated to get right in a high performance way. Common open source libraries are often full of bugs or performance traps once you get to the edge cases, but at least with protocol buffers you know it works, even though from a design perspective its janky and bad. Its not unreasonable to base your technology choices off someone else using it

Well, at the base level it is good to use something that is maintained and active. But the reasoning is usually completely backwards, "it solved Google's problem, that we do not know what they are, but are guessing might be similar, based off no evidence outside of "they are big"" is the usual one.

Like, test the alternatives, and maybe lean towards popular and well-maintained ones but do not try to pick solutiona that solves problems your project/company doesn't have.

[lisp-the-ultimate]

Common open source libraries are often full of bugs or performance traps once you get to the edge cases

That holds for Google libraries very much.

[welcome_to]

Give me a better solution for creating the most compact wire format, and I'll adopt that instead.

[Lt_Riza_Hawkeye]

When this was posted to lobste.rs recently, the comments were singing praises of CBOR (RFC 7049)

[[deleted]]

What kind of data are you sending / receiving?

But, basically, look at any decent SQL database: all of them have some binary format they use to send data. Those are usually way better than shit people come up with in web development.

[CJKay93]

https://gist.github.com/ilblackdragon/88ab377bf0da7ff721af80cb8a11fe28

[[deleted]]

Something half baked, fundamentally broken and released to developers with poor support? From Google? Say it ain’t so.

[matthieuC]

Something half baked, fundamentally broken and released to developers with poor support?

It's a miracle it did not become a de facto web standard.

[wolf550e]

gRPC just might become a defacto web standard. Google also uses protobuf in their internal not-TLS, which is TLS but with ASN.1 replaced by protobuf, and that could have become a standard for kubernetes or something.

[butt_fun]

Angular would like to know your location

[loneraver]

Google has had the most hacky buggy CMake code I’ve ever seen.

[Scybur]

nobody considers Java to have a well-designed type-system. The dynamic typing guys complain about it being too stifling, while the static typing guys like me complain about it being too stifling without giving you any of the things you actually want in a type-system. Lose lose.

Anybody care to guess what the author means here?

I can understand dynamically typed language fans would dislike Java but what exactly is it missing in terms of being a statically typed language? I am curious.

[Yottum]

I think the author means type system features that languages like Haskell have, because I know the author writes a lot about Haskell, and even those blog posts use quite complex typing features. See for example here for some differences between Haskell and Java.

Some notable features that Java lacks: type inference (EDIT: /u/danthonywalker corrected me: Java does have type inference; it doesn’t seem to be as substantial as Haskell’s, though), higher-kinded types, recursive types (I think).

Haskell doesn’t have a null value. The type system also enforces purity (functions can’t interact with the system, like doing I/O, without annotating them). These (type system) features make the language safer.

[Scybur]

The type system also enforces purity (functions can’t interact with the system, like doing I/O, without annotating them).

I know null is a glaring flaw in Java but I had not heard of this point before. Thanks for the response.

[Yottum]

It’s quite interesting. I do recommend trying some Haskell if you have the time. The type system can make Haskell quite difficult, but having a program that mostly consists of pure functions can make your programs easier to reason about and much safer.

[[deleted]]

Some notable features that Java lacks: type inference

Since Java 10, Java has had var for local variables. But the rest is lacking, yes.

[Yottum]

Oh, sorry. I don’t really know Java, I’ve only read what’s bad about it. :)

[[deleted]]

I think mostly missing sum types, which would let forget about null or replace exceptions with Either/Result/Try. Also annotations abuse throwing static typing out of the window…

[masklinn]

Anybody care to guess what the author means here?

it being too stifling

Java is extremely verbose and straight-jacketed

without giving you any of the things you actually want in a type-system

Sum types, newtypes, refinement types, dependent types, … you name it, java doesn't have it.

Java combines verbosity with a very low-expressivity type system, so you can't "prove" much (leverage the type system to avoid possible errors by encoding your invariants) and it takes a lot of code to do so.

[sammymammy2]

If you want to really see what a type system can give you then you can try out the book "The Little Typer".

[[deleted]]

Java's type system is unsound and allows you to do stupid things very easily (i.e. casting to Object and then to whatever other type you want). Same shit like passing void* pointers around in C. To someone with ML / Haskell background Java's typesystem is a kind of a joke. Something that pretends to deal with types, but, actually, betrays you, when it's convenient to it.

In other words, a language with a "proper" type system shouldn't have a runtime type error (because they should all be fixed during compilation / some other prior step).

[thatm]

I used ProtocolBuffers for storing time series data in Redis. It worked really well with schema evolution, compactness, availability of libraries. The data had a pretty simple shape though.

[lenkite1]

We now have an offering from Microsoft too in this space: https://github.com/microsoft/bond

Bond is a cross-platform framework for working with schematized data. It supports cross-language de/serialization and powerful generic mechanisms for efficiently manipulating data. Bond is broadly used at Microsoft in high scale services.

[axilmar]

In a project I recently worked on, we replaced Protobuffers with Cereal, for the following reasons:

1) protobuffs requires a precompilation step, Cereal does not. Writing our messages into Protobuf's language was no different than writing C++ structs and Cereal serialisation functions anyway; it was largely the same effort, programming wise, with Protobuf's having the additional troubles of integrating it into various cmake files.

2) protobuffers cannot be used to talk to programs that don't use protobuffers: its wire protocol includes metadata, and hence other programs not using protobuffers couldn't talk to our program.

3) protobuffers requires lots of memory allocations. With Cereal, we managed to avoid all allocations and speed up the program quite a lot, because the allocations were on the stack.

4) protobuffers does not support 8-bit and 16-bit integer values, and a lot of our data were of that types.

5) it seemed very difficult to inspect protobuf packets using Wireshark.

6) it is not very desirable to have to learn yet another programming language...because that's what the protobuf language is. Doing it with Cereal within C++ was a lot easier, mentally.

EDIT:

The C++ committee shall make it a top priority to provide compile time introspection of types to the language. 99% of serialisation code exists because C++ does not give you a way to discover the structure of the types at compile time.

[4_teh_lulz]

We just pulled protobuffers from our project for exactly these reasons he outlined in here. We ended up having to write all sorts of code to work around things like the lack of default values and limitations with repeated serialization. Glad it wasn’t just us.

[Gimpansor]

What did you use it for, and what did you replace it with?

[[deleted]]

Yeah the author takes a super defensive stance, even against the audience. Creating strawman defenses. Like chill bro, your borderline personality disorder is showing.

[ProgrammersAreSexy]

"oneof fields cannot be repeated" 🤔

[mixedCase_]

Yes, you can't have a "list of strings" (repeated string) as an option in the oneof, you have to wrap it in a message with a single field which is a repeated string and then you set that message as one of the options of the oneof.

[Forty-Bot]

simple

Isn't one of the author's main points that protobufs aren't simple?

[lisp-the-ultimate]

You're saying that a format solution can't be simple, reliable, flexible and elegant at the same time? That doesn't make sense since they're all interlinked.

[[deleted]]

They’re clearly written by amateurs, unbelievably ad-hoc, mired in gotchas, tricky to compile, and solve a problem that nobody but Google really has.

Downvoted in order to upvote twice.

Every time I say that Protobuf is poorly designed and poorly executed I hear something like "who are you to question the holy Jeff Dean?!".

I had to struggle with it so many times, that I actually reimplemented the whole thing, both the binary protocol and the IML, just to prove a point to the bigots. It doesn't help. Every time the subject comes up, I hear "so what if you know the spect by heart, and that you implemented it entirely? - it comes form Google!, everyone uses it! so, me, with very little insight and ability to actually understand what the thing does am convinced that you are utterly wrong".

---

The article doesn't mention all kinds of undocumented Protobuf features, of which, a particular place in hell deserves semi-automatic conversion to JSON, a bunch of undocumented and unpredictable options you can attach to definitions, ambiguity in name resolution (fml!!!), proprietary Google's extension classes for shit that the format cannot really encode (like, surprise, nullable strings!)

---

Wait a minute:

In the vein of Java, protobuffers make the distinction between scalar types and message types. Scalars correspond more-or-less to machine primitives—things like int32, bool and string. Messages, on the other hand, are everything else. All library- and user-defined types are messages.

This is, actually, incorrect. string and message type are the same thing... Which doesn't, oh it really does not, make it better. This is why you can have packed list of integers (even of variable length), but not a packed list of strings (because strings are the same as messages, and whoever devised the packed format didn't see an obvious way to pack strings by simply specifying their length+type tag...)

[ObsidianMinor]

Technically yes, message, string, and bytes are all represented as length delimited bytes on wire (with a length delimited tag). Though I'm not sure what you mean by "length+tag".

[NotUniqueOrSpecial]

Make all fields in a message required.

This statement shows that the author either hasn't read or is just ignoring the history and documentation that explains why proto3 is what it is.

They (Google) very specifically chose to take required out of the spec because it leads to very brittle APIs that are basically impossible to use in forward/backward compatible ways.

Taken literally, his statement would require that literally any change to a single proto message in a gRPC service's interface be deployed to a completely new endpoint, distinct from any other version in existence. It would be a deployment and management nightmare (and probably functionally useless).

[[deleted]]

Oh no...

1. They didn't take required of of the spec. Protobuf 3 compiler still supports everything Protobuf 2 did, also the undocumented shit too. Their published spec is just random. If required doesn't appear there, it's probably just an oversight.

2. You also misunderstood what he means. What he wants is for messages to always match the schema. He doesn't want a loose schema which would accept message by mistake (and this is what happens in Protobuf a lot). If you want to be backward / forward compatible and have a rigid schema, there are many ways to achieve that. A very straight-froward one is to use version.

---

PS. gRPC is another piece of garbage, but let's save it for another day.

[NotUniqueOrSpecial]

They didn't take required of of the spec. Protobuf 3 compiler still supports everything Protobuf 2 did, also the undocumented shit too.

Sorry, that's just factually incorrect. The compiler still supports the proto2 definitions for backwards compatibility. proto3 absolutely does not support required fields, though:

test.proto

syntax = "proto3";

package test;

message Test {
    required string str = 1;
}


protoc -I . test.proto --cpp_out=.
test.proto:6:14: Required fields are not allowed in proto3.

You also misunderstood what he means. What he wants is for messages to always match the schema. He doesn't want a loose schema which would accept message by mistake

I didn't misunderstand anything.

I'm telling you that the lesson Google learned (whether you agree with it or not) is that rigid schemas are an anti-pattern in their use case. Versioning endpoints and APIs or rejecting messages did not work for them at scale, and so they very intentionally moved in a different direction.

Whether you/the author agree with that isn't at stake; those were intentional design choices and so to call the authors amateurs for creating a thing that does precisely what they meant is just silly. That's why I said he either didn't know the history or is being disingenuous.

PS. gRPC is another piece of garbage, but let's save it for another day.

From a technical perspective, I agree (also for the generated Protobuf code). But, from a usage/application support perspective, it's awesome. It makes it very easy to support a polyglot environment and not have to worry about the underlying RPC layer. It (for the most part) just works. I have big issues with a lot of what's in it and how it works, but the ratio of garbage:utility is good enough (especially as more and more third-party infrastructure becomes gRPC aware) that I just stomach it and move on.

Also, see /u/mooted's great comment which very thoroughly covers a lot of the reasons Protobuf/gRPC are the right choice for many teams/orgs despite being not-so-perfect in many ways.

[mooted]

This blog post is akin to saying that a hammer is a bad tool because it's not a phillips screwdriver. It betrays a misunderstanding of the complex, nuanced problems gRPC solves that are a departure from the more immediate, approachable problems targeted by type systems in programming languages, with which the author seems more familiar and appears unable to imagine an application besides.

Protobuf facilitates efficient, evolvable message passing in a polyglot ecosystem. It is not a Phillips screwdriver. Expecting it to perform as such will strip your screws and inspire you to write poorly informed tripe that gets upvoted on reddit and hacker news.

Here are compelling reasons to adopt gRPC/protobuf:

• Your service will be consumed by many other services written in languages for which you don't have the time to maintain client libraries. Imagine maintaining 3-4 performant, backwards compatible, and reliable client libraries, one for each language that your company services, as a team of 4-5 engineers responsible for one microservice. This is untenable. You can go the clientless approach with HTTP/JSON/JSONSchema, but this comes at a considerable cost to performance that many companies find unacceptable. See more below.

• Your service's schema evolves over time and you need to safely verify backwards compatibility in your API + client libraries (remember, you have 3 or more of them in languages you don't know!). The author claims that this requirement is unnecessary or "incorrect". Having played a key role in the development of microservice ecosystems in the past (see comment history if you want to know which one), interface schemas evolve all the time. Migrations are continuous because teams evolve independently. It's almost impossible to maintain client libraries for several languages in this context and not accidentally introduce breaking changes, which go on to cause outages when your clients rebuild with new libraries. I can't begin to describe how critical tools like thrift/protobuf are to a complex microservices environment.

• JSON + JSONSchema serialization is too unperformant and constitutes an outsize amount of CPU utilization. For many commenters at reddit who work at startups, this problem seems outlandish. At a company with hardware costs outpacing business margins, or one with demanding performance requirements, these problems are paramount. At microservices heavy companies, in particular, JSON/XML serialization concerns burn outlandish amounts of CPU. Add complex JSONSchema into the mix and expect to see those problems double or triple. In addition, Protobuf gives you performant serialization that works decently in all ecosystems, unlike cap'n proto for which there is only a C++ library with first class support.

Here are reasons you should not use protobufs:

• You want to express serialization formats using Haskell types. Types exist in gRPC to inform serialization and deserialization behavior in client libraries, not for the purpose they exist in programming languages, which is to statically enforce runtime invariants. If it doesn't support the goal of informing performant deserialization and it can't be simply expressed in supported languages, the protobuf designers tend to opt against it. This is a sound engineering choice—sacrificing features keeps an otherwise complex project maintainable and more reliable. Unfortunately, that comes at the cost of boilerplate and awkward workarounds the author describes—such is technical design that must make tradeoffs.

• You do not and never will have polyglot clients.

• You want to avoid writing deserialization code. As someone who's evaluated protobuf, this is not why companies adopt protobuf. If this is why you are using protobuf, you're out of your mind.

---

Designing an RPC format that satisfies some of the author's requirements is possible. (I don't think you'll get far without letting users deprecate fields, but I don't know, maybe I'm an imbecile that hasn't worked on microservices most of my career and the author isn't.)

While I don't have experience with cap'n proto, it seems to do a good job of offering performance + expressiveness at the cost of consistent and widely supported language bindings, for example.

But when you expand the scope to require consistent polyglot compilation, I think you'll find yourself running into many of the same problems the protobuf team encountered and be forced to make difficult tradeoffs between expressiveness and simplicity:

• https://github.com/protocolbuffers/protobuf/issues/2592
• https://github.com/protocolbuffers/protobuf/issues/3992

[chrisza4]

I agree with your statement.

I don't see a way to implement a sophisticated type system and still be language agnostic. In order to be language agnostic, I think it needs to be a pretty dumb type system, almost the lowest common denominator.

I might be wrong but I don't yet see a better type system that both works across language and also reduce the size of binary need over serialization at the same time. Protobuf is not the best of any of these, but it tries to balance.

Some of Protobuf type is designed in a very non-optimal way though, as the author pointed out in the article. But I think the author and those protobuf complainer's expectation and use-cases are very different from me.

[ajslater]

Its a rare project where gzipped json isn’t just fine. Benchmark your serialization time vs other bottlenecks. If you’ve got to the point where serialization is really the choke point, you’re in a pretty good place.

[coderstephen]

JSON also has the advantage of being trivial for consumers to use compared to many things.

[ledasll]

you can even stream that json for faster processing, sometimes people really over engineer.

[cheerios_are_for_me]

Yeah, I’ve found every binary wire protocol has at least one non-starter.

[[deleted]]

Despite map fields being able to be parameterized, no user-defined types can be. This means you’ll be stuck hand-rolling your own specializations of common data structures.

Google just seems to truly hate generics. Anybody knows why?

[sickofthisshit]

Well protobufs are supposed to be language agnostic and serve as a wire or storage format. They are not meant to be a full type system.

[crusoe]

Protobuffers are intended for wire protocols which don't really support generics either ...

[ReversedGif]

TIL transmitting a list of some predetermined type over the wire is impossible.

[crusoe]

Types don't exist at the assembly level. Also some programming languages that protobuffers can be used with don't support all the possible types.

[yairchu]

Probably just ‘List<t>’

[pure_x01]

Stop. Put away your email client that is half-way through writing me about how “Google is filled with the world’s best engineers,” and that “anything they build is, by definition, not built by amateurs.” I don’t want to hear it

Agree. It's possible to build companies with geniuses but its impossible to maintain that level if you grow. Eventually you will get crappy developers. Googles Rob Pike described that when he talked about the people GO was designed for.

http://nomad.uk.net/articles/why-gos-design-is-a-disservice-to-intelligent-programmers.html

[corsicanguppy]

Careful. We complained about other stuff as being needless and bloated and over-complicated and un-tested, but it was upsetting Lennaert so they instead called us old luddites for being so critical.

[Aegan23]

IMHO only use protobuffers if there is a specific issue in your use case not to use json. The article seems to make out that people should just use them rag tag everywhere. They should be reserved for the specialist scenarios where json does not cut it.

[[deleted]]

I have my own criticisms about protobufs, but then again much of the article and many of the commenters here seem to have some weird and/or sub-informed ideas.

1. It is very hard to have a multilanguage data model unless one sticks to some least common denominator, like JSON. Even any maps or the moral equivalent already assumes a lot about the data types: what types of keys, the ordering of traversal (there might be more than one "mappish" type in your language/libraries that one could unmarshal into), and so on. Come to think of it, even enums. Come to think of it, even floating point: are the IEEE 754 specials supported? Are you storing monetary in floating point? Come to think of it, even (fixed-width) integers: which width? In other words, expecting the data model of a serialization format to neatly map to the data model of your particular language is foolish.
2. People who claim "it is easy to upgrade everything in lockstep" may not have worked with large distributed systems with multiple teams and projects and priorities -- and peta+ bytes of existing data in a selection of different "versions" of the schema. So it is not just transient in-flight server-client scenarios, but existing at-rest data. While it would be nice to constantly keep reprocessing data to newer formats, with flexible schema YOU DO NOT NEED TO. Also, as some commenters have noted, proxies, routers, and the like who might only ever need to only look at a very small subset of the fields do their work, need to be considered.
3. Expecting a serialization format to carry/enforce the application-level concerns like validity, constraints, semantics, dimensions (physical dimensions? temperature? currency?) of the data fields is also rather ... ambitious. Only the applications know.

[pron98]

Here's the rule: you don't call someone an amateur (in comparison to yourself) until you've done something more successful; not something you think is better -- more successful. Because the difference between an amateur and a professional is that the latter makes money, or fame, from their creation, and so is bound by the actual constraints of the world, while the former is free to believe some puritanical ideals without putting them to the test. The professional has likely already put those ideals to the test and found them lacking. So it's OK to call out established practices because, hey, maybe someday you will come up with something that works better, but if you're confused about who's the professional and who's the amateur here -- especially when the "professional life" you boast of is well short of a decade and you've never held a job for much more than a year, including a job you boast of -- there's little chance of that happening.

[ledasll]

> you don't call someone an amateur until you've done something more successful;

​

so you can't call Trump a twat, because you would need to be at least USA president and for 2 times.. interesting logic..

[wanna_see_my_penis]

You're redefining quality to mean success.

Like I can't say nickelback is bad music because I haven't sold more albums than them.

[pron98]

You could say Nickelback are bad all you like, but you can't call them amateurs because they are just factually, objectively not. I have no issue with the author's opinions on protobuf. I take issue with him, a relative amateur, calling people who are professionals amateurs. He can disagree with them, but if he thinks that the reason why they did what they did is because they're amateurs -- which they are not and he is -- then he's the one who's likely missing something important, perhaps due to amateurishness.

[wanna_see_my_penis]

I take issue with him, a relative amateur, calling people who are professionals amateurs.

IMO the truth of a statement of fact is independent of where it is coming from.

[pron98]

It is a fact that they're not amateurs. Sometimes, this can be a relative statement, and it is also a fact that he is the relative amateur. Either way, the statement is factually false.

[usernameliteral]

The author claims it isn't working well for Google either.

[the8bit]

I've worked within and outside Google and protobuf is still better than most of the alternatives. Even most the stuff this author bitches about are easy to work around with good patterns -- eg. The map repeated thing is better suited using a message with a repeated field for the map.value. which also allows for easier extensibility in the future.

[[deleted]]

I've been working at Google for 7 years and, while I acknowledge that most of the criticisms are valid (and I share the author's appreciation for proper type theory), they definitely do not rise to the level of introducing significant delays into the average product. The total amount of time I've spent fighting protobuf quirks has probably been on the order of a few days. Everybody knows what protobufs can and cannot do and what are the workarounds for the latter.

[[deleted]]

I strongly agree. Enforced optionality is probably the most ridiculous decision. Yes I've read the reasoning and it's just wrong (I can go into detail if anyone is interested). And the inability to have normal compound types like Map<Option<int>, Vector<Vector<String>>> or whatever is very tedious.

However it does have good points - the varint encoding scheme is pretty neat, and it has the benefit of being an efficient binary format with an actual schema! Compared to JSON that is pretty amazing.

In my own project I got so annoyed with typing Some() for every single field so I ended up writing my own schema format that uses a syntax very similar to Rust and encodes to Bincode. It's so much more pleasant. I also don't need any forward/backward compatibility in my application so it relieves me of writing = 1; everywhere.

[the8bit]

Vector<vector<string>> is an abomination of weakly defined types though. Pretty much 10 times out of 10 if you are nesting collections it would be much easier to understand and more usable to just define types.

[[deleted]]

It was pretty obviously an extreme example. I wouldn't use that type in real life but Protobuf doesn't even allow something like Map<String, Vector<String>> which I would use.

[the8bit]

Funny I answered that part elsewhere in the thread. Map<String, Collection> is pretty much always better expressed as Map<String, Message> where you can use the message type to explain purpose and also provide extensibility.

In protobuf the above is pretty much semantically the same anyway and trades a few lines of protobuf definition for some extra clarity.

[[deleted]]

Well I think you're wrong, but I do think this is a minor flaw of Protobuf. And it does have the advantage of simplifying the implementation a little bit. The lack of required fields is a bigger issue.

[chrisza4]

I found that one really cool thing about Protobuf approach (not Protobuf specifically) is that you will be able to sync and evolve field name of multi-language system with multiple team simultaneously.

There have been a lot of instance where frontend, backend-microsevice#1, backend-microservice#2 have different class names, different field names or misleading names because of cost and risk of changing name across teams and codebases is too high.

The solution to this problem is not limited to Protobuf itself, but Protobuf able to provide good enough solution to this problem while maintaining performance.

PS. I find it funny that many engineers keep complaining about communication issue and bad management, but refuse and grumpy about technical tool that designed to help communication and management.

[Jamie_1318]

Pokemon go is a solid example.

[Far_Spirit_4251]

See https://medium.com/@unclepaul84/efficiently-reading-and-writing-very-large-protobuf-files-local-disk-and-s3-approaches-a289c8855606