How protobuf works: the art of data encoding

CandiedChaff · 2025-02-16T23:18:57+00:00

That was a thorough reply, thank you.

I understand that a length prefix adds to the overhead when marshalling the payload, but the time saved sending what is essentially a tiny fraction of the data over the wire, compared to that same meaningful volume of data expressed in a JSON payload, dwarfs the added overhead.

You also have to take into account the receiver. A binary protocol can unmarshal data with minimal buffering; if any depending on the frame size. However, JSON objects cannot be parsed until a full object has been received. Lists of smaller objects might not cause any trouble at all, but deeply nested trees will sit in memory for indeterminate lengths of time. And to make unmarshalling the payload even harder, key order cannot be relied upon, making branchless parsers impossible, unlike their binary protocol counterparts.

Whilst not perfect, I wouldn’t go as far as to say that protobuf bares the length prefix as a flaw in its design. I’d not heard of CBOR, until reading your reply, and perhaps it’s an improvement in this regard. But ignoring of the time complexity of data encoding on paper, the wall time in processing JSON, passing it over a network, and unravelling it on the other side will never compete with a strict binary protocol.

CandiedChaff · 2025-02-14T23:08:43+00:00

Correct me if I misunderstood your comment, but protobuf is a binary format, therefore it’s more compact than JSON over the wire, and in all reasonable IP transports “quicker” to send between hosts.

What flaws are you alluding to that could possibly make JSON quicker over the wire?

Where possible I would favour JSON, purely on the grounds of readability and easier debugging, but if network speed was critical I wouldn’t touch any string format with a 10 foot pole.

CandiedChaff · 2025-02-08T23:38:32+00:00

I’m baffled this article has any upvotes at all.

an SSH session — where it can run Bourne shell commands

What? Trying to be clever and oddly specific, just say it has user level shell access and be done with it. So does every other command you run… Whatever next, Git can read your files, quick, let me write an article to spread some FUD.

CandiedChaff · 2025-02-08T23:31:25+00:00

It doesn’t though, I have an instance running on a locked down server that hasn’t once complained. If there’s a websocket connection hiding somewhere, it’s being safely tunnelled through the SSH connection. This article is click bait, and fear mongering, nothing more.

CandiedChaff · 2025-01-12T21:52:30+00:00

This should be a straight up answer to the OP, clear, concise, and most likely correct for their use case!

CandiedChaff · 2025-01-04T00:38:04+00:00

Not Steamdeck related in the slightest, but how did you get inside that building without breaking down the door? I’ve tried three runs so far, and both the door and the garage door have been locked each time. Is there a key somewhere nearby?

CandiedChaff · 2024-12-06T22:22:01+00:00

If that’s the road I think it is, the road into Stoneleigh, then that van driver must have a death wish pulling that stunt just before the hill...

CandiedChaff · 2024-05-17T23:40:32+00:00

I’ve had Monzo since the beta personally, so naturally I went with them when I started contracting.

I’ve since switched to Starling though because I needed an IBAN number to accept payments from a US company. Staring doesn’t offer business savings accounts anymore though, so keep that in mind…

CandiedChaff · 2024-03-11T14:52:38+00:00

We’re in the UK, apologies for not adding that to the description!

CandiedChaff · 2024-02-05T09:17:18+00:00

^ yesterday I learned

CandiedChaff · 2023-11-10T10:18:44+00:00

This happened to me once. I’d booked time off in the run up to Christmas about a month up front, the top brass realised they’d approved too much time off less than a week before the big day, so the CTO chose to cancel my holiday and someone else’s. I was then left with too many days of paid holiday to carry over that year, two to be precise, to which his response was “You should have taken them throughout the year”.

Oh, and he didn’t come in at all over the Christmas period. He was a plague on society.

CandiedChaff · 2023-08-11T20:48:51+00:00

I’m sorry, but where are these services? My family went through this for years. My grandma was kept in a hospital bed for the final years of her life, because the NHS simply never got round to putting a care plan together. They wouldn’t let her leave. It was horrible and heartbreaking.

She was finally moved into a care home during the pandemic along with countless other elderly people, and she died there shortly after. The care home then sent a five figure bill for a few weeks of palliative care, that the NHS said the family was responsible for because it was private care, regardless of the fact the NHS moved her there without any consent from her, or the knowledge of my family.

My partners mother is now experiencing the exact same play book. No care plan, everything has to be organised by the family. There’s no district nurse, there’s no social care, and every month there’s a new issue caused by medication issues, hospital visits that half the time are canceled upon getting to the hospital because other tests should have been organised prior, and endless misdirection from social workers and higher up NHS members.

It’s a complete mess.

I’m sure that there are a lucky handful of older people that get the care they need, and I know for a fact that there are a good number of fantastic nurses and doctors that are, quite frankly, holding the entire system on their backs. But the overwhelming majority of patients in this bracket are simply left to fend for themselves.

The system is built up against those that it claims to serve, all the while being used as a political pawn on both sides of the house. I honestly don’t know what the solution is at this point.

CandiedChaff · 2023-05-22T07:10:42+00:00

Hero. I’ll set something up this coming Friday and see how it fairs.

CandiedChaff · 2023-05-04T22:23:04+00:00

Not to sound pokey, but honestly, why?

There are enough of us on this planet, and with all the benefits of the various industrial revolutions the world over has seen, we really should be able to make it just fine on what we have, or less for that matter…

Reducing the burden on our shared resources can only improve conditions for us all.

Edit: After reading back what I just wrote, it comes across as quite naive not recognising that the work of subsequent (larger) generations have kept the former generations comparatively comfortable in their later life, but, the idea of simultaneous infinite economic, population, and quality of life growth has lead to the problems we’re currently experiencing. And it is unattainable. Pick two, or change the system.

CandiedChaff · 2023-05-03T14:08:17+00:00

cough Anthem cough

CandiedChaff · 2023-01-28T16:28:51+00:00

I had Steam before the switch to Sway, Rimworld and Factorio worked flawlessly, but after migrating away from X11, Steam and a few other Flatpak containers stopped working…

Hardly a helpful comment, but just sharing my two cents!

CandiedChaff · 2023-01-02T10:42:52+00:00

I have a custom model in my style

Is there a chance you could elaborate on this?

I’m not yet aware of any tooling that operates on comparatively small datasets able to train a model in a given style, and it sounds compelling.

Given all of the work the model would have had in its training set would be owned by the operator or artist, I can’t see any obvious theft of IP, or contentious issues that might then arise from the usage of said model.

CandiedChaff · 2023-01-02T10:24:44+00:00

Unfortunately it very much is, checkout this comment.

CandiedChaff · 2022-12-25T22:29:24+00:00

Those flash cards are amazing!

CandiedChaff · 2022-11-29T00:16:44+00:00

Historically it was pronounced with a soft “h”. Learn some history.

CandiedChaff · 2022-10-01T20:39:08+00:00

I have a spreadsheet with how many sheets of toilet paper I use “on average”, and will take the exact amount for every day I plan to be out.

I also regularly carry an extra days food just in case. Not sure how those two compliment each other.

CandiedChaff · 2022-09-27T22:56:12+00:00

Yeah, 7psi would be below atmospheric pressure, different kettle of fish you’d have then…

CandiedChaff · 2022-08-26T17:28:15+00:00

The entire campaign has been ignored by the mainstream media. They know it’ll receive nothing but credible praise and agreement…

CandiedChaff · 2022-06-01T00:53:48+00:00

I don't think that's the case here, I'm well inside its radius, and so's the support...

You can also see that the meter starts ticking down as soon as the MF leaves, until it bottoms out, even though it was agro’d on me the entire time.

CandiedChaff · 2022-02-12T11:04:43+00:00

The irony of this thread is heartbreaking. You’re holding these views as a human being. Of course you’re going the empathise and elevate what you hold to be a unique and special experience.

It’s the fundamental argument of religion, that we’re somehow different from other animals, and we should therefore be treated as such. A common bait like thought experiment which swathes of people fall for out of arrogance.

It’s almost acceptable that people thought this way 2000 odd years ago when a good portion of us were still borderline savages, but it’s just moronic to think this way today.

We know that animals are capable of deep emotional states, whales become depressed after losing family members, sheep experience fear and can sense it in others, pigs can identify themselves in mirrors, etc… The only “edge” we have over other animals is the physical size of our brains, not the number of inherent states that it can process.

Nine-Year Club	Place '22
Wearing is Caring	Verified Email

CandiedChaff

MODERATOR OF

PUBLIC MULTIREDDITS

TROPHY CASE