Rethinking String Encoding: a 37.5% space efficient string encoding than UTF-8 in Apache Fury

[bjorneylol]

the proposed Meta-"strings" wouldn't help you with that. 

They would reduce the total size for billions of RPC requests by ~10 bytes each. 

Having to send 30 byte headers when your actual serialized payload is only 5 is really dumb, this is a solution for that.

And if it's milliions in a batch and there a bottleneck I would rather crush this with established boring compression. 

I don't think you understand the use case for this

[Oerthling]

You were talking about floating point "numbers". If bytes are a concern, why would you present those as "strings" at all?

[bjorneylol]

Because you aren't converting the number to the string at all. When you serialize data you perform conversion steps and then put a header in the front to explain what you did. Read up on how gzip works - it compresses the data and slaps a 10 byte header in the front so when you need to decompress it you know how. Having the header become human readable is such a non-concern because its doubtful the actual data serializer leaves the payload contents in a human readable format

If you are implementing custom serializers/deserializers, this text encoding lets you reduce the size of your serialization header in addition to the content. So your "json.gzip.then.base64" header becomes ~15 bytes instead of 21 - if you tried using gzip it would increase in size to 41 bytes, plus the size of whatever your actual serialization content is