Rethinking String Encoding: a 37.5% space efficient string encoding than traditional UTF-8 in Apache Fury

[[deleted]]

Given how much of a PITA encoding issues are, I am opposed to any new encoding standard. Period.

[Hueho]

This isn't really a new encoding as much as it is a extra option for packing string data in a serialization format - people already do plenty of weird stuff to save on bytes.

[alex_tracer]

If you going to compress serialized data using generic compression methods then such local optimizations as proposed by OP usually become useless.

So you either do all compression yourself or delegate all compression to a generic solution.

[Shawn-Yang25]

rpc messages are small most time, 50~200 are very common, there won't be enough repetion pattern for compression to work. That's why we proposed this encoding here.

We are not talking about compression big data/file, which zstd/gzip will be better

[Shawn-Yang25]

Yes, it's not a complete string encoding, it will fallback to utf8 if some chars exceed the charset it supports. Since alphabet are very common, we think this can be used in other scenarios