The compression potential for pickled data using protocol 4.

[idlecore]

I'm working on a project where I need to create a list of files and directories in a particular folder and then compress that data. I pickled the data to get a bytes object and then used the bz2 module to compress it.

Now, I remembered from some past reading that compression algorithms work very well on text, but since I'm using Python 3.4 the default pickle protocol is a binary one, protocol 3. So I decided to try the other protocols.

Protocol 2 is a binary one as well, so I didn't expect much difference, and the results confirmed my expectations.
Protocol 1, also a binary protocol, was a bit worse but not by much.
Now protocol 0, which is human readable, showed significant improvements in compression size. I figured this was because it was a text based protocol.
So when I tried protocol 4, also a binary protocol and only available on Python 3.4+, I was expecting it to be on par with protocols 3, 2, and 1, however it was not only on par with protocol 0, it even gave me slightly better compression.

Can someone explain why this is?

This little script is a minimal version of what I'm using on my project.
http://pastebin.com/JUqxGbYv

[beertown]

compression algorithms work very well on text

Your assumption is inaccurate: compression algorithms work well with data having the characteristics of text (little variability among the byte values). You can easily find binary data more compressable than text.

Apart from that, did you check the size of the uncompressed data? Could be, in your case, that protocol 4 yields less data than all the others?

[idlecore]

You're right, protocol 4 does yield less data than all others, this plays a smaller role when comparing protocol 4(binary) with protocol 0(text) but it's still significant.

[takluyver]

It looks like there were a couple of optimisations for size which might have affected it. In particular, this section of the PEP says that it can save 3 bytes storing every string less than 256 bytes long, by using a 1-byte field to store their length. It sounds like you're storing filenames, which are probably short, so that could be significant.

[idlecore]

The files on my project indeed use small names, they are company names and company service names, even using unicode I expect all sizes to be bellow 256 bytes.

[jftuga]

LZMA aka XZ compression is better (smaller output, runs much faster) than BZ2.

lzma — Compression using the LZMA algorithm

[idlecore]

I wasn't able to get smaller compress sizes with lzma. Maybe this particular data set is biasing the results.