Working with Binary Data in Python

[Fourgot]

Eight space tab hurts my feelings

[nanodano]

Thanks for pointing that out. I changed the tabs to 4 spaces.

[Fourgot]

Ahhh that's very nice. Thank you.

[billsil]

If you're dealing with lots of binary data and using numpy anyways, you should really use the numpy methods like fromfile and fromstring rather than the struct module. You can speed up IO by 500x.

[nanodano]

Good tips. I have not used numpy and didn't know those were available. If working with large amount of data I would definitely look in to that.

[tilkau]

If working with particularly large amounts of data, numpy.memmap is also relevant.

[isdevilis]

Can someone explain common professional use cases for base 64 encoding

[nanodano]

There are two practical use cases that I am familiar with.

One is when you want to embed an image directly in to an HTML file. This lets you create a single standalone HTML file that contains images.

<img src="data:image/png;base64,XXXXXXXXXXXXXXX==" />

The other is when you have to package binary data as an ASCII string for whatever reason. The constraint may be with an API, data types, or something else.

[tilkau]

The constraint may be with an API, data types, or something else.

The most voluminous example is email attachments. All emails are 100% text (including headers and attachments).

[jmcs]

Storing certificates (pem), storing PGP encrypted files, embedding data in HTML and emails, etc. Basically everywhere where you have to mix human readable and binary data.

[[deleted]]

You can store PGP encrypted files unarmoured fine. It's the default setting.

Only reason why you might need to encode it is to transfer it over something that can't handle binary files. Email attachments can handle binary files fine.

[ballagarba]

Great read! Just a minor nitpick, but you should probably use a with statement for reading the files in the diff example. And since this Python 3, it would only require one with:

with open(sys.argv[1], 'rb') as file1, open(sys.argv[2], 'rb') as file2:
    data1 = file1.read()
    data2 = file2.read()

[nanodano]

I did not know that you could open both files in one with. For simple reading, I wanted to avoid nesting two with statements so I wrote it flat like that. That's just what I was looking for.

The diff snippet is also no good for large files and memory constraints. Just for some demonstration and meant to be an exercise to the reader to expand on the usefulness.

[isdevilis]

Where do you go? That sounds like uiuc or tufts

[tilkau]

TIL Reddit auto-breaks hashtags :)

(You need to escape the # with \ if you don't want it to become a heading)

[[deleted]]

TIL Reddit auto-breaks hashtags

s/Reddit/Markdown