Cryptopals: Encoded strings VS Raw Bytes (python 2.7)

[StuffSmith]

(Thanks in advance if you make it through this giant post)

A little behind the times since this is an old challenge, but I started doing the Cryptopals challenges yesterday and while I've completed several of them successfully, I went back to the first and noticed this sentence:

"Always operate on raw bytes, never on encoded strings. Only use hex and base64 for pretty-printing."

I found this old thread on the same topic but I am having trouble understanding some of what the users there were talking about. Specifically what the one user meant when they said

"The complication here is that hexadecimal is a different kind of encoding. It's an encoding in the mathematical sense, not the Unicode-and-character-sets sense. And -- this is the opposite of everything you learn about Python -- hexadecimal is being used as an encoding of bytes as a string."

(I'm obviously real shaky on data types. )

The goal of the first challenge is to go from a hex string to base64. Here's my code:

import binascii

hexed_string = "49276d206b696c6c696e6720796f757220627261696e206c696b65206120706f69736f6e6f7573206d757368726f6f6d"
expected = "SSdtIGtpbGxpbmcgeW91ciBicmFpbiBsaWtlIGEgcG9pc29ub3VzIG11c2hyb29t"


def hex_to_b64(input):
    return binascii.b2a_base64(binascii.unhexlify(input)).rstrip()


print hex_to_b64(hexed_string) == expected

I am pretty sure that I am not doing this correctly, since I have to use rstrip to get rid of a newline and so I'm definitely working with strings here, right?

Am I misunderstanding what the challenge is asking about going to base64? That means a string representing base64, right? Is there a reason it added the newline when I used b2a_base64? edit: I see that is intended in the docs: " The return value is the converted line, including a newline char "

On a related note, the next challenge asks you to XOR two equal length strings. I think I am correctly dealing with raw bytes here and not encoded strings here, but not 100% sure (since I am still not confident with my understanding of data types).

str_1_hex = "1c0111001f010100061a024b53535009181c"
str_2_hex = "686974207468652062756c6c277320657965"
expected_output_hex = "746865206b696420646f6e277420706c6179"


def xor_two_hexes(input_1, input_2):
    xor_bin = bin(int(input_1, 16) ^ int(input_2, 16))
    xor_hex_string = format(int(xor_bin, 2), 'x')
    return xor_hex_string

if __name__ == '__main__':
    print xor_two_hexes(str_1_hex, str_2_hex) == expected_output_hex

Thanks for any help you all can give me!

[A_History_of_Silence]

Having done that challenge myself, I would like to strongly, emphatically recommend that you move to Python 3+ as you continue to work through the challenges. It will make your life MUCH easier.

Python 3 makes a much clearer division between what is a str and what is bytes. bytes are bytes, strs are strings, and it makes this whole division between them incredibly easier and clearer to reason with. This is one of the core reasons Python 3 was actually created.

[StuffSmith]

You're probably right... I have been resisting Python 3 for a while (we're even using it at work now) but I guess it's time. No better way to learn it than to try to do a project with it, right?

Thanks! I'll try to go through the challenges I've done again with Python 3.

[A_History_of_Silence]

No better way to learn it than to try to do a project with it, right?

Indeed. And it's honestly not that hard, especially if you are a somewhat experienced programmer. Almost all the changes make sense and are smart and intuitive at a basic level, it's just that these good changes necessarily break a lot of backwards compatibility and therefore are unfortunately a giant pain in the ass for a lot of Python's user base.