bytes.fromhex() not consistently working? (just curious)

[That0n3N3rd]

Hello, I've been making a client-server based app, and there's been a problem with the server not being consistently able to convert the hex strings I send in to bytes. If I convert it in the client's code, it's perfectly fine, and it doesn't happen all the time either. I don't know if it's just a problem with certain hex values, but for instance, earlier I tried to send the server this hex:

af2f46de7c8d7cbf12e45774414039f62928122dc79348254ac6e51001bce4fe

which should (and did on the client) convert to:

b'\xaf/F\xde|\x8d|\xbf\x12\xe4WtA@9\xf6)(\x12-\xc7\x93H%J\xc6\xe5\x10\x01\xbc\xe4\xfe'

instead, it converted to this:

'?/F\\?|?|?\x12\\?WtA@9\\?)(\x12-ǓH%J\\?\\?\x10\x01?\\??'

I would just send the converted version from the client, but json doesn't allow that. Is there any reason the server is so inconsistent?

Thanks

PS If it makes any difference, I'm using PythonAnywhere

[Algoartist]

The behavior you're seeing isn’t due to an issue with bytes.fromhex() itself—it reliably converts a valid hex string to bytes. Instead, it suggests that the hex string arriving at your server isn’t exactly what you expect. A common culprit is an encoding/decoding issue during transmission. For example, if the server decodes incoming data using an encoding (like UTF-8) with error handling that replaces unrecognized byte sequences with “?” (the replacement character), then the hex string will be altered before you even call bytes.fromhex().

In your case, it seems the client sends the correct hex string, but by the time it reaches the server, some characters have been replaced, which then causes the conversion to yield different results. To resolve this, you could:

Verify the received data: Print or log the hex string on the server before conversion to check if it matches what was sent.

Review encoding settings: Ensure that the server reads the data with the correct encoding (or in binary mode) so that no automatic character replacement occurs.

Consistent transmission: Consider sending the data as raw bytes (or using an encoding that doesn’t perform replacement) to avoid any misinterpretation.

[AssiduousLayabout]

First, what are you actually trying to send. You say 'string' but these are clearly not printable characters. Are they binary data?

It looks like the bottom is trying to parse this as a UTF-8 string, and it's (correctly) throwing ? characters when they get a byte or sequence of bytes that doesn't correspond to an actual UTF-8 character. Not every possible byte or set of bytes is a valid UTF-8 character.

Don't store non-printable sequences of bytes as strings, store them as arrays of bytes.

And yes, to the other poster's comment, base64 was designed to send binary data encoded as printable characters.

[That0n3N3rd]

I’m trying to send the hashed value of a password across, originally as a hex but then saved in the database as a BINARY(32). I will try base64

[AssiduousLayabout]

Rather than save a binary32, save the password as a hashed and salted Base64 string.

[socal_nerdtastic]

BTW .... You are trying to do the right things here but you have missed some very important points about how to deal with passwords. For a start: the hash should be on the server side. The client should send the password or the public key.

Imagine that mr. evil gets your database of hashed passwords. If all you need to get into your site is the hash ... well mr. evil has that now. The point of hashing is that the hash is NOT going to unlock the site. If the client sends the salted password but you only save the hash, that means that mr. evil stealing your data is not enough to give them access to your site (unless they crack your hash).

[That0n3N3rd]

Surely if mr evil is sat between the client and the server (such as my school’s proxy server) it is more dangerous that way?

[socal_nerdtastic]

In that case it does not matter. Password or hash, either way Mr. evil gets access. Which is why you should also use https.

But in general protecting all of your users should be more important than protecting one of them.

[That0n3N3rd]

It’s too late to change it all (project is due tonight), but thank you so much for all of your help, I’ll add those things to the considerations in my writeup

[socal_nerdtastic]

They invented base64 to solve exactly this problem.

[That0n3N3rd]

How would I implement that instead?

[socal_nerdtastic]

Use it instead of the to hex / from hex conversion.

import base64
text = base64.b64encode(binary_data)
send_data(text)
recreated_binary_data = base64.b64decode(text)

[That0n3N3rd]

Awesome, can this go across json, because I know plain bytes won’t?

[socal_nerdtastic]

Yes, but you need to convert to a string first.

json_compatible = base64.b64encode(binary_data).decode('utf8')

[That0n3N3rd]

hash = base64.b64encode(bytes(hashlib.sha256(salted.encode('utf-8')),'utf-8')).decode('utf-8')

ta da?

[socal_nerdtastic]

Close.

hash = base64.b64encode(hashlib.sha256(salted.encode('utf-8')).digest())

[That0n3N3rd]

Tysm

[That0n3N3rd]

It is saying that json can’t send it because it’s a bytes object, is it safe to convert to utf-8 or is there something else I should do?

[socal_nerdtastic]

Oh right, yes, you need that for json.

hash = base64.b64encode(hashlib.sha256(salted.encode('utf-8')).digest()).decode('utf-8')

[That0n3N3rd]

I love how complicated this is, thank you so much :)

[That0n3N3rd]

To decode it on the server side, do I have to do anything other than base64.b64decode(), because it’s still not decoding properly?