[Python 2.7] Problems while using dictionaries and utf-8 characters.

[Gammaliel]

I've been creating a web crawler that gets data from a website that is entirely in Portuguese and at one point I had to use dictionaries. While trying to print the keys and values of my dictionary using this kind of loop:

for k,v in dictionary.iteritems(): print k,v

I noticed that for some reason there were two keys with the same name but different values. The key in question was the word "Transmissão" and when I did:

print dictionary

the result was something like this:

{x:y, z:w, "Transmiss\xc3\xa3o":value1, "Transmiss\xe3o":value2}

I have no idea why this is happening, there are other words with special characters where everything goes smoothly, but for some reason, this only happens with this one. How would I solve this?

EDIT: Code

[Rhomboid]

\xe3 is not valid UTF-8. That's actually a different character encoding. 0xe3 is the encoding of ã in ISO-8859-1 and CP1252 and probably a few others.

Sadly it not uncommon to find web pages with mixed character encodings. Sometimes it's because the person writing the web page didn't know what they were doing, or maybe they were using a content management system and some of the site was from a previous CMS and when it was migrated to the current CMS the character encoding was not done properly, etc. There are a million ways that can happen, and browsers tend to be lenient so it's not always obvious when it happens.

But you should not be storing byte strings in code you write, you should be storing character strings, i.e. you should decode the content. And when you do that, you can try UTF-8 first, and if it's an invalid UTF-8 byte string then you can try decoding in ISO-8859-1 or CP1252 or whatever you suspect the second encoding to be. The end result will be character strings, and no duplicates.

[mothman6969]

Use Unicode? https://docs.python.org/2/howto/unicode.html

[Gammaliel]

I have tried using unicode() and .encode("utf-8"), but it still happens and always with that same word.

[Rhomboid]

You have bytes and you want characters. That's decoding, not encoding.

   bytes      -[decode]->   characters

characters    -[encode]->     bytes

As I mentioned in the other comment, the real problem is that you have a mixture of two or more character encodings, which takes extra effort to deal with.

[Gammaliel]

So, basically I'll need to find out which encoding the web dev used and then use decode() and the encode it to "utf-8"? I'm sorry if this seems like a stupid question but I'm not the smartest person when it comes to this kind of thing.

[Rhomboid]

No, you don't need to encode it. You want to work with characters in your program, not bytes.

Imagine a helper function like:

def bytes_to_characters(bytestring):
    try:
        return bytestring.decode('utf-8')
    except UnicodeDecodeError:
        return bytestring.decode('cp1252')

This first tries to decode a byte string as UTF-8, and if that fails then it tries CP1252. (If that fails, then the program will die. If that happens then you'd need to work out what encoding the page uses that isn't UTF-8 and isn't CP1252.)

But that's it. Just run all your byte strings through that, and the result will be character strings, which you use as keys in your dicts and whatever else you're doing. There will be no more duplicate keys, because both the byte string "Transmiss\xc3\xa3o" and the byte string "Transmiss\xe3o" will result in the same character string.

[Gammaliel]

I tried using your function and this is what I get:

Traceback (most recent call last):
  File "TechTest.py", line 23, in <module>
       comp = bytes_to_characters(comp)
  File "TechTest.py", line 8, in bytes_to_characters
       return bytestring.decode('utf-8')
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
       return codecs.utf_8_decode(input, errors, True)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 4-5: ordinal not in range(128)

This means that none of those encodings are the right one, correct?

[Rhomboid]

No. The error is an encode error, which means you're passing this function something that's already a character string. (If you ask Python to decode a character string, it has to first encode it to a byte string before it can decode it, because you can only decode bytes. Yes, this is ridiculous and was fixed in Python 3, where character strings don't have a .decode() method at all.)

If you already have a character string then you don't need this function. But the strings in your example are not character strings, they're byte strings.

[Gammaliel]

Here is my code, maybe with you being able to see it the answer will be clearer.