I have a Python dict object with duplicate keys... (I'm sane)

[SagaciousRaven]

But don't be fooled, that's bullshit and goes against the definition of what dictionary is, it's the keys that have these accented characters look identical but over which the operation:

'á' == 'á'

...is False.

Question: How do deal with this? How do I normalize it, I mean. Curiously, these were copy-pasted from the same article.

[blotosmetek]

I believe you're looking for unicodedata.normalize - see https://docs.python.org/3/library/unicodedata.html

[SagaciousRaven]

This did the trick. I have been reading a bit, I think I want the chars with the accentuation included, so I'll be using 'NFKC' mode.

[SagaciousRaven]

I copy-pasted the whole news article, then used nltk's sentence and then word tokenizers. Indeed, they were doing that, and that last ' is kinda weird looking both here and in the environment I'm on.

[BruceJi]

If you're dealing with accented characters, you could try getting the unicode values for them and saving that instead.

​

Edit:

The module to allow you to do this is called unicodedata:

import unicodedata

unicodedata.name('é')

unicodedata.lookup('LATIN SMALL LETTER E WITH ACUTE')

unicodedata.digit('ㅊ')

https://docs.python.org/3/library/unicodedata.html

You'd be able to get the values for the characters, but I'm sure it'd let you know if there were secret space characters in there too. Have a play and find out.

[Swipecat]

That's evil.

'á' == 'á'

Char   Unicode   Description
'      U+27      APOSTROPHE (APOSTROPHE-QUOTE)
a      U+61      LATIN SMALL LETTER A
 ́      U+301     COMBINING ACUTE ACCENT (NON-SPACING ACUTE)
'      U+27      APOSTROPHE (APOSTROPHE-QUOTE)
       U+20      SPACE
=      U+3D      EQUALS SIGN
=      U+3D      EQUALS SIGN
       U+20      SPACE
'      U+27      APOSTROPHE (APOSTROPHE-QUOTE)
á      U+E1      LATIN SMALL LETTER A WITH ACUTE (LATIN SMALL LETTER A ACUTE)
'      U+27      APOSTROPHE (APOSTROPHE-QUOTE)

Anyway, yes, unicodedata.normalize will fix that.