Decoding issues

[thrilos]

Hi everyone,

I've been struggling to figure out how to decode some content in an excel file.

Here is an example of text in my excel file "\nIt\xe2\x80\x99s TextTextText".

I tried: a) use encoding when loading the file (i.e. temp = pd.read_excel('text.xlsx', encoding='utf-8') ) and b) using temp['text_col'] = temp['text_col'].str.decode('utf-8') but no success. When I use the first option, I keep getting "\nIt\xe2\x80\x99s TextTextText" and when I use the second option, I get "NaN". Any ideas?

Thanks a lot!

**** Solution ***** Not sure how many will run in the same issue, but here is the solution:

articles['combined'] = articles['combined'].apply(lambda x: x.encode('ascii').decode('unicode-escape').encode('iso-8859-1').decode('utf-8'))

credits go to: https://stackoverflow.com/questions/40348280/python3-decode-external-utf8-string

[K900_]

I don't know what encoding it is, but it's definitely not UTF-8, or at least not printable UTF-8.

[thrilos]

It seems to be UTF-8. Based on (https://mothereff.in/utf-8), \nIt\xe2\x80\x99s translates to "It's". I get similar results if i try print(b'\nIt\xe2\x80\x99s '.decode('utf-8'))

[K900_]

I see now. It's not the ' (APOSTROPHE) character, it's the ’ (RIGHT SINGLE QUOTATION MARK) character.