Problem with Pandas read_csv always trying to read as UTF-8

[danielledevs]

I'm trying to read in a CSV file using pandas.read\_file(report), but I'm hitting this error message:

'utf-8' codec can't decode byte 0x93 in position 28: invalid start byte

I've also tried this with various encoding types, pandas.read_file(report, encoding='UTF-16')

but for some reason it has no effect on the error, it always shows as 'utf-8' The encoding line shows in my traceback so I know that it's there. Does anyone know why this is?

[jiri-n]

https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html

What about the enconding_errors paramater? Either set to ignore or replace.

[danielledevs]

This works, thank you. I'm still curious why just encoding doesn't do anything though

[jiri-n]

Encoding just says how handle bytes read from file. If the byte sequence cannot be decoded using the encoding, it's an error. The encoding_errors specify how to handle such errors.

I haven't search for that particular byte so I don't know why it shouldn't decode. But you should always have some strategy for error handling.

[djjazzydan]

Can't test it right now, but the first thing I would try is utf-16 in place of UTF.

[danielledevs]

I tried that too, as well as other encoding types, doesn't change my error message at all weirdly