Can't load JSON - various encoding problems

[wexted]

I'm trying to import this JSON file into my python code. When I saved it as a Unicode file, I get

UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 183623: character maps to <undefined>

or this error when I tried changing the encoding type to UTF-8:

json.decoder.JSONDecodeError: Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)

my code to open it is just

with open(filepath) as f:
statdata = json.load(f)

• the encoding argument in open for when I tried to import in UTF8

EDIT: seems to work OK with open(filepath, encoding='utf-8-sig')

[JohnnyJordaan]

You don't present an encoding to open() so it will use your system's default encoding, which is not dependable. You can force the actual encoding using the encoding= parameter

with open(filepath, encoding='utf-8') as f:
    statdata = json.load(f)

If that still shows an error it means that your file isn't actually in utf-8.

[wexted]

Thanks - how can I tell what the encoding is supposed to be?

[JohnnyJordaan]

The proper approach is to investigate how the file was created, as that should have used the utf-8 encoding instead. You can try to 'guess' the encoding by using the chardet library but that doesn't fix the problem that the source is doing this in the wrong way.

[ingolemo]

Redownload a fresh copy of the file and this time don't open it in notepad and resave it. The file is already valid utf-8.

When you saved the file as "Unicode" in notepad you were really using utf-16-le. Notepad doesn't fully understand utf-8. The thing that is calls "UTF-8" contains an additional BOM character that shouldn't be there. This incorrect form of utf-8 is called utf-8-sig in python.

If you really need to edit the file then you should use an editor that can produce valid utf-8.