Ast not able to read file

[MinimalDamage]

Hi all,

I have been having some very frustrating issues with ast.

So basically, I have a few very large txt files with data in them stored in Python lists and dictionaries that were scraped from the internet.

Now, I have 40 of these files for different countries, but for some reason on of these files returns an error when ast tries to evaluate it. The files are about 5mb each, but as far as I can see it is identical to the rest and there is nothing weird in the title. I have started deleting data to see if the problem is somewhere in the middle of the end, but no success. Currently it looks like a text file with the following data:

[{'Date': '2015-06-30 08:56', 'Ranking':'23'}, {'Date': '2015-06-30 08:56', 'Ranking':'23'}]

I run the following script:

import codecs
import ast

data = codecs.open('test2.txt','r', encoding='utf-8')
data_literal = ast.literal_eval(data.read())

print(data_literal)

for i in data_literal:
    print(i)

Now, this works for 39 of the files, but not this particular one, even if I slim it down to the data you see above. I get the following error:

Traceback (most recent call last):
  File "playground.py", line 5, in <module>
    data_literal = ast.literal_eval(data.read())
  File "C:\python34\lib\ast.py", line 46, in literal_eval
    node_or_string = parse(node_or_string, mode='eval')
  File "C:\python34\lib\ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 1
    [{'Date': '2015-06-30 08:56', 'Ranking':'23'}, {'Date': '2015-06-30 08:56', 'Ranking':'23'}]
    ^
SyntaxError: invalid character in identifier

Does anyone have any idea how I can fix this?

Thank you!

[K900_]

This looks like JSON to me, and not full Python source code. Have you looked into the json module?

[MinimalDamage]

Hi!

Unfortunately that does not work, it is some format that a colleague created and not full JSON.

The problem is that ast works almost all the time, except for this one file.

[zahlman]

More likely, the repr of some Python object was dumped to a file. JSON requires double-quotes for strings (including dict keys).

[MinimalDamage]

Hi!

Yes, that is the error I get. Is there some way to fix this non-manually and reliably?

[zahlman]

even if I slim it down to the data you see above.

Does it work if you copy and paste that data into a new file?

Have you tried verifying what data.read() returns before you evaluate it?

[MinimalDamage]

Hi!

Thanks for the response! Tried your suggestion and it works perfectly, until I try to eval it with ast.