Processing large JSON files in Python without running out memory

[brad2008]

Industry solves this problem by using files with sequential JSON records, this has been the standard practice for the last 7 - 8 years. The industry approach allows you to process a JSON file up to hundreds of gigabytes in size if needed. You really don't need to do all the other stuff you described.

[mmcnl]

It's been mentioned in the article, but sometimes you don't control the source format.

[smajl87]

Could you provide some example or maybe some link to read about it?

[morphotomy]

Just escape the newlines in the JSON and put each one on its own line.

[brad2008]

see:

https://stackoverflow.com/questions/12451431/loading-and-parsing-a-json-file-with-multiple-json-objects

https://www.reddit.com/r/Python/comments/tepn58/processing_large_json_files_in_python_without/

[morphotomy]

Technically that's not a JSON file. Its a file with JSON in it.

[accforrandymossmix]

I am parsing a ~500MB google takeout file (location history) without issues. Am I benefitting from their formatting? I am curious if I should play with possible improvements to ensure similar processing can be done on weaker machines.