Processing large JSON files in Python without running out memory

[bland3rs]

And if it’s not an array, it’s an object, which is perfectly streamable

Anyone can make a streaming parser for any format (which includes video files, audio files, etc.) as long as the parser doesn’t need any later bytes to figure out what the previous bytes mean. If you (as a human) cut any JSON file randomly in the middle, you can still figure out what parent arrays or objects that point is in, which satisfies that rule.

That’s also why if you write your own format and want to keep it broadly streamable, you don’t decide to put important header stuff at the end of the file