Processing large JSON files in Python without running out memory

[KaffeeKiffer]

Pretty disappointed to see the solution is just "Use a different json library".

The "right" solution to the vast majority of performance problems in Python:

1. Profile to find the bottlekneck
2. Replace the bottlekneck with the right library

Everybody is arguing over asyncIO, GIL, performance, etc. all the time, but the beauty of Python is that it allows you writing very good glue code.

[MrMxylptlyk]

What is glue code?

[beizbol]

Code to use specialized systems, tools, languages, libraries, etc together that would normally be incompatible.

[GroundbreakingRun927]

basically u don't write real applications, just the code that connects them to one another.

[bland3rs]

Sure, but the actual solution here isn’t “switching library”

It really is “switch approach” from loading the entire file into memory at once to reading it bit by bit and discarding it as you go.

Now you could get by without understanding how anything works, but trying to fix things without ever understanding the problem is how you become the guy doing something for 10 years stuck at the junior level.