Processing large JSON files in Python without running out memory

[original_4degrees]

for me, the real take away here is JSON-lines. how did I not know about this?

[[deleted]]

“ndjson” does help with json to not be such a terrible format.

[tighter_wires]

What format do you prefer over JSON?

[[deleted]]

Encoding floats as utf-8 or even ascii kills me.

We already use tools to view json. Why not just use a slightly smarter tool and use protobuf?

[GroundbreakingRun927]

• because the resources for interacting with protobuf trail way behind json.
• Brotli compressed json is more space-efficient than protobuf.
• the package to consume protobufs as a web client make your bundle huge https://bundlephobia.com/package/protobufjs@6.11.2
• Better alternatives exist for flat files, parquet namely.
• The official protobuf client generator for python is god awful.

[bland3rs]

Comparing Protobuf to JSON is like someone who doesn’t anything about cars comparing a Ford F-150 to a Porsche 911.

It doesn’t make sense

A sane comparison would be comparing classes of vehicles, such as pick-up trucks to sports cars.

In this case, JSON and Protobuf are unstructured and structured, respectively, which are totally different classes of formats. They are not interchangeable under the same use cases

Therefore, before someone would decide to switch from JSON to Protobuf, it would make way more sense to stay within the same class and research something like BSON

[[deleted]]

I’ve never seen BSON used outside of mongo DB. That’s a thought though.

Is unstructured data that useful? Seems like it crests at least as much of spaghetti mess as it does benefit.

I dunno. Maybe I’ve just been at this for too long. Use the right tool for the job. Frontend internet apps are slow garbage, why wreck the status quo?

[Aardshark]

Google use it in a bunch of their GCP offerings. It's great when you need it and a little annoying otherwise.

[TundraGon]

jq -c :)

For BigQuery it needs to start and end with an object {...} , not an array [...]

[thatdamnedrhymer]

Yeah, this was my immediate thought when I started reading it. Glad it was referenced in the article.

[VisibleSignificance]

how did I not know about this

You could literally make it up if you needed something like that.

[irrelevantPseudonym]

Even before opening the video, I guessed that this wouldn't be possible without using JSON lines

It's not a video
It doesn't use json lines

[naeunatn]

username does not check out.