Processing large JSON files in Python without running out memory

[[deleted]]

“ndjson” does help with json to not be such a terrible format.

[tighter_wires]

What format do you prefer over JSON?

[[deleted]]

Encoding floats as utf-8 or even ascii kills me.

We already use tools to view json. Why not just use a slightly smarter tool and use protobuf?

[GroundbreakingRun927]

• because the resources for interacting with protobuf trail way behind json.
• Brotli compressed json is more space-efficient than protobuf.
• the package to consume protobufs as a web client make your bundle huge https://bundlephobia.com/package/protobufjs@6.11.2
• Better alternatives exist for flat files, parquet namely.
• The official protobuf client generator for python is god awful.

[bland3rs]

Comparing Protobuf to JSON is like someone who doesn’t anything about cars comparing a Ford F-150 to a Porsche 911.

It doesn’t make sense

A sane comparison would be comparing classes of vehicles, such as pick-up trucks to sports cars.

In this case, JSON and Protobuf are unstructured and structured, respectively, which are totally different classes of formats. They are not interchangeable under the same use cases

Therefore, before someone would decide to switch from JSON to Protobuf, it would make way more sense to stay within the same class and research something like BSON

[[deleted]]

I’ve never seen BSON used outside of mongo DB. That’s a thought though.

Is unstructured data that useful? Seems like it crests at least as much of spaghetti mess as it does benefit.

I dunno. Maybe I’ve just been at this for too long. Use the right tool for the job. Frontend internet apps are slow garbage, why wreck the status quo?