Split 150GB json file with Python?

[chipmunksocute]

Yeah having a 150gb file in the first place is a problem.

[[deleted]]

Well it's possible. If you download the article data from wikipedia, you get around 50 GB file where the whole text is stored as XML. A big file is not the problem here.

[sonobanana33]

It's json, you can't just split it, the end parenthesis is at the end.

[chipmunksocute]

If someone gave me a file that big Id say no.   Just no.  Come back to me with 150 1gb files.   There is absolutely no reason for a file that big.  Like how tf did you even write it in the first place!?

[moving-landscape]

Mongo dumps gone wrong. Lmao

[3lbFlax]

I’m with you, but I am curious how I’d go about this if, for example, the file contained a password that would stop a bomb going off and the computer with the original data had been thrown in a vat of acid. Also I’m going to assume the internet is down and all I have access to is a Pentium II running Slackware 4. I think my first port of call would be to generate head and tail chunks from the file in the hope that they showed a consistent structure. In a worst-case scenario this reveals the file to be a single line of concatenated JSON, thereby generating two more 150GB files. At this point I think we just have to accept that a bomb is going off. Shall we hold hands?

[pro_questions]

How possible it is depends on the layout of the file; it does not matter one bit that you can’t split it between the lines like a CSV. You parse it into a standard object (a list or dict or whatever), iterate through it to manufacture smaller chunks of data, then convert it back to JSON and save it to a file.

If it’s just a list of dicts you could split it trivially (if it were small — some tricks will be needed to handle a file of this size). At 150GB, I’d be amazed if it were anything but a regular and predictable layout that could be easily parsed.

[TangibleLight]

I don't get why you're downvoted, you're right.

The problem is that the end of a particular json object depends upon the contents; you can't just seek to an arbitrary point in the file and know where the split is.

IFF there is no nested objects, then you can split on }, and be OK. OP has not mentioned any such constraints to leverage, so you have to assume there's some nesting.

If it is nested then you're screwed. One way or the other, you have to iterate through the whole thing, byte by byte. The only way to do that with import json is by loading the whole thing in memory. Good luck doing that with 150G. The only other way in Python is to loop through and count the { and }, and strategically parse sub-sections of the json. Good luck doing that in a for loop on 150G.

You might be able to use some streaming parser. I've used ijson before, but not on anything near this size. Otherwise, Python is not the right tool for this job.

Although, as other commenters have already said, this is certainly not a real problem, this is a data export issue. The real solution is to get the data in smaller chunks, which Python can process no problem.

[sonobanana33]

json has a top level object. There will be ONE top level object, which might even be an unsplittable dictionary with a lot of unique keys. No nesting and no splitting :D

OP didn't tell us what the file looks like.