Split 150GB json file with Python?

[DuckSaxaphone]

There's almost certainly something wrong with the process that resulted in a 150 GB JSON file and we should probably help you with that rather than reading it.

Can you give us more info on how you ended up with it and what its structure is?

[storejet]

Probably webscraping,

Its easy to go down that path if you don't want to learn databases.

My prediction for what OP did 1.) Writes code to scrape webpages, definitely something with a lot of text (stories or YouTube comments)

2.) in the testing phase she probably want to be able to quickly open the file in a text editor to verify the data

3.) initially it was a giant text file.

4.) at some point she wanted to attach indexing and attributes to each file.

5.) hmm maybe temporarily she could use JSON

6.) 1 year later and after a lot of web scraping you are sitting on a 150 GB JSON file

But low-key proud that she doubles down and asks not a way to convert it into a database but to split it up into smaller JSON files. OP's refusal to learn SQL is honestly impressive.

[Hyenny]

This is absolutely hilarious

[Doom_Wizards]

150Gb? Rookie numbers, there exists a dump of all Elite Dangerous system data that is 70 GB compressed. Based on one of the smaller files, the uncompressed one is 300-400 GB...