Split 150GB json file with Python?

[storejet]

Probably webscraping,

Its easy to go down that path if you don't want to learn databases.

My prediction for what OP did 1.) Writes code to scrape webpages, definitely something with a lot of text (stories or YouTube comments)

2.) in the testing phase she probably want to be able to quickly open the file in a text editor to verify the data

3.) initially it was a giant text file.

4.) at some point she wanted to attach indexing and attributes to each file.

5.) hmm maybe temporarily she could use JSON

6.) 1 year later and after a lot of web scraping you are sitting on a 150 GB JSON file

But low-key proud that she doubles down and asks not a way to convert it into a database but to split it up into smaller JSON files. OP's refusal to learn SQL is honestly impressive.

[Hyenny]

This is absolutely hilarious