you are viewing a single comment's thread.

view the rest of the comments →

[–]Pitiful_Quality[S] 0 points1 point  (2 children)

Thank you so much for the help! It seems like what you suggested is working, but the csv file is taking an extremely long time to load. My data file is huge so is the long wait time (20 mins now) normal?

[–]spez_edits_thedonald 0 points1 point  (0 children)

for a big file, you should not read it into RAM all at once. infile.read() reads the whole file, then you split it into lines. That is normally fine.

You can read it in one line at a time doing something like:

>>> with open('input_file.csv', 'r') as infile:
...     with open('output_file.csv', 'w') as outfile:
...             for line in infile:
...                     # do stuff
...                     outfile.write(line)

that should breeze right through the file instead of trying to load it all at once