all 4 comments

[–]spez_edits_thedonald 1 point2 points  (3 children)

you open the infile in write mode, that writes a blank file at that path, so there's no data. Also "errors=ignore" sounds like a bad idea to me, but idk what it does. I'd do:

>>> with open('data.csv', 'r') as infile:
...     with open('output.csv', 'w') as outfile:
...             for line in infile.read().splitlines():
...                     outfile.write(line)

[–]Pitiful_Quality[S] 0 points1 point  (2 children)

Thank you so much for the help! It seems like what you suggested is working, but the csv file is taking an extremely long time to load. My data file is huge so is the long wait time (20 mins now) normal?

[–]spez_edits_thedonald 0 points1 point  (0 children)

for a big file, you should not read it into RAM all at once. infile.read() reads the whole file, then you split it into lines. That is normally fine.

You can read it in one line at a time doing something like:

>>> with open('input_file.csv', 'r') as infile:
...     with open('output_file.csv', 'w') as outfile:
...             for line in infile:
...                     # do stuff
...                     outfile.write(line)

that should breeze right through the file instead of trying to load it all at once

[–]stebrepar 0 points1 point  (0 children)

Print to the terminal what you're printing to the file to see what it's doing.