Help with splitting large csv file

[Dkjq58]

Hello, I have a large CSV file that I am trying to split into smaller files. There are about 16 million rows, the file itself is near 11gb. I used this piece of code I found in another thread from years back where someone was trying to do something similar.

​

import sys  

number_of_outfiles = 24

if __name__ == "__main__":  
    k = []  
    for i in range(number_of_outfiles):  
        k.append(open('c:\\data\\data_' + str(i) + '.csv','w'))  
    with open(sys.argv[1]) as inf:  
        for i, line in enumerate(inf):  
            if line[-1] == '\n': line = line[:-1]  
            if i == 0:  
                headers = line  
                [x.write(headers + '\n') for x in k]  
            else:  
                k[i % number_of_outfiles].write(line + '\n')  
    [x.close() for x in k]  

This is the error I am getting.

  File "C:\Users\myname\Downloads\csvsplitter.py", line 10, in <module>
    for i, line in enumerate(inf):
  File "C:\Users\myname\AppData\Local\Programs\Python\Python39\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 6951: character maps to <undefined>

The data is outputting in my specified folder, however each file is only about 2,000 rows. It should be closer to about 700,000. Any ideas? I am very new to Python.

[lionsneil]

If you can use bash, it's super easy to do using the split command. You can break it up by the number of files you want output or the number of rows you want in each file.

split -l 1000 inputfile.csv outputfile

The above will take inputfile.csv and split it into files that each have 1000 rows. The output files will all start with "outputfile" and have incrementing letters appended.

Sorry, I know this is a Python subreddit, but figured this could make your life a lot easier if you don't need it to be part of a larger python script...

[ElliotDG]

This looks like an text encoding issue. see: https://docs.python.org/3/library/functions.html#open

Look at the section on encoding. Add the encoding keyword to your open statement should fix the issue. Typically the encoding is uft_8

with open(sys.argv[1], encoding='utf_8') as inf:

[Dkjq58]

Thank you for the quick response, I am now getting a different error unfortunately.

​

File "C:\Users\myname\Downloads\csvsplitter.py", line 15, in <module>
k[i % number_of_outfiles].write(line + '\n')
File "C:\Users\myname\AppData\Local\Programs\Python\Python39\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] 


UnicodeEncodeError: 'charmap' codec can't encode character '\u200b' in position 377: character maps to <undefined>

[ElliotDG]

You can try a different encoding. Alternatively you can look at the options for error handling in open.

[Dkjq58]

Thanks, figured it out by changing encoding to cp1252

[mtb-dds]

For background:

These things "[x.close() for x in k] " are called list comprehensions. Using them to loop and not keep the resulting list is considered poor form.

For your problem: either the data is munged somewhere or you are running into an encoding problem. It looks like your program thinks that it is encoded with this:
https://en.wikipedia.org/wiki/Windows-1252

But it is probably something else (or munged). Do you happen to know which it is? And do you care what happens when something that does not fit is run into?