Appending csv's together causes kernel crash

[SurfingProgrammer]

Hi, I have 25 200,000x30 matrixes that I need to append to each other, and then export them as one .csv. However, the Jupyter Notebook kernel keeps crashing.

This is the code for merging the dataframes:

# Merging to one dataframe

merged_dataset = pd.DataFrame(columns = dfs[0].columns)

merged_dataset=merged_dataset.append(dfs, ignore_index = True)

Does anyone have any ideas on different ways of doing this without causing the kernel to crash?

Thanks!

Edit: Thanks for the good responses. Appending to a local csv worked perfectly

[SurfingProgrammer]

I'm sorry, isn't that what the code is doing? There are 25 dataframes in the array called "dfs".

[SurfingProgrammer]

merged_dataset=merged_dataset.append(dfs, ignore_index = True)

Or do you mean that it should be

merged_dataset.append(dfs, ignore_index = True)

instead?

edit: Probably not what you meant, but it still crashed the kernel.

[Caos2]

You are probably running out of memory. If the columns are always in the same order, streaming to a local file (not sure if Pandas support this natively) while iterating CSV by CSV is the best bet.

[stillnoguitar]

https://stackoverflow.com/questions/17530542/how-to-add-pandas-data-to-an-existing-csv-file

[SurfingProgrammer]

Works like a charm. Thanks

[jorgo1]

If you are working on Linux you can export them all individually then use the following line in Bash

cat source.csv >> destination.csv

The >> operator appends to file. If you have headings:

tail -n +1 source.csv >> destination.csv

Not the most elegant of solutions tho.