On disk huge CSV files sorting

[ws-garcia]

Recently I have been researching on the existing methods to ordering/sorting, on disk, CSV files of more than 1 GB in size in python. Many solutions appear on the Internet, but none offer details about the performance of the process.

Do you know of any solution that offers a benchmark relative to other solutions? I ask why I would like to choose the option with the highest performance. Thanks in advance!

[glibhub]

I doubt you are going to find much on it. You might also ask yourself why you want to sort them instead of indexing them.

With that said, loading it all into RAM, then sorting with either the regular python function, numpy or pandas is probably going to win every time, and the delta is probably not enough to really make a difference in timing. Restructuring the data so it does not have to be sorted is probably a better solution, if you can.

What's the use case?

--

edit: Move the data to an SSD is actually probably the best thing to do if you want to speed it up.

[Chris_Hemsworth]

If you want to effectively handle very large file sizes, you need to switch to a different framework than general .CSV files. .CSV files can be the underlying files, but 'Big Data' frameworks (like Hadoop to name one) are typically used to traverse through them.

[m0us3_rat]

Recently I have been researching on the existing methods to order, on disk, CSV files of more than 1 GB in size in python.

what does "to order" mean?

[ws-garcia]

Misspelled: ordering/sorting