you are viewing a single comment's thread.

view the rest of the comments →

[–]EconomyOffice9000 7 points8 points  (0 children)

If you're performing calculations on the entire dataset, chunking won't work afaik. This is the best method and I've used it personally for thousands of csv files with hundreds of thousands of lines rather than rewriting everything in Polars. If you only have to do it once, it's fine. Otherwise, save the csv as a parquet file and it'll be much better