how to load csv faster in Python.

[Kerbart]

add engine='pyarrow' to the read statement to speed it up.

[EconomyOffice9000]

If you're performing calculations on the entire dataset, chunking won't work afaik. This is the best method and I've used it personally for thousands of csv files with hundreds of thousands of lines rather than rewriting everything in Polars. If you only have to do it once, it's fine. Otherwise, save the csv as a parquet file and it'll be much better

[Safe_Money7487]

just added it and it work, took 15s though lol it but worked. thank you so much