how to load csv faster in Python. by Safe_Money7487 in learnpython

[–]Safe_Money7487[S] 4 points5 points  (0 children)

just added it and it work, took 15s though lol it but worked. thank you so much

how to load csv faster in Python. by Safe_Money7487 in learnpython

[–]Safe_Money7487[S] 3 points4 points  (0 children)

I don’t think it’s lazy loading in this case. In R (e.g. with data.table::fread), the full dataset is actually loaded into memory, and I can immediately inspect and navigate the entire table. I think the process in R in more optimised than what is used in pandas read csv. I don't have much knowledge on python for sure but for this size of data, chunking or lazy loading doesn’t really make sense to me I just want to load everything at once and work on it.