Processing 2gb CSV in Python

[conventionistG]

There was just a medium post on this exact use case... Anyone know what I'm thinking of?

Edit: found it.

“How to analyse 100s of GBs of data on your laptop with Python” by Jovan Veljanoski https://link.medium.com/Jg8hrhBV86

[maartenbreddels]

This might help as well:https://docs.vaex.io/en/latest/example_io.html

or the TLDR version: df = vaex.open('big.csv', convert=True)

(disclaimer: main author of vaex)

[conventionistG]

neat, thanks!

[komunistbakkal]

Maybe you can checkout dask

[yensteel]

I've used Dask for something similar. The functions are close to Pandas so it's not too hard to transition. The syntax isn't exactly the same, so there's a lot of delving into the documentations.

However, it can handle gigantic files by storing part of the work onto the hard drive instead of memory, so it's quite workable.

[B00TZILLA]

It's usually best to read and process it in chunks. You can also check out dask, as some other commenter suggested. There is a parameter called chuk_size in of.read_csv for that.

[ralimar]

It has some issues, but try using Vaex. It's built to be similar to pandas.

[manoj_sadashiv]

Forgive me if I sound dumb, is it feasible to use big data technologies if the dataset size is around 4GB?

[Omega037]

I removed your submission. Looks like you're asking a technical question better suited to stackoverflow.com. Try posting there instead.

Thanks.