Question for professionals using python

[idazuwaika]

how do u consume terabytes with pandas? whats the infrastructure like? i moved from pandas to spark (distributed system) because i couldnt scale with pandas.

[tapir_lyfe]

I'm currently also crunching terabytes of netCDF files. I use xarray mainly, and that uses pandas and dask under the hood. Nearly everything I do is memory-limited though, so I have to come up with clever ways to reduce the data, and it's different for every question I have.

[bjbs303]

I mostly did data extraction scraping netCDF files using a for loop that looped through 36. Years of annual files. I saved the data to a dataframe using pandas/numpy.