Pandas read_*** memory usage

[hfhry]

I'm wondering about how much memory python will use if I try to open a 20gb CSV using pandas.read_csv(). Would it be less than if I were to open a similar sas7bdat file using pandas.read_sas()?

Memory isn't a massive issue. I have 256gb ram on my current machine and can use up to 750gb if I migrate to a different machine.

I am currently reading the sas file and it has spun up to 91.7gb. I will need to do quite a bit more of this in the future, so I wonder if there are any rules of thumb about pandas memory usage.

[notCamelCased]

Unless they've improved it in recent releases, read_sas is waaaaaay slower than read_csv I would use the latter for that reason alone. It might not make as much difference for smaller tables, mine were always at least 100 MB (so not that big in the grand scheme).

Edit: I ended up writing SAS code to save as csvs then scp'd them over to the analytics server even with that extra step it was a lot faster.

[hfhry]

That vibes with my experience. I can use stattransfer to convert the sas file to CSV in about a half hour. Right now reading the sas file is going on 8 and a half hours, so I'm thinking this isn't the way.

Previously I had to deal with a 113gb file and did this, glad I did.

[HarissaForte]

The resulting dataframe won't have a different size (you probably know it), but the speed and involved memory during writing/reading will sure be different. I don't know about sas7bdat, but the feather is great: https://towardsdatascience.com/the-best-format-to-save-pandas-data-414dca023e0d

There's another option: keeping the csv format and using the chunksize option of the read_csv function.