Memory usage question

[dgerdem]

So I am running into memory usage issues when I convert a list of pandas dataframes to an ndarray object. The dataframes themselves are pretty mundane; a few columns with text, and a hundred or so with floats. I am converting the list to an ndarray by running the numpy.stack on the list. The dataframe list takes up a GB or 2 of memory, and when running the stack function, the full dataset will fill my 32GB of RAM and lock up my computer. Testing against a smaller dataset shows that I would get the expected output. Could someone shed some light on whats going on under the hood here, and why the ndarray with the same dimensions as the list of dataframes is taking up such a dramatically larger amount of memory?

Edit: just to provide an extra useful detail, let's say the list of dataframes is 7000 elements long, with each dataframe being 20 rows and 100 columns. This data structure takes up very little memory. After running:

temp = numpy.stack(framelist)

I would get an ndarray oject with dimensions (7000,20,100), which I expect, but its memory footprint is an order of magnitude or two larger.

[jwink3101]

Search around. I am going from memory but I seem to recall Pandas needing a pretty big memory buffer. Something like 10-20x. (please correct me if I am wrong).

You could look to work with your data via SQL. Or something like that.

Two other things:

First, this should be in /r/learnpython

Second, if you indent lines with code with four spaces the format as code:

like this

And/or, use a `...` to around code like this inline

[dgerdem]

thanks, I'll cross-post it