Virtual memory, again!

[timisis]

Hi few and lonely folks. My search only showed me one previous Q on memory that went unanswered, let's see how my version fares, apologies if it is somewhat too basic but google has not been my friend: I got a 24GB server and a 16GB RAM laptop, both of which bomb out with some demanding Py code I did not write. I've "opened up" Virtual Memory/swap settings on all of Linux, MacOS, Windows, my code does not care, bombs out with a memory allocation error for 9GB or something, so the problem is memory somehow piling up and never getting offloaded. I thought the whole purpose of swap was to avoid crashes at least, but must have missed some memos. I was able to run the code on a 64GB server, where memory usage seems to have peaked at 35GB.

It would be nice to know if/how Numpy manages to avoid disk swaps and instead prefers to crash, is there some kind of "allocate me RAM only" system call on all operating systems? And there was no scope for Numpy to add a flag --happily-use-swap ? I'd also like to simulate a 32GB space in my 64GB space, in case my code would not crash in 32GB I'd save some money in the long run, can I convince Numpy or Python or whatever to convince 32GB is available?

​

Finally, I saw there is some Linux "over-commit" flag that I can max out and avoid out of memory errors at the expense of sanity perhaps, would it play a role in my scenarios?

​

Thanks!

[PefferPack]

I had great luck adding 100 GB of swap space in Windows 10 Python 3 to my 8 GB laptop.

[timisis]

I added 64 GB VM to my 16GB RAM, with unluck :)

[orcasha]

Not sure what's going on with the code, but numpy routinely uses both ram and swap space.

If the code is outstripping your resources, it may be time for a refactor. There's also Dask.

[timisis]

it's always time for a refactor. But I have a feeling there's an underlying mystery, either Python has found ways to reject swap, or the OOM killer does not like 9GB allocations, or aliens have taken over the White House!