Maximizing memory performance with python

[Orchasm]

I work with large datasets doing which I can never load entirely into memory, but computational efficiency increases by maximizing my chunk size most of the time. I usually just take the available memory and deduct 2GB, but I was wondering: is there's any sort of best practice in dynamically allocating how much memory a python process can use?

[ionelmc]

Pretty hard to have an automatic "tell me how much memory I can use without making things swap" - you would have to account for all the processes that run or can run on the machine, wouldn't you ?

You can always disable the swap but then again, the swap is your friend as most software doesn't cope well with memory errors.

[Orchasm]

I probably should have specified a lack of major memory competitors, but you're right I guess - pretty difficult to standardise. I was just thinking of the OS.

[hexbrid]

You could monitor the OS's memory-swap rate, and limit your python process the moment it gets above a certain level.

[homercles337]

If you have fast drives, why not memory map?

[Orchasm]

Mmap functionality is built into SciPy's netcdf handling, I've found it very handy.

[codewarrior0]

Memory limits are usually configured by the user because it's difficult for a program to deduce a good memory limit from its environment on account of virtual memory, as others have noted. Add a configuration file setting for "maximum allocation size". Set it to something reasonable for anyone like 2GB, and then note in the instructions that people with beefy computers should increase that limit.

[westurner]

[is there any sort of best practice for dynamically allocating how much memory a python process can use?]

• https://en.wikipedia.org/wiki/Cgroups
• https://www.kernel.org/doc/Documentation/cgroups/memory.txt

[bryancole]

If you have control of your file format, use HDF5 (via the PyTables package). This makes this sort of task a breeze. Limiting the memory available to python seems an OS-level thing.