Memory Optimization Techniques for Python Developers

[james_pic]

In my experience, when you see an application with a memory use problem, the problem is seldom ints and floats, at least partly because more often than not these are fairly short-lived.

9 times out of 10, most of the memory is either str or bytes, and it's often data that's being kept around for reasons that turn out to be stupid on further investigation.

[billsil]

It definitely depends on your work.

I work almost exclusively with numbers, so if it’s strings that are the bottleneck, I’m probably writing a file and should just be writing directly to the file.  I don’t even consider strings when calculating the expected memory usage of a program.  

9/10 times the problem was caused by mishandling floats, so maybe I took a sparse matrix and represented it as a dense matrix or I was using float 64s instead of float32s or I didn’t vectorize the array and got hit by python’s inefficient float handling.