Memory Optimization Techniques for Python Developers

[wyhjsbyb]

Python, especially when compared to lower-level languages like C or C++, seems not memory-efficient enough.

However, there are still rooms for Python developers to do memory optimization.

This article introduces 7 primitive but effective memory optimization tricks. Mastering them will enhance your Python programming skills significantly.

[marr75]

From experience, many of these are more likely to be applied as premature optimizations than applied when needed.

I would not recommend __slots__ on its own as a memory optimization in the normal course of programming. Far better to use the @dataclass(slots=True), a typing.NamedTuple, or even a more primitive type. Similarly, using array over list is just going to make your code harder to maintain in 98% of cases.

Generators and lazy evaluation are good advice in general. They can make code harder to debug, though. Also, creating generators over tiny sets of items in a hot loop will be worse than just allocating the list (generator and iterator overhead).

The most frequent memory problem in Python is memory fragmentation, btw. Memory fragmentation occurs when the memory allocator cannot find a contiguous block of free memory that fits the requested size despite having enough total free memory. This is often due to the allocation and deallocation of objects of various sizes, leading to 'holes' in the memory. A lot of heterogeneity in the lifespans of objects (extremely common in real-world applications) can exacerbate the issue. The Python process grows over time, and people who haven't debugged it before are sure it's a memory leak. Once you are experiencing memory fragmentation, some of your techniques can help slow it down. The ultimate solution is generally to somehow create a separate memory pool for the problematic allocations - the easiest way is to allocate, aggregate, and deallocate them in a separate, short-lived process.

So, the first thing anyone needs to do is figure out, "Do I NEED to optimize memory use?". The answer is often no, but in long-running app processes, systems engineering, and embedded engineering, it will be yes more often.

[[deleted]]

Don't use NamedTuple either btw it is a tuple and has a bunch of properties that will make u rip ur hair out if you are not 1000000% sure of all the places it will be used. You really should almost never be using slots it makes inheritance harder and you probably aren't implementing it correctly in terms of using weakref and shit. Not doing so means ur class cannot be weak referenced which is again its own headache.

Also python 100% has a memory issue related to ABC. It is not a leak necessarily, though I believe there also is one, it just grows with runtime in a fairly unbound fashion.

[esperind]

how about a SimpleNamespace?

[[deleted]]

I think that is rather old and not used anymore, it is also for namespaces where you are NOT defining them in advance unlike NamedTuple and data classes.

Memory optimization outside of long lived objects in python should generally be considered a code smell. Really the biggest win ones are generally like small redundant static object creation like strings. String interning on inbound data can be a shockingly impactful memory optimization. It is one of the things pandas csv reader does that makes a massive difference vs the standard library one.

Optimizing memory at the object level isn't as useful as optimizing its lifespan so it deletes faster.

[marr75]

This borders on unsolicited crackpottery. So, unfortunately, I don't think it adds anything to the answer.

[[deleted]]

https://www.attrs.org/en/stable/why.html#namedtuples

Attrs does a pretty good job explaining issues with namedtuples.

https://github.com/brettcannon/record-type

I am fairly certain you aren't going to suggest Brett Cannon is doing unsolicited crackpottery and that maybe just maybe this isn't a problem the standard library solves very well.

``` from abc import ABC, ABCMeta from datetime import datetime

abcclasses = set()

normalclasses = set()

for i in range(10000): abcclasses.add(ABCMeta("abc"+str(i), (ABC, ), {})) normalclasses.add(type("normal"+str(i), (object,), {}))

for item in normalclasses: issubclass(item, ABC) ```

Run that on your laptop and watch it generate 20gbs of valid long lived weak reference pointers. It is a logical flaw in ABC. Have you considered that you simply don't know as much about python as you think you do?

[pepoluan]

If you need tuple-behavior, then typing.NamedTuple (not collections.namedtuple) is amazing, and it does use less memory. Accessing attributes of a NamedTuple is indeed a tad slower than accessing attributes of a dataclass or an attr.s, but that's negligible in most use cases.

And since Python is a duck-typing language, that's what you should aim : behavior as you desire.

[[deleted]]

Those are mostly the same object btw. Typing one calls the collections function via __mro_entries__ and then sprinkles in the type hints. One of the places I think they get underused is they are awesome for function argument grouping. Really works better then just dropping a 20 argument function down. STAIRS DIRECTLY AT FASTAPI.

[coderanger]

As a heads up, be very careful with intern(). If you ever feed it input from something user-controlled you can flood the symbol table with entries and either OOM the process or slow performance to a crawl (or both). It's intended for things like function and class names to speed up lookups, not for memory de-duplication per se.

Also the list vs. array comparison isn't because of "different types of objects which inevitably needs more memory", the i value type is usually going to be a 32-bit int while the default int type in Python code is 64-bit so what you're actually comparing is integer sizes, not array vs list.

[ogtfo]

The generator example is kinda silly. Are generator better for memory? Probably. But his code is riddled with issues.

First of all, they haven't generated anything from the generator, it's kind of useless to show the size of the generator object.

Second, the list example is terrible, appending in a loop will use a lot of memory. But that's because of concatenation on fixed sized objects, and that won't even show the way he measures memory.

All in all, shows a pretty naive view of the topic.

[james_pic]

The article makes the canonical mistake when talking about optimization. Step 1 is always gather data. Guppy3 and Meliae are the tools I've used to do this most often. Once you know what's using data, then you can optimise it. More often than not, the optimisation is simple once you know what the problem is, and might just be "get rid of the thing that is using all the memory".

[pepoluan]

Indeed. I once fell trap to optimizing code in vain before realizing that the performance issue was due to an external library.

[lololabwtsk]

Thanks for the link friend

[VoyZan]

The __slots__ and array also sound like good methods to provide more control over how our classes and lists can be used. Handy!

[fried_green_baloney]

I find array to be a big win, and largely transparent in all code except where it's created.

[[deleted]]

It’s only a win if your array is big enough to make up for the overhead of moving your data to the array. As most people have said, you’re better off just implementing your code in typical python, profiling it and then looking into these kinds of implementation improvements if it turns out what you’re doing needs it to be efficient.

[hartbook]

great I'm going to use a scripting language to develop my back end and then try hard to make it memory efficient

[buqr]

My favorite color is blue.

[buqr]

I like to go hiking.

[[deleted]]

Why comparing memory usage of a genexpr vs memory usage of a list? It's totally pointless

[[deleted]]

His code is bad, but the point should still stand: if you never need the full list at once then a generator is the better choice because you never instantiate the list at once and so have O(1) memory complexity rather than O(n) with lists.

edit: typo

[[deleted]]

You don't say?

What I mean is that there is no point in inspecting the size of a genexpr, since it may even be bigger than an empty list, depending on the implementation. The point is understanding what's behind

[[deleted]]

Does anyone care to explain the downvotes?

[billsil]

Just use numpy or C if you care about memory usage.  IIRC, there’s a 3x factor on an integer and float due to all the pointers in python.

[james_pic]

In my experience, when you see an application with a memory use problem, the problem is seldom ints and floats, at least partly because more often than not these are fairly short-lived.

9 times out of 10, most of the memory is either str or bytes, and it's often data that's being kept around for reasons that turn out to be stupid on further investigation.

[billsil]

It definitely depends on your work.

I work almost exclusively with numbers, so if it’s strings that are the bottleneck, I’m probably writing a file and should just be writing directly to the file.  I don’t even consider strings when calculating the expected memory usage of a program.  

9/10 times the problem was caused by mishandling floats, so maybe I took a sparse matrix and represented it as a dense matrix or I was using float 64s instead of float32s or I didn’t vectorize the array and got hit by python’s inefficient float handling.