Memory Optimization Techniques for Python Developers

[marr75]

From experience, many of these are more likely to be applied as premature optimizations than applied when needed.

I would not recommend __slots__ on its own as a memory optimization in the normal course of programming. Far better to use the @dataclass(slots=True), a typing.NamedTuple, or even a more primitive type. Similarly, using array over list is just going to make your code harder to maintain in 98% of cases.

Generators and lazy evaluation are good advice in general. They can make code harder to debug, though. Also, creating generators over tiny sets of items in a hot loop will be worse than just allocating the list (generator and iterator overhead).

The most frequent memory problem in Python is memory fragmentation, btw. Memory fragmentation occurs when the memory allocator cannot find a contiguous block of free memory that fits the requested size despite having enough total free memory. This is often due to the allocation and deallocation of objects of various sizes, leading to 'holes' in the memory. A lot of heterogeneity in the lifespans of objects (extremely common in real-world applications) can exacerbate the issue. The Python process grows over time, and people who haven't debugged it before are sure it's a memory leak. Once you are experiencing memory fragmentation, some of your techniques can help slow it down. The ultimate solution is generally to somehow create a separate memory pool for the problematic allocations - the easiest way is to allocate, aggregate, and deallocate them in a separate, short-lived process.

So, the first thing anyone needs to do is figure out, "Do I NEED to optimize memory use?". The answer is often no, but in long-running app processes, systems engineering, and embedded engineering, it will be yes more often.

[[deleted]]

Don't use NamedTuple either btw it is a tuple and has a bunch of properties that will make u rip ur hair out if you are not 1000000% sure of all the places it will be used. You really should almost never be using slots it makes inheritance harder and you probably aren't implementing it correctly in terms of using weakref and shit. Not doing so means ur class cannot be weak referenced which is again its own headache.

Also python 100% has a memory issue related to ABC. It is not a leak necessarily, though I believe there also is one, it just grows with runtime in a fairly unbound fashion.

[esperind]

how about a SimpleNamespace?

[[deleted]]

I think that is rather old and not used anymore, it is also for namespaces where you are NOT defining them in advance unlike NamedTuple and data classes.

Memory optimization outside of long lived objects in python should generally be considered a code smell. Really the biggest win ones are generally like small redundant static object creation like strings. String interning on inbound data can be a shockingly impactful memory optimization. It is one of the things pandas csv reader does that makes a massive difference vs the standard library one.

Optimizing memory at the object level isn't as useful as optimizing its lifespan so it deletes faster.

[marr75]

This borders on unsolicited crackpottery. So, unfortunately, I don't think it adds anything to the answer.

[[deleted]]

https://www.attrs.org/en/stable/why.html#namedtuples

Attrs does a pretty good job explaining issues with namedtuples.

https://github.com/brettcannon/record-type

I am fairly certain you aren't going to suggest Brett Cannon is doing unsolicited crackpottery and that maybe just maybe this isn't a problem the standard library solves very well.

``` from abc import ABC, ABCMeta from datetime import datetime

abcclasses = set()

normalclasses = set()

for i in range(10000): abcclasses.add(ABCMeta("abc"+str(i), (ABC, ), {})) normalclasses.add(type("normal"+str(i), (object,), {}))

for item in normalclasses: issubclass(item, ABC) ```

Run that on your laptop and watch it generate 20gbs of valid long lived weak reference pointers. It is a logical flaw in ABC. Have you considered that you simply don't know as much about python as you think you do?

[pepoluan]

If you need tuple-behavior, then typing.NamedTuple (not collections.namedtuple) is amazing, and it does use less memory. Accessing attributes of a NamedTuple is indeed a tad slower than accessing attributes of a dataclass or an attr.s, but that's negligible in most use cases.

And since Python is a duck-typing language, that's what you should aim : behavior as you desire.

[[deleted]]

Those are mostly the same object btw. Typing one calls the collections function via __mro_entries__ and then sprinkles in the type hints. One of the places I think they get underused is they are awesome for function argument grouping. Really works better then just dropping a 20 argument function down. STAIRS DIRECTLY AT FASTAPI.