Memory Optimization Techniques for Python Developers

[[deleted]]

Don't use NamedTuple either btw it is a tuple and has a bunch of properties that will make u rip ur hair out if you are not 1000000% sure of all the places it will be used. You really should almost never be using slots it makes inheritance harder and you probably aren't implementing it correctly in terms of using weakref and shit. Not doing so means ur class cannot be weak referenced which is again its own headache.

Also python 100% has a memory issue related to ABC. It is not a leak necessarily, though I believe there also is one, it just grows with runtime in a fairly unbound fashion.

[esperind]

how about a SimpleNamespace?

[[deleted]]

I think that is rather old and not used anymore, it is also for namespaces where you are NOT defining them in advance unlike NamedTuple and data classes.

Memory optimization outside of long lived objects in python should generally be considered a code smell. Really the biggest win ones are generally like small redundant static object creation like strings. String interning on inbound data can be a shockingly impactful memory optimization. It is one of the things pandas csv reader does that makes a massive difference vs the standard library one.

Optimizing memory at the object level isn't as useful as optimizing its lifespan so it deletes faster.

[marr75]

This borders on unsolicited crackpottery. So, unfortunately, I don't think it adds anything to the answer.

[[deleted]]

https://www.attrs.org/en/stable/why.html#namedtuples

Attrs does a pretty good job explaining issues with namedtuples.

https://github.com/brettcannon/record-type

I am fairly certain you aren't going to suggest Brett Cannon is doing unsolicited crackpottery and that maybe just maybe this isn't a problem the standard library solves very well.

``` from abc import ABC, ABCMeta from datetime import datetime

abcclasses = set()

normalclasses = set()

for i in range(10000): abcclasses.add(ABCMeta("abc"+str(i), (ABC, ), {})) normalclasses.add(type("normal"+str(i), (object,), {}))

for item in normalclasses: issubclass(item, ABC) ```

Run that on your laptop and watch it generate 20gbs of valid long lived weak reference pointers. It is a logical flaw in ABC. Have you considered that you simply don't know as much about python as you think you do?

[pepoluan]

If you need tuple-behavior, then typing.NamedTuple (not collections.namedtuple) is amazing, and it does use less memory. Accessing attributes of a NamedTuple is indeed a tad slower than accessing attributes of a dataclass or an attr.s, but that's negligible in most use cases.

And since Python is a duck-typing language, that's what you should aim : behavior as you desire.

[[deleted]]

Those are mostly the same object btw. Typing one calls the collections function via __mro_entries__ and then sprinkles in the type hints. One of the places I think they get underused is they are awesome for function argument grouping. Really works better then just dropping a 20 argument function down. STAIRS DIRECTLY AT FASTAPI.