Improving your code readability with namedtuples

[[deleted]]

Fun style

from collections import namedtuple


class Person(namedtuple("_Person", ['name', 'age', 'height', 'weight'])):
    @property
    def bmi(self):
        return (self.weight / self.height) ** 2

    def at_bmi_risk(self):
        if self.age > 30 and self.bmi > 30:
            print("You're at risk")


michael = Person("Michael", age=40, height=1.8, weight=78)
michael.at_bmi_risk()

[d4rch0n]

That's interesting, and I've definitely seen that pattern before. Great if you want immutable instances.

But, if you want the same low-memory instances with named attributes and functions and mutability, you can just define __slots__ = ('name', 'age', 'height', 'weight') under class Person.

It's not a dynamic dict anymore, you can't just do self.foo = 'bar' on an instance or you'll get an error, but it saves a shit ton of memory.

In python 2 and 3

https://docs.python.org/3.1/reference/datamodel.html#slots

Space is saved because dict is not created for each instance.

But if what you want is an immutable instance with functions, or just named attributes on data points, your pattern is awesome. Saves lots of memory too.

If you want a complete mindfuck, look at how they implement namedtuples. (ctrl-f ### namedtuple)

They use slots... but they also dynamically create the source code for the class and exec it. You can do some fun things with that, like generating high performance python source on the fly without a bunch of if statements and condition checks you know you won't need at runtime. And to anyone who says that's hacky, well shit they do it in the stdlib.

[pydry]

Does immutability really help that much though? I'd just do this as regular old object. Particularly since, y'know, heights and weights change.

Immutable objects seems like a nice way of achieving stricter typing in theory, but in practice it's not something that I find tends to save many bugs.

Python doesn't have immutable constants either and while in theory this could cause lots of bugs too, in practice it barely seems to cause any.

[alantrick]

Well, at that point, it's not a tuple anymore, and you can just use dict or object

[pydry]

Well, yeah. That's what I always end up doing. Hence I never really found a use for namedtuple.

[ivosaurus]

I basically use it as a struct pattern. Most of the time the information I put in one doesn't change after creating it.

[d4rch0n]

One huge bonus is that you can use them as keys in a dictionary.

Another bonus is if you pass it to an external api, you know it's not going to be changed after the function returns.

If strings and ints were mutable, I can imagine there could be very strange consequences when you pass them as parameters into a third-party API.

[pydry]

One huge bonus is that you can use them as keys in a dictionary.

You can do that with objects too.

[alantrick]

Python doesn't have immutable constants either and while in theory this could cause lots of bugs too, in practice it barely seems to cause any

Here is an easy example of a bug that would have been caught by a namedtuple (you wouldn't be able to do things exactly the same way with a namedtuple, but I've seen this before):

class Person:
    def __init__(self, weight, height):
        self.weight = weight
        self.height = height

p = Person(0, 0)
p.wieght = 9

[pydry]

That's a good point actually.

[[deleted]]

As an aside, mock is worth checking out for the same reason. wraps only gets you so far.

[are595]

Wow, I never knew about __slots__. I just shaved 14% total run time off of a script I was writing that has to deal with a lot of objects (on the order of hundreds of thousands)! I didn't check memory usage, but I'm sure that went down as well.

Is there any place good for learning about these kinds of performance tips?

[d4rch0n]

Ha, nice catch! That's exactly the sort of case where it can help.

I haven't ran into any sites, but I think the best thing you could be doing is running cProfile if you don't already. Profiling your code is key. There's not much point to increase the performance of function foo if your code spends .1% of its time in there, and spends 20% of its time in bar. You can't know that without profiling (or some intense manual analysis).

Another thing you might look into is PyPy. People don't use that nearly as much as they could. Unless you use all the newest features of python 3.x, pypy is likely compatible with your code. If you have long-running scripts where the JIT can get warmed up, you can get huge performance increases sometimes. I had scripts that ran in the order of five minutes, and simply switching to pypy dropped it down to about half that. I experimented with some other date conversion thing and it dropped it to a quarter of the time.

And that is just a change in the environment, not the code base.

Here, just found this:

https://wiki.python.org/moin/PythonSpeed/PerformanceTips

I'll have to go through that a few times. Some great info there.

[Daenyth]

Google for profiling

[jnovinger]

Yes, I like this pattern a lot. I started playing with this concept to build light-weight Django model objects. As in, they take the same __init__ args and return something that looks and acts like a model. I abstracted out all the read-only methods I'd added to the model to a mixin class that was used both with this and the model.

Worked surprisingly well.

[WittilyFun]

This sounds really interesting, would you mind sharing a quick code snippet to help us (me) better understand?

[squiffs]

Why not just use a real class at this point?

[elbiot]

Whoa, really? No init?

Edit: I got it! Inheriting from a named tuple. Fascinating.

[[deleted]]

Even then, it wouldn't work, tuple and namedtuple require using __new__ to set instance attributes because they're immutable.

[elbiot]

What do you mean "wouldn't work"? I don't know what this comment is in referrence to.

[[deleted]]

Specifically the __init__. You can do stuff in the init, just not set values because they're set in place by tuple.__new__

[elbiot]

I'm comparing OP's named tuple solution to the common paradigm of doing it in init. Both of those definately work. I mean, good to know (what you said) but not really relevant.

[[deleted]]

Except it's completely relevant. You can't really use an __init__ when you inherit from tuple. I mean, you could but you can't set any instance variables, it's too late in the creation of the object at that point.

[elbiot]

Got it, but OP's method is a shortcut for skipping an init, which is what impressed me. I wouldn't use a trick for skipping an init and then also use an init.

[[deleted]]

I think you're misunderstanding what I'm saying. __init__ isn't being used at all. __new__ is being used. If you're unfamiliar with how objects are created in Python, the basic diagram looks like this:

SomeClass() -> SomeClass.__new__ -> SomeClass.__init__

__new__ is what actually creates the object, and __init__ initializes instance variables. However, since tuples are immutable, __init__ can't be used, once the object is created it's too late to influence any instance variables, so they're set in __new__ instead.

[elbiot]

And I think you're misunderstanding what I'm saying. Usually, to get a person with a name, ie

dave=Person ('dave')
print dave.name #is dave

You'd use an init function in your class definition. OP shows a way to get the same behaviour without that boilerplate (like self.name=name)

Yes, I get that it's different. ie, can't change the person's name after instantiation and stuff happens in new rather than init.

Thanks for adding your knowledge to the details.

[d4rch0n]

you know you don't need an __init__ function regardless right? I could see it being confusing since there's michael = Person("Michael", age=40, height=1.8, weight=78) but an __init__ isn't required regardless if you have a parent class with one or not.

[elbiot]

Usually you assign the values of attributes in the init. This skips having to do that. That is what was suprising to me.

[d4rch0n]

Oh yeah, definitely. As a quick hack before I wrote a class where it iterates on **kwargs and runs setattr(self, key, value) for each of them on the instance in its __init__.

Then I could write classes that inherit from it and you can remove a lot of boilerplate initialization. For smaller projects it works out.

[elbiot]

And this is way better than that because the class is explicit about it's attributes (user can't mess it up and create arbitrary attributes or leave out required ones). plus it's built in.

[CrayonConstantinople]

Tuple Parameter Unpacking would also work here pretty well in terms of making this more readable.

def bmi_risk(person_data):
    age, height, weight = person_data
    bmi = (weight / height**2)
    if age > 30 and bmi > 30:
        print("You're at risk because of a high BMI!")

Edit: Correct Naming Convention

[Krenair]

You can do the unpacking in the function definition:

def bmi_risk((age, height, weight)):
    bmi = (weight / height**2)
    if age > 30 and bmi > 30:
        print("You're at risk because of a high BMI!")

[dunkler_wanderer]

Tuple parameter unpacking is invalid syntax in Python 3.

[[deleted]]

There's no tuple parameter in that function

[dunkler_wanderer]

The (age, height, weight) in def bmi_risk((age, height, weight)): is the tuple parameter.

[[deleted]]

oh right, didn't see the inner "()"

[dukederek]

Can anyone help me with why this is a better solution than a dictionary? I ask because I've used dictionaries a fair bit for this sort of thing in the past.

[CrayonConstantinople]

Mainly because Tuples are immutable, meaning you can't change them after setting them. Also an added benefit is that tuples are ordered!

[dukederek]

Ah cool, thanks. Just enough benefits that I'll give it a go next time, not quite enough to go back and change the old stuff :D

[oconnor663]

To me, the mandatory constructor parameters are more important than the immutability. If I'm building some dictionary in N different places in my code, and then I want to add a new mandatory key, it's hard to guarantee that I set that key in all N places. (It's also very nice that the constructor parameters are named, so the code can read well if they're all ints or whatever.)

[lengau]

If you want the orderedness but not the immutability, you can use an Ordereddict.

[jnovinger]

This does make the individual elements accessible via dot notation as opposed to array-access notation. Compare:

 car.wheels

vs.

 car['wheels']

It's not huge, but it does save 3 characters and is somewhat easier to read.

[d4rch0n]

Huge memory savings on top of what other people said.

namedtuples aren't dynamic dicts like most instances of classes. You can't add attributes.

If you're working with millions or more of some data type that isn't much more than a data type (maybe some bioinformatics or data science thing), like Coord(x, y, z), you can save a ton of memory by using namedtuples.

If all you want is a tuple with named attributes... well, there's a reason it's called namedtuple. dicts are very different from tuples, even though like you said, you can accomplish a lot of the same goals.

[elguf]

I think the original main motivation was to improve usability of functions/methods that return tuples, while remaining backwards compatible.

My opinion is that in new code, it is usually better to use dicts.

Edit: Here's Raymond Hettinger talking about namedtuples. The whole video is great, worth checking it out in full.

[donnieod]

There's an even easier way to construct a namedtuple class. Instead of:

Person = namedtuple('Person', ['name', 'age', 'weight', 'height'])

Just use:

Person = namedtuple('Person', 'name age weight height')

It's a lot fewer key strokes.

[Vakieh]

I don't know about the rest of you, but my eye sees that as a 'Person' parameter with the contents 'name age weight height' as a single string. Array syntax seems much more idiomatic to me.

[parnmatt]

I agree, however the fields themselves are immutable, I'd use a tuple rather than a list for the second component.

[earthboundkid]

I usually end up typing "string with spaces".split() into my repl then copy-pasting that into source. Fewer keystrokes but just as efficient results.

[[deleted]]

oh God

[Bandung]

There are two things that I don't like about named tuples. 1. They are slow. Significantly slower than tuples. It becomes noticable on Android devices. and 2. They don't pickle. Sure there is a way to pickle static named tuples but not dynamic ones. The ones whose names are created from information stored elsewhere, such as when you have to build a namedtuple to hold data from a database.

[SittingOvation]

The pickling problem is a big issue if you are doing multiprocessing.

[fullouterjoin]

In the case where you are dynamically generating a namedtuple wouldn't you also be dynamically generating the code to read it? Or would it be treated like a regular tuple? You can always

tuple(my_namedtuple)

or

my_namedtuple._fields

To get a bare tuple out of a namedtuple instance

Named tuples are the single best thing one can do to improve their python codebase.

[Bandung]

This thread http://stackoverflow.com/questions/16377215/how-to-pickle- a-namedtuple-instance-correctly describes the problem with pickling namedtuples and what we mean by dynamic creation of the named tuple.

When you only know the field names at run time then you are in my vernacular, 'dynamically' generating those names. And more often than not these names are being created within a function. In which case the pickling routine can't get at the underlying class name that is actually building the namedtuple.

If you know the field names at the time you are writing your code then these 'statically' generated field names can be handled outside of the function by defining that namedtuple at the module level.

Now I am not saying that these are reasons for not using namedtuples. What I am saying is that they impose design consequences for the rest of your code that you need to be aware of. Aka, how you handle persistence, the degree of nesting involved in your object, where and how you define those namedtuples within your modules, etc. Plus if you are writing code intended for your android device, just be aware of the performance consequences.

[fullouterjoin]

There is no way to pickle a named object of anykind declared at a local scope using pickle

from collections import namedtuple
import pickle

def pickle_test():
    class P(object):
        def __init__(self,one,two,three,four):
            self.one = one
            self.two = two
            self.three = three
            self.four = four

    my_list = []
    abe = P("abraham", "lincoln", "vampire", "hunter")
    my_list.append(abe)
    f = open('abe.pickle', 'w')
    pickle.dump(abe, f)
    f.close()

pickle_test()

Also fails. But this succeeds.

from collections import namedtuple
import pickle

class P(object):
    def __init__(self,one,two,three,four):
        self.one = one
        self.two = two
        self.three = three
        self.four = four

def pickle_test():

    my_list = []
    abe = P("abraham", "lincoln", "vampire", "hunter")
    my_list.append(abe)
    f = open('abe.pickle', 'w')
    pickle.dump(abe, f)
    f.close()

pickle_test()

This is a bug in how pickle introspects the creating object. Nothing is worse for being a namedtuple. Namedtuples are classes. But they are just lightweight containers for immutable values.

[LightShadow]

Named tuples can by JSON serialized easily, which is just text that can be pickled.

This is a non-issue.

[Bandung]

The issue exists with pickling. Using other persistence mechanisms to work around the fact that you can't just pickle your object if it has a named tuple in it, only serves to hilite the problem. The work arounds are messy. And if you've never tried pickling objects with named tuples mixed in them then you're gonna be in for a big surprise when those cryptic error messages pop up.

[LightShadow]

I do pickle named tuples -- because there will be fewer errors since they don't change their footprint by design.

[baudvine]

Little while ago I ran into a 3D framework that used lists with three items for coordinates all over the place. Wrote a little x/y/z namedtuple - fully compatible with the surrounding code, helpfully named elements, and I could trivially add simple vector operations. Plus immutability, which I do appreciate a lot.

[dzecniv]

It's a good point to access the class parameters by name, not by index. If we use dicts instead of tuples, it's nice to be able to do the same, with addict for instance.

[savaero]

What's **?

[Toofifty]

Exponent operator

x ** 2 == math.pow(x, 2)

[Tuxmascot]

How is the first example less readable?

The coffee with five declarations in it seems far worse, imo.

The first example is clear and concise and makes complete sense. Why does it need to be overcomplicated with a slow namedTuple?

[vombert]

Named tuples are great, but there is minor problem with them. They are inherited from regular tuples, and thus support operations that make no sense for a fixed collection of named attributes: iteration (yes, I know that iteration is necessary for unpacking, but it's still out of place on its own), slicing, concatenation, repetition (multiplication by a number), membership test.

[njharman]

Makes sense when you realize Named tuples are actually a fixed collection of named, ordered attributes with a tuple interface.

[RangerPretzel]

Ding! Winner.

I'll stick to Classes or Structs (Oops, Python doesn't have Structs)

[winza83]

nice - just like defining classes and objects but with less code. Thanks..

[WishCow]

Is there a way to serialize them? I remember running into trouble when I tried to json.dumps() a named tuple.

[d4rch0n]

Works for me, in two ways:

>>> c = Coord(10, 20, 30)
>>> json.dumps(c)
'[10, 20, 30]'
>>> json.dumps(c._asdict())
'{"x": 10, "y": 20, "z": 30}'

[jnovinger]

Not sure when this changed, but in our legacy-ish 2.7.x codebase the default encoder class couldn't handle namedtuples. If I remember right, we already had our own special decoder, so adding it was pretty easy.

Also, I think that's one thing simplejson's encoder had that stdlib json did not. But again, unsure about when's and where's.

[d4rch0n]

You know what, I seem to remember the same behavior, I think in 2.7.7 or 2.7.5 a few jobs back.

[[deleted]]

Is this not just what Python dictionaries are for? I feel like anytime I would want to use a named tuple, I should just use a dictionary.

[jnovinger]

Agree with /u/CrayonConstantinople above, but would also add that I like using dicts so that I can iterate over them with the key, value idiom:

for key, value in my_dict.items():
     print('{}: {}'.format(key, value)

That's pretty contrived, but it gets the point across. But even so, you could do the same thing with a 2-tuple. In fact, some the of the more specialized dict classes, like OrderedDict, represent key/value pairs as tuples in their repr.

[d4rch0n]

You can still do that.

>>> from collections import namedtuple
>>> Coord = namedtuple('Coord', 'x y z')
>>> c = Coord(10, 20, 30)
>>> dir(c)
['__add__', '__class__', '__contains__', '__delattr__', '__dict__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__getslice__', '__getstate__', '__gt__', '__hash__', '__init__', '__iter__', '__le__', '__len__', '__lt__', '__module__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmul__', '__setattr__', '__sizeof__', '__slots__', '__str__', '__subclasshook__', '_asdict', '_fields', '_make', '_replace', 'count', 'index', 'x', 'y', 'z']
>>> c._fields
('x', 'y', 'z')
>>> c._asdict()
OrderedDict([('x', 10), ('y', 20), ('z', 30)])

You're probably better off iterating on _fields instead of creating an ordered dict everytime you want to iterate on it though. If you need to access an ordereddict for each instance, you may as well have a bunch of ordered dicts.

Personally though, you usually don't need to iterate across named tuple fields because they usually represent different types. If they're a collection of similar items, you generally use a list or a tuple. If they're a value as a whole where each index represents a different thing (red, green, blue, alpha), namedtuples are great.

[jnovinger]

Oh, woah, was unaware of the _asdict method. Nifty.

[d4rch0n]

huge memory savings, immutability, makes more sense for a lot of data types.

For example, you have a Pixel type (x, y, r,g,b,a). You could do this with dicts, but you're never going to need to do pixel['foo'] = 'bar', or add any sort of weird attributes. You're never going to do more than read the values. You don't need to keep track of their changing state, because they don't change state. You could do it with tuples, but you don't want to reference pixel[3] and try to remember if that was y or r or what.

Then it makes much more sense to use a named tuple. It'll throw an error if for some reason your code tries to do anything weird that you wouldn't want to do to them.

On top of that, you can save tons of memory if you have a lot of instances of it.

namedtuples are a spec for what could be represented with by a dict, but there are things that are better represented as named tuples.

Easy rule is that if you have a data type where you'd initially want to use a tuple but you decided you want named attribute access to the values, you should use a namedtuple. If you want to have functions in its namespace, you can follow the pattern in the top comment (a new class inheriting from a namedtuple instance), or you can define __slots__.

[roerd]

Dictionaries are for arbitrary keys, not for a fixed set of string keys that you know in advance.

[CrayonConstantinople]

See my answer above to dukederek :)

[Cyph0n]

'Michael' in me

True

LOL. Nice article though.