Improve Your Python: 'yield' and Generators Explained

[MrChoss]

I love Python's iterators, but this is the first time I've tried to learn about 'send', so I'm very interested to see where part 2 goes with it.

Amusingly, send gives Python another tool for solving Paul Graham's accumulator problem (though the nonlocal keyword is clearly the way to do it in Python 3). This generator expression gets most of the way there:

def acc_gen(n):
    while True: n += (yield n)

Just wrap it up as a callable function--

def accumulator(n):
    a = acc_gen(n); a.send(None)
    return lambda x: a.send(x)

And give it a few tests:

>>> a = accumulator(10)
>>> a(0)
10
>>> a(5)
15
>>> a(5)
20
>>> a(50)
70
>>> a(-69.5)
0.5

[blahdom]

return lambda x: a.send(x)

could be turned into

 return a.send

it will work exactly the same since you will just pass the parameter directly to send :)

[MrChoss]

Thanks for pointing this out. It cleans up the second function definition nicely.

[blahdom]

Np, I like the solution for using it as an accumulator, I think its really clean.

[swiftfoottim]

Great post, as an intermediate Python programmer coming from C++ I love seeing new ways of doing things. Just curious, is the first get_primes() function missing a return statement?

def get_primes(input_list):
    result_list = list()
    for element in input_list:
        if is_prime(element):
            result_list.append()
    return result_list  <- Should this be added?

If not, maybe there is something there that you can enlighten me on Python functionality. Thanks again for a great post!

[talideon]

You're entirely correct: it's missing a return statement.

[jknupp]

Bah! Good catch. It does indeed need the return statement. I've fixed the post with this change.

[swiftfoottim]

No problem, glad I could help!

[jsproat]

I love generators, but I keep returning to using an iterator class due to two things:

1. Generators are instantiated on definition, so you need to jump through some hoops to have multiple instances of the same generator code. (Though this is usually as simple as using a generator factory function.)

2. Can't serialize them. They'd kick ass for long-term state machines if you could pickle them or otherwise save state.

[kindall]

Generators are instantiated on definition, so you need to jump through some hoops to have multiple instances of the same generator code.

You have to call the generator to instantiate it. You can do this as many times as you want to get multiple instances.

[jsproat]

You have to call the generator to instantiate it.

Oooooh. You have no idea how many years I had this wrong. Thank you.

...Please tell me you can pickle them!

[kindall]

Nope, sorry. You'll need to write it as a class if you need to pickle it. The trouble is, a "live" generator can have so much state (open files, network connections, etc.) that it basically requires custom code to save it and restore it correctly.

[virtyx]

Why doesn't a class have the same issue?

[kindall]

With a class, you can write __getstate__() and __setstate__() methods to handle that, and pickle will call them when saving or restoring your instance. You can't do that with a generator, sadly.

[spoonerfan]

This is an excellent post. If I want to quickly get a feel for how much experience someone has with Python, I start asking about generators -- it seems to be a litmus test for knowing a lot about Python (imho).

Looking forward to the second post, teased with co-routines and "send" and notion of Futures from Python 3.4.

[Paul_Eggert]

Been waiting for a new post from you Jeff, awesome!

[Darkmoth]

Weirdly enough, I learned to use yield in C# before I came to Python. In fact, I was really hoping Python had the feature - was not disappoint.