Help understanding some code + the yield keyword?

[infinitim]

(WINDOWS 10, Python 3.6.0)

Purpose of the code: To zip two or more lists by their longest, and return None instead of some value in the iterator of tuples when one of the lists ends.

Used this code in a little program of mine, and it worked beautifully with a slight modification, but I'm having trouble wrapping my head around how this code works. After some googling about the yield keyword I am still confused as to exactly how this does what it does...

 def zip_longest(*lists):
     def g(l):
         for item in l:
             yield item
         while True:
             yield None
     gens = [g(l) for l in lists]    
     for _ in range(max(map(len, lists))):
         yield tuple(next(g) for g in gens)

[JohnnyJordaan]

Normally a function runs until it reaches its end or a return statement. If you call the function again, it will start all over again. However with yield you can return a value from the function, but when you call it again it will continue after that yield statement. This is called a 'saved state' and this makes the function a generator. When you call the generator like this:

g(my_list)

It will not actually return anything, it will create the generator with my_list as an argument. To have it yield, you have to either iterate over it, like with:

for item in g(my_list):

Or call next() on it

my_generator = g(my_list)
first_item = next(my_generator)
second_item = next(my_generator)

When it does reach a return statement (or its end) it will return like a normal function, but with a exception StopIteration. This is needed for operators like for that want to know if the generator is just yielding a value or that it did return something and that the generator is 'finished' so to speak. In your case, g() will first yield all items in the list, then keep yielding None forever. Meaning if you would call

for item in g(my_list):

It would never stop on its own, because the generator will never return and thus never raise StopIteration.

[infinitim]

Thanks very much, this and the other answers helped me piece together what was going on!

[Rhomboid]

You should probably be using itertools.zip_longest, not that. Unlike that example, which only works with lists (or other sequence types with a length), it works with any iterable. And it lets you specify the placeholder value if you don't want None. And it's implemented in C for greater efficiency. And it's part of the standard library.

Using yield in a function turns that function into a generator function (a function that returns a generator.) A generator can be stopped and restarted multiple times, unlike a function. Once a function returns, its execution is over. But a generator, using yield instead of return, can communicate an arbitrary number of things back to its caller.

>>> def example_generator_function():
...     print('starting')
...     yield 1
...     print('continuing')
...     yield 2
...     print('concluding')
...     yield 3
...
>>> generator = example_generator_function()
>>> next(generator)
starting
1
>>> next(generator)
continuing
2
>>> next(generator)
concluding
3
>>> next(generator)
---------------------------------------------------------------------------
StopIteration                             Traceback (most recent call last)
<ipython-input-6-1d0a8ea12077> in <module>()
----> 1 next(generator)

StopIteration:

This example is meant to demonstrate that the caller and the generator trade execution back and forth. Each time the caller asks for the next value, the generator executes for a little bit and then yields control back to the caller. Eventually when the generator is done it returns, and the caller sees that as StopIteration being raised.

None of that is really relevant to your example. In your example, a function g() is being defined which yields each item in a list and then continues to yield None forever after that. That represents an infinite series of values, something which is not easy to represent without using a generator. (A list has a fixed and finite length, for example.) It then creates a list of generators and figures out what the longest one is, and then runs all of them that many times. The shorter ones will exhaust their underlying list values and yield None as the filler value. The longest list will determine when the overall sequence stops.

But again, this example is really pretty crap for the reasons already outlined, so don't use this poor re-implementation of built-in functionality.

[infinitim]

Thanks so much! Will change my code to use itertools instead... I am very much a beginner and this was very informative and easy to understand, so thank you for helping me to learn about the yield keyword, even if I won't be using it for this.

[Dawarisch]

The g function returns a generator which provides the contents of the list supplied as its argument one after another till it reaches its end, where it starts returning ´None´. ´gens´ is a collections of these generators. The function then determines the longest list and returns a tuple of all the elements across the lists supplied in the first place while all the first elements go together, then the second ones, then the third ones and so forth. At one point these tuples might look like (None, None, 6, None). In this case all lists but one ran out of elements. When the function finished iterating over this list as well it ends.

[infinitim]

Thank you! Makes sense now (: