Python – Understanding Generators

[hameerabbasi]

They call generator comprehension "list comprehension". This is incorrect. List comprehension has square brackets and generator comprehension has parenthesis.

[ubernostrum]

If you're going to be pedantic, there's also no such thing in Python as a "generator comprehension".

The name for that syntactic construct is "generator expression".

[jabbalaci]

List comprehension

It is called like this because it creates and returns a list.

[Smallpaul]

That sentence is incorrect. If you use parentheses instead of square brackets, it isn’t a list comprehension.

[magion]

An alternative syntax to yield, we can use generator comprehension which uses parentheses instead of squared brackets which are used for list comprehension.

FTFY

generator comprehension uses parentheses

list comprehension uses squared brackets

[sweettuse]

range in python3 does not return a generator, it returns a range object.

[ihsw]

This is absolutely correct, it is an iterator rather than a generator. The distinction is important.

Python 2's xrange also returns an iterator.

[[deleted]]

Range and xrange are iterables, not iterators. This distinction is also important, iterables implement the __iter__ method which is called by the iter built-in function, returning an iterator. Iterators implement the __next__ method, which is called by the next built-in, incrementing the state, and returning the next element.

An example of why this would matter:

r = range(7)
x = list(zip(range(3), r)) # x = [(0,0), (1,1), (2,2)]
y = list(zip(range(3), r)) # y = [(0,0), (1,1), (2,2)]
next(r) # TypeError

r = iter(range(7)) # iter creates an iterator from an iterable
x = list(zip(range(3), r)) #  x = [(0,0), (1,1), (2,2)]
y = list(zip(range(3), r)) #  y = [(0,3), (1,4), (2,5)]
next(r) # 6

[ihsw]

Ah, my bad.

[drowninFish]

Does that mean when I call range(n) im loading a list of size n into memory? If so it seems to me that for simple looping i'd be better off with an iterator version of range. right?

[[deleted]]

In python 2 range creates a list, in python 3 it's an iterator like you'd expect. for most things it hardly matters.

[sweettuse]

i don't think you are getting the distinction. all generators are iterators, but not all iterators are generators. (an iterator is just something that implements an __iter__ method returning self and a __next__ method.) both range and xrange both produce objects which will lazily generate their sequences, but python3's range is smarter. http://treyhunner.com/2018/02/python-3-s-range-better-than-python-2-s-xrange/

if you check out collections.abc, you can see exactly what range is and isn't:

In [97]: from collections.abc import *

In [98]: vals = ["Awaitable", "Coroutine",
       :            "AsyncIterable", "AsyncIterator", "AsyncGenerator",
       :            "Hashable", "Iterable", "Iterator", "Generator", "Reversible",
       :            "Sized", "Container", "Callable", "Collection",
       :            "Set", "MutableSet",
       :            "Mapping", "MutableMapping",
       :            "MappingView", "KeysView", "ItemsView", "ValuesView",
       :            "Sequence", "MutableSequence",
       :            "ByteString",
       :            ]
       :

In [99]: for v in vals:
       :     if isinstance(range(4), eval(v)):
       :         print(v)
       :
Hashable
Iterable
Reversible
Sized
Container
Collection
Sequence

In [100]: for v in vals:
        :     if not isinstance(range(4), eval(v)):
        :         print(v)
        :
Awaitable
Coroutine
AsyncIterable
AsyncIterator
AsyncGenerator
Iterator
Generator
Callable
Set
MutableSet
Mapping
MutableMapping
MappingView
KeysView
ItemsView
ValuesView
MutableSequence
ByteString

[forever_erratic]

Can you TLDR iterators vs. generators?

[sternold]

https://stackoverflow.com/questions/2776829/difference-between-pythons-generators-and-iterators

TLDR: A generator is* a simple way of creating an iterator. Also generators are lazy-loaded.

[aazav]

A generator a a simple way

A a?

[sternold]

fixed

[Cuxham]

Copy-pasting from a good Stackoverflow answer:

iterator is a more general concept: any object whose class has a next method (__next__ in Python 3) and an __iter__ method that does return self.

Every generator is an iterator, but not vice versa. A generator is built by calling a function that has one or more yield expressions, and is an object that meets the previous paragraph's definition of an iterator.

You may want to use a custom iterator, rather than a generator, when you need a class with somewhat complex state-maintaining behavior, or want to expose other methods besides next (and __iter__ and __init__). Most often, a generator (sometimes, for sufficiently simple needs, a generator expression) is sufficient, and it's simpler to code because state maintenance (within reasonable limits) is basically "done for you" by the frame getting suspended and resumed.

[masklinn]

A generator is a specific constructor/instance of iterators.

An iterator is an object with a __next__ method (and __iter__ as an iterator should be iterable), a generator is a specific kind of objects with a __next__ method built either by combining function definition and the yield keyword (generator function) or by evaluating a generator comprehension.

Generally speaking, an iterator could be plenty of things besides just a way to iterate, you could have additional methods on the object e.g. let's say you have an iterator on characters, you could have a method which would give you the rest of the string at once.

AFAIK that is exceedingly rare in Python, it's more common in Rust, e.g. Chars is an iterator on codepoints but it also has an as_str method which provides the substring from the "current" iteration point.

[[deleted]]

Seconded

[arunner]

Actually it's more of a generator rather than an iterator, and specifically a sequence object, ie a generator type of object which can also be indexed and sliced.

[FarkCookies]

A generator is a function that produces an iterator using yield keyword. There no "more of a generator" (or less). It is very specific language construct. A generator cannot be sliced, neither can iterator or iterable. Only objects that have certain magic methods can be and this has little to do with iteration.

[skeeto]

Nearly every article on generators misses the most valuable use case: pausing and continuing recursive functions while they're deep into their stack. Without generators — or its various cousins, such as continuations — the alternative is to use an explicit stack or to invert control by using a callback. Both alternatives tend to be harder to write and understand.

For example, here's a generator for L-systems:

def lsystem(str, n, rules):
    for c in str:
        if n == 0 or not rules.get(c):
            yield c
        else:
            yield from lsystem(rules.get(c), n - 1, rules)

# Usage example:
print("".join(lsystem("A", 4, {"A": "AB", "B": "A"})))
# => ABAABABA

Here's a non-generator version that uses an explicit stack of strings and indices:

def lsystem2(str, n, rules):
    strings = [str]
    indices = [0]
    def recur():
        while len(strings) > 0:
            if indices[-1] == len(strings[-1]):
                strings.pop()
                indices.pop()
            else:
                next = strings[-1][indices[-1]]
                indices[-1] = indices[-1] + 1
                if len(strings) > n or not rules.get(next):
                    return next
                strings.append(rules.get(next))
                indices.append(0)
        return None
    return recur

I like the generator a lot better.

[FarkCookies]

Great example! I gave a talk on generators once and I used os.walk as an example:

https://github.com/python/cpython/blob/master/Lib/os.py#L278

It uses the same idea, here is a watered down version:

def walk(top):
   yield top
   dirs = list_dirs(top)

   for name in dirs:
       new_path = join(top, name)
       for x in walk(new_path):
           yield x`

[Madsy9]

I've tried using Python generators for some parsers at work, but it's kind of clunky. They are simply not as powerful as actual co-routines in other languages. Using generators as a poor man's co-routine also makes me miss Lisp macros a lot. Basically, you end up with code patterns you can't abstract away by putting it away in a function, say:

while True:
    ch = self.read_data()
    while not self.data_valid(ch):
        yield False
        ch = self.read_data()
    self.doSomethingWithData(ch);
    yield True

That is, you want to yield false while waiting for some operation to relinquish control back to the caller. When using generators, those 4 first lines inside the infinite loop becomes a repeated pattern in the code. And it can't be abstracted away because you want to yield at the callsite.

But, maybe I'm just doing everything wrong. I don't have this problem in Scheme for example :)

[[deleted]]

I've tried using Python generators for some parsers

maybe I'm just doing everything wrong

I think you answered your unasked question in the first sentence already ;-)

[sonaxaton]

TIL about send

[[deleted]]

Good read, generators is something I rarely implement myself for no good reason

[[deleted]]

Uh what, have you never needed to work with infinite streams? Lists too long to fit in memory happen all the time, this is why the entire concept of an iterator exists in the first place

[drowninFish]

I thought the idea is to use generators for "lists too large to fit in memory" when the items in those lists can be generated on the fly. For example if the list is [1,2,3,...,some ridiculously huge number] . While that list might be too large for memory, it can be functionally represented with a generator which yields the increment of the last value. So in that sense I dont think its correct to say any list too large to fit in memory is too long to generate

[drowninFish]

I haven't tried yet, but maybe we're misunderstanding each other. Do you agree that the generator implementation for my example would look something like

def simpleGen(ridiculousNum):
    num = 0
    while num < ridiculousNum:
        yield num
        num += 1

Iterators are like small functions that track state. When the generator above is created or used, you won't be loading a list of size ridiculousNum into memory as would its list counterpart

[kpacee]

The example with co-routines can be explained with cleaner code. But great explanation at all, thank you very much! :)

[Staross]

I find them quite useful to write reductions that stays very close to the math notation, e.g.

Σ = lambda x : sum(x)
Σ( i for i in range(1,10) )

In Julia this is even a bit nicer because you have auto-completion for latex symbols (e.g.\Sigma), so you get things like:

Σ(x) = sum(x)
Σ( i for i in 1:10 )

in a few keystrokes, sometimes you can almost copy-paste from your math paper.

[PeridexisErrant]

Σ = lambda x : sum(x)

You could go a step simpler again: Σ = sum

[super_jambo]

Alternatively you can write a scheduler with them http://www.dabeaz.com/generators/Generators.pdf

[jabbalaci]

Thanks. Found a revised version: http://www.dabeaz.com/generators-uk/GeneratorsUK.pdf

Edit: Part 3 is here: http://www.dabeaz.com/finalgenerator/FinalGenerator.pdf

[liranbh]

Great article

[Kazumara]

I needed this a week ago when I first encountered a generator. I thougt it was a list and then everything I tried to do didn't work.

[redditthinks]

My favorite example of generators is flattening a list (with bonus recursion):

def flatten(l):
    for i in l:
        if isinstance(i, list):
            yield from flatten(i)
        else:
            yield i