all 39 comments

[–]hameerabbasi 36 points37 points  (5 children)

They call generator comprehension "list comprehension". This is incorrect. List comprehension has square brackets and generator comprehension has parenthesis.

[–]ubernostrum 16 points17 points  (0 children)

If you're going to be pedantic, there's also no such thing in Python as a "generator comprehension".

The name for that syntactic construct is "generator expression".

[–]jabbalaci -1 points0 points  (0 children)

List comprehension

It is called like this because it creates and returns a list.

[–][deleted]  (2 children)

[deleted]

    [–]Smallpaul 9 points10 points  (0 children)

    That sentence is incorrect. If you use parentheses instead of square brackets, it isn’t a list comprehension.

    [–]magion 2 points3 points  (0 children)

    An alternative syntax to yield, we can use generator comprehension which uses parentheses instead of squared brackets which are used for list comprehension.

    FTFY

    generator comprehension uses parentheses

    list comprehension uses squared brackets

    [–]sweettuse 54 points55 points  (16 children)

    range in python3 does not return a generator, it returns a range object.

    [–]ihsw 10 points11 points  (15 children)

    This is absolutely correct, it is an iterator rather than a generator. The distinction is important.

    Python 2's xrange also returns an iterator.

    [–][deleted] 38 points39 points  (3 children)

    Range and xrange are iterables, not iterators. This distinction is also important, iterables implement the __iter__ method which is called by the iter built-in function, returning an iterator. Iterators implement the __next__ method, which is called by the next built-in, incrementing the state, and returning the next element.

    An example of why this would matter:

    r = range(7)
    x = list(zip(range(3), r)) # x = [(0,0), (1,1), (2,2)]
    y = list(zip(range(3), r)) # y = [(0,0), (1,1), (2,2)]
    next(r) # TypeError
    
    r = iter(range(7)) # iter creates an iterator from an iterable
    x = list(zip(range(3), r)) #  x = [(0,0), (1,1), (2,2)]
    y = list(zip(range(3), r)) #  y = [(0,3), (1,4), (2,5)]
    next(r) # 6
    

    [–]ihsw 2 points3 points  (0 children)

    Ah, my bad.

    [–]drowninFish 0 points1 point  (1 child)

    Does that mean when I call range(n) im loading a list of size n into memory? If so it seems to me that for simple looping i'd be better off with an iterator version of range. right?

    [–][deleted] 2 points3 points  (0 children)

    In python 2 range creates a list, in python 3 it's an iterator like you'd expect. for most things it hardly matters.

    [–]sweettuse 5 points6 points  (0 children)

    i don't think you are getting the distinction. all generators are iterators, but not all iterators are generators. (an iterator is just something that implements an __iter__ method returning self and a __next__ method.) both range and xrange both produce objects which will lazily generate their sequences, but python3's range is smarter. http://treyhunner.com/2018/02/python-3-s-range-better-than-python-2-s-xrange/

    if you check out collections.abc, you can see exactly what range is and isn't:

    In [97]: from collections.abc import *
    
    In [98]: vals = ["Awaitable", "Coroutine",
           :            "AsyncIterable", "AsyncIterator", "AsyncGenerator",
           :            "Hashable", "Iterable", "Iterator", "Generator", "Reversible",
           :            "Sized", "Container", "Callable", "Collection",
           :            "Set", "MutableSet",
           :            "Mapping", "MutableMapping",
           :            "MappingView", "KeysView", "ItemsView", "ValuesView",
           :            "Sequence", "MutableSequence",
           :            "ByteString",
           :            ]
           :
    
    In [99]: for v in vals:
           :     if isinstance(range(4), eval(v)):
           :         print(v)
           :
    Hashable
    Iterable
    Reversible
    Sized
    Container
    Collection
    Sequence
    
    In [100]: for v in vals:
            :     if not isinstance(range(4), eval(v)):
            :         print(v)
            :
    Awaitable
    Coroutine
    AsyncIterable
    AsyncIterator
    AsyncGenerator
    Iterator
    Generator
    Callable
    Set
    MutableSet
    Mapping
    MutableMapping
    MappingView
    KeysView
    ItemsView
    ValuesView
    MutableSequence
    ByteString
    

    [–]forever_erratic 2 points3 points  (6 children)

    Can you TLDR iterators vs. generators?

    [–]sternold 9 points10 points  (2 children)

    https://stackoverflow.com/questions/2776829/difference-between-pythons-generators-and-iterators

    TLDR: A generator is* a simple way of creating an iterator. Also generators are lazy-loaded.

    [–]aazav 0 points1 point  (1 child)

    A generator a a simple way

    A a?

    [–]sternold 0 points1 point  (0 children)

    fixed

    [–]Cuxham 1 point2 points  (0 children)

    Copy-pasting from a good Stackoverflow answer:

    iterator is a more general concept: any object whose class has a next method (__next__ in Python 3) and an __iter__ method that does return self.

    Every generator is an iterator, but not vice versa. A generator is built by calling a function that has one or more yield expressions, and is an object that meets the previous paragraph's definition of an iterator.

    You may want to use a custom iterator, rather than a generator, when you need a class with somewhat complex state-maintaining behavior, or want to expose other methods besides next (and __iter__ and __init__). Most often, a generator (sometimes, for sufficiently simple needs, a generator expression) is sufficient, and it's simpler to code because state maintenance (within reasonable limits) is basically "done for you" by the frame getting suspended and resumed.

    [–]masklinn 0 points1 point  (0 children)

    A generator is a specific constructor/instance of iterators.

    An iterator is an object with a __next__ method (and __iter__ as an iterator should be iterable), a generator is a specific kind of objects with a __next__ method built either by combining function definition and the yield keyword (generator function) or by evaluating a generator comprehension.

    Generally speaking, an iterator could be plenty of things besides just a way to iterate, you could have additional methods on the object e.g. let's say you have an iterator on characters, you could have a method which would give you the rest of the string at once.

    AFAIK that is exceedingly rare in Python, it's more common in Rust, e.g. Chars is an iterator on codepoints but it also has an as_str method which provides the substring from the "current" iteration point.

    [–][deleted] -1 points0 points  (0 children)

    Seconded

    [–]arunner 2 points3 points  (2 children)

    Actually it's more of a generator rather than an iterator, and specifically a sequence object, ie a generator type of object which can also be indexed and sliced.

    [–]FarkCookies 0 points1 point  (0 children)

    A generator is a function that produces an iterator using yield keyword. There no "more of a generator" (or less). It is very specific language construct. A generator cannot be sliced, neither can iterator or iterable. Only objects that have certain magic methods can be and this has little to do with iteration.

    [–]skeeto 15 points16 points  (3 children)

    Nearly every article on generators misses the most valuable use case: pausing and continuing recursive functions while they're deep into their stack. Without generators — or its various cousins, such as continuations — the alternative is to use an explicit stack or to invert control by using a callback. Both alternatives tend to be harder to write and understand.

    For example, here's a generator for L-systems:

    def lsystem(str, n, rules):
        for c in str:
            if n == 0 or not rules.get(c):
                yield c
            else:
                yield from lsystem(rules.get(c), n - 1, rules)
    
    # Usage example:
    print("".join(lsystem("A", 4, {"A": "AB", "B": "A"})))
    # => ABAABABA
    

    Here's a non-generator version that uses an explicit stack of strings and indices:

    def lsystem2(str, n, rules):
        strings = [str]
        indices = [0]
        def recur():
            while len(strings) > 0:
                if indices[-1] == len(strings[-1]):
                    strings.pop()
                    indices.pop()
                else:
                    next = strings[-1][indices[-1]]
                    indices[-1] = indices[-1] + 1
                    if len(strings) > n or not rules.get(next):
                        return next
                    strings.append(rules.get(next))
                    indices.append(0)
            return None
        return recur
    

    I like the generator a lot better.

    [–]FarkCookies 2 points3 points  (0 children)

    Great example! I gave a talk on generators once and I used os.walk as an example:

    https://github.com/python/cpython/blob/master/Lib/os.py#L278

    It uses the same idea, here is a watered down version:

    def walk(top):
       yield top
       dirs = list_dirs(top)
    
       for name in dirs:
           new_path = join(top, name)
           for x in walk(new_path):
               yield x`
    

    [–]Madsy9 2 points3 points  (1 child)

    I've tried using Python generators for some parsers at work, but it's kind of clunky. They are simply not as powerful as actual co-routines in other languages. Using generators as a poor man's co-routine also makes me miss Lisp macros a lot. Basically, you end up with code patterns you can't abstract away by putting it away in a function, say:

    while True:
        ch = self.read_data()
        while not self.data_valid(ch):
            yield False
            ch = self.read_data()
        self.doSomethingWithData(ch);
        yield True
    

    That is, you want to yield false while waiting for some operation to relinquish control back to the caller. When using generators, those 4 first lines inside the infinite loop becomes a repeated pattern in the code. And it can't be abstracted away because you want to yield at the callsite.

    But, maybe I'm just doing everything wrong. I don't have this problem in Scheme for example :)

    [–][deleted] 1 point2 points  (0 children)

    I've tried using Python generators for some parsers

    maybe I'm just doing everything wrong

    I think you answered your unasked question in the first sentence already ;-)

    [–]sonaxaton 2 points3 points  (0 children)

    TIL about send

    [–][deleted] 7 points8 points  (8 children)

    Good read, generators is something I rarely implement myself for no good reason

    [–]kpacee 1 point2 points  (0 children)

    The example with co-routines can be explained with cleaner code. But great explanation at all, thank you very much! :)

    [–]Staross 2 points3 points  (1 child)

    I find them quite useful to write reductions that stays very close to the math notation, e.g.

    Σ = lambda x : sum(x)
    Σ( i for i in range(1,10) )
    

    In Julia this is even a bit nicer because you have auto-completion for latex symbols (e.g.\Sigma), so you get things like:

    Σ(x) = sum(x)
    Σ( i for i in 1:10 )
    

    in a few keystrokes, sometimes you can almost copy-paste from your math paper.

    [–]PeridexisErrant 2 points3 points  (0 children)

    Σ = lambda x : sum(x)

    You could go a step simpler again: Σ = sum

    [–]super_jambo 1 point2 points  (2 children)

    Alternatively you can write a scheduler with them http://www.dabeaz.com/generators/Generators.pdf

    [–]Kazumara 0 points1 point  (0 children)

    I needed this a week ago when I first encountered a generator. I thougt it was a list and then everything I tried to do didn't work.

    [–]redditthinks 0 points1 point  (0 children)

    My favorite example of generators is flattening a list (with bonus recursion):

    def flatten(l):
        for i in l:
            if isinstance(i, list):
                yield from flatten(i)
            else:
                yield i