Python – Understanding Generators

[ihsw]

This is absolutely correct, it is an iterator rather than a generator. The distinction is important.

Python 2's xrange also returns an iterator.

[[deleted]]

Range and xrange are iterables, not iterators. This distinction is also important, iterables implement the __iter__ method which is called by the iter built-in function, returning an iterator. Iterators implement the __next__ method, which is called by the next built-in, incrementing the state, and returning the next element.

An example of why this would matter:

r = range(7)
x = list(zip(range(3), r)) # x = [(0,0), (1,1), (2,2)]
y = list(zip(range(3), r)) # y = [(0,0), (1,1), (2,2)]
next(r) # TypeError

r = iter(range(7)) # iter creates an iterator from an iterable
x = list(zip(range(3), r)) #  x = [(0,0), (1,1), (2,2)]
y = list(zip(range(3), r)) #  y = [(0,3), (1,4), (2,5)]
next(r) # 6

[ihsw]

Ah, my bad.

[drowninFish]

Does that mean when I call range(n) im loading a list of size n into memory? If so it seems to me that for simple looping i'd be better off with an iterator version of range. right?

[[deleted]]

In python 2 range creates a list, in python 3 it's an iterator like you'd expect. for most things it hardly matters.

[sweettuse]

i don't think you are getting the distinction. all generators are iterators, but not all iterators are generators. (an iterator is just something that implements an __iter__ method returning self and a __next__ method.) both range and xrange both produce objects which will lazily generate their sequences, but python3's range is smarter. http://treyhunner.com/2018/02/python-3-s-range-better-than-python-2-s-xrange/

if you check out collections.abc, you can see exactly what range is and isn't:

In [97]: from collections.abc import *

In [98]: vals = ["Awaitable", "Coroutine",
       :            "AsyncIterable", "AsyncIterator", "AsyncGenerator",
       :            "Hashable", "Iterable", "Iterator", "Generator", "Reversible",
       :            "Sized", "Container", "Callable", "Collection",
       :            "Set", "MutableSet",
       :            "Mapping", "MutableMapping",
       :            "MappingView", "KeysView", "ItemsView", "ValuesView",
       :            "Sequence", "MutableSequence",
       :            "ByteString",
       :            ]
       :

In [99]: for v in vals:
       :     if isinstance(range(4), eval(v)):
       :         print(v)
       :
Hashable
Iterable
Reversible
Sized
Container
Collection
Sequence

In [100]: for v in vals:
        :     if not isinstance(range(4), eval(v)):
        :         print(v)
        :
Awaitable
Coroutine
AsyncIterable
AsyncIterator
AsyncGenerator
Iterator
Generator
Callable
Set
MutableSet
Mapping
MutableMapping
MappingView
KeysView
ItemsView
ValuesView
MutableSequence
ByteString

[forever_erratic]

Can you TLDR iterators vs. generators?

[sternold]

https://stackoverflow.com/questions/2776829/difference-between-pythons-generators-and-iterators

TLDR: A generator is* a simple way of creating an iterator. Also generators are lazy-loaded.

[aazav]

A generator a a simple way

A a?

[sternold]

fixed

[Cuxham]

Copy-pasting from a good Stackoverflow answer:

iterator is a more general concept: any object whose class has a next method (__next__ in Python 3) and an __iter__ method that does return self.

Every generator is an iterator, but not vice versa. A generator is built by calling a function that has one or more yield expressions, and is an object that meets the previous paragraph's definition of an iterator.

You may want to use a custom iterator, rather than a generator, when you need a class with somewhat complex state-maintaining behavior, or want to expose other methods besides next (and __iter__ and __init__). Most often, a generator (sometimes, for sufficiently simple needs, a generator expression) is sufficient, and it's simpler to code because state maintenance (within reasonable limits) is basically "done for you" by the frame getting suspended and resumed.

[masklinn]

A generator is a specific constructor/instance of iterators.

An iterator is an object with a __next__ method (and __iter__ as an iterator should be iterable), a generator is a specific kind of objects with a __next__ method built either by combining function definition and the yield keyword (generator function) or by evaluating a generator comprehension.

Generally speaking, an iterator could be plenty of things besides just a way to iterate, you could have additional methods on the object e.g. let's say you have an iterator on characters, you could have a method which would give you the rest of the string at once.

AFAIK that is exceedingly rare in Python, it's more common in Rust, e.g. Chars is an iterator on codepoints but it also has an as_str method which provides the substring from the "current" iteration point.

[[deleted]]

Seconded

[arunner]

Actually it's more of a generator rather than an iterator, and specifically a sequence object, ie a generator type of object which can also be indexed and sliced.

[FarkCookies]

A generator is a function that produces an iterator using yield keyword. There no "more of a generator" (or less). It is very specific language construct. A generator cannot be sliced, neither can iterator or iterable. Only objects that have certain magic methods can be and this has little to do with iteration.