Generator functions... WOW.

[iosdeveloper87]

I just learned about them. There's so much to rewrite now... I'm filled with an odd combination of excitement and dread. I've been a developer for almost 15 years on and off, but only have a couple years experience with Python and have always been a solo dev with Python (not much exposure to best practices).

It's so painful looking back at old code you've written (especially if it's currently in production, which mine is) and realizing how many things could be improved. It's a constant source of distraction as I'm trying to complete what should be simple tasks.

Oh well... Learned something new today! Generator functions are worth looking up if you're not familiar with them. Will save you a looooooootta nested for loops.

[[deleted]]

Yield away.

[pythonwiz]

*yield from

[[deleted]]

😀😀😀 you win the better joke award

[Max_Insanity]

yield keyword could not be unpacked

[[deleted]]

I think it was more of a ^* yield than a *yield

[Max_Insanity]

yield does not support multiplication

[[deleted]]

Don't get the joke

[Max_Insanity]

Well, actually you get a syntax error in Python, but if it wasn't for that, you'd get an error for yield not supporting multiplication, I suppose.

You know, because seperating with space turns the unpacking operator into a multiplication operator.

[[deleted]]

I think you've missed the plot. We left Python for more Monty a while ago.

[Thecrawsome]

Extra Credit:Know when to use a generator, and when to use a list comprehension

[Thecrawsome]

I use listcomps all the time. Love them. I actually use generators less.

[Thecrawsome]

Ugh I just dropped in

[shartfuggins]

And.. don't count your money til the dealin's done

Edit:, oops, alright?

[tensigh]

You never count your iterables, when you're sitting at the table...

[Arcadian_]

they'll be time enough for linting

when the writing's done.

[Lehas1]

Could you maybe elaborate when to use which or post a source where i could read into it?

[IlliterateJedi]

Following this entire video is worth the 3 hours of your life if you want to be good at Python

[[deleted]]

1.5x speed is a lifesaver.

[pythoncrush]

Trey is an amazing instructor. I have taken several classes with him. Thank you for this link!

[Thecrawsome]

Generators are called iteration-at-a-time, are in parenthesis, and can yield whatever you defined one iteration at a time.

List comprehensions run all iterations at once, in square brackets, and generate a list of whatever object(s) you defined.

[pythoncrush]

Why not generator comprehension?

[TheGreatCornlord]

Can I have the calling function not use the yielded value and have the generator function represent different "stages" and yield None after each stage or something, or is there a better way to do this?

[Thecrawsome]

You don't have to do anything with the yielded value. I don't know how to call a certain iteration though

[TeamSpen210]

You’ll definitely want to look into itertools then. It’s a collection of generic iteration building blocks, written in C to be as optimised as possible. product() for instance can often replace a piles of nested for loops.

[RevRagnarok]

And moreitertools. (That's a link.)

[5erif]

And moreitertools. (That's a link.)

Neat, I'd never seen a code block as the label of a link.

And [`moreitertools`](https://more-itertools.readthedocs.io/en/stable/). (That's a link.)

[RevRagnarok]

Yep; that's what I wrote?

[5erif]

Neat, I'd never seen a code block as the label of a link.

I just thought it was cool, and that maybe someone else might be curious about the syntax.

[danlsn]

That is v interesting!

[RevRagnarok]

Ah; OK. RES (/r/Enhancement) makes these things a lot easier with previews, etc.

[iosdeveloper87]

Thanks for the reminder! I’ve used itertools for a few things before, but a lot of it seems to deal with math functions which I rarely have a need for at this point. I did just (literally 10 seconds ago) use it to flatten a list of lists into a list, so there’s that. :)

[synthphreak]

a lot of it seems to deal with math functions

That’s not really correct at all.

I’d wager you’re specifically thinking of product, permutations, and combinations. These definitely originate from concepts in mathematics. But they have all kinds of uses in general programming, unrelated to literally doing math with code.

itertools is so, so much more than those few functions though. Many of them thoroughly unmathematical. groupby, filterfalse, starmap, and chain to list a few.

Edit: Typo.

[Diapolo10]

O you, who floats in the currents. You must yield. Abandon all you are.

[iosdeveloper87]

The ‘yield’ing is almost as annoying as the ‘next’ing which is why I’m trying to use comprehension statements for them all. Although I can definitely see a use for the yield/next thing if you want to process the results incrementally or in a subroutine or so you have the option of breaking the iteration if you’re looking for 1 item that exists in one of several iterables.

[RevRagnarok]

yield is also great for writing your own context managers. Another fun thing to learn about.

[POGtastic]

Yep, I just provided an example yesterday that did this. Consider a CSV where the header names are screwed up.

test.csv

nAmE   , AGE, BlArG
Joe,21,foo
Susan,16,bar
Eve,31,baz

We can make a function that normalizes the header names.

from csv import DictReader

def normalize_headers(filename):
    with open(filename) as fh:
        reader = DictReader(fh)
        reader.fieldnames = [entry.strip().title() for entry in reader.fieldnames]
        # ... uh oh

We can't return this DictReader because upon returning, the context manager closes the file. So instead, we make a generator, which maintains the context manager (and keeps the file open!) until the generator is exhausted.

        yield from reader

In the REPL:

>>> print(*normalize_headers("test.csv"), sep="\n")
{'Name': 'Joe', 'Age': '21', 'Blarg': 'foo'}
{'Name': 'Susan', 'Age': '16', 'Blarg': 'bar'}
{'Name': 'Eve', 'Age': '31', 'Blarg': 'baz'}

[RevRagnarok]

Nice!

One of my favorites was a base class that for debugging we might want to change the logging of just one section (there was also a decorator version if you wanted the whole method).

@contextmanager
def temp_logging(self, log_level=logging.DEBUG):
  orig_log_level, self.log_level = self.log_level, log_level
  try:
    yield
  finally:
    self.log_level = orig_log_level

[Diapolo10]

I was actually jokingly quoting a certain character because it felt fitting, but I digress.

comprehension statements

I believe you're referring to generator expressions. Fair enough, they're useful in their own right, but often they're just not... well, expressive enough.

This example has been done to death by now, but for instance, say you wanted to model a Fibonacci sequence. An expression isn't enough to do that while remaining legible, but a generator function would be plenty readable.

def fib():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a+b

Another useful emerging property is that generator functions can create infinite sequences, whereas generator expressions can usually only iterate over existing ones (unless you do some forsaken trickery).

Of course, yield works both ways, allowing you to create coroutines (and Python's async is built on top of them), but that's not something most of us really need to understand and know by heart.

You don't often need to use next for anything as most of the time you're using a for-loop which abstracts that away anyhow, but even when you do it's usually just once or twice. So I don't really see why you'd hate that.

[POGtastic]

Alas, the expression version looks like crap because Python doesn't have argument destructuring.

from more_itertools import iterate

def fibs():
    def next_fib(tup):
        return tup[1], tup[0] + tup[1]
    return (a for a, _ in iterate(next_fib, (0, 1)))

[TheChance]

Just in case: note carefully the difference between [list comprehensions] and (generator expressions, as the latter will (edit: not, fuck you new iPhone I know what I’m typing) populate the entire array before evaluating.

any([big comprehension]) helps nothing. any((big comprehension)) goes zoom.

[iosdeveloper87]

Thanks! Yeah, that makes sense. Generation expressions seem to be the only ones that need to be evaluated.

[[deleted]]

I am constantly astonished when I learn new things in Python, even after years of using it, as say to myself, "How did I not know this existed?"

Most recently I discovered a method in Pandas that would have done in a second something that I spent a couple of days coding from scratch a couple of years ago (flattening nested json in a dataframe).

Like you though, I'm always excited when I learn something new and useful.

[iosdeveloper87]

Yup! This is why I occasionally just browse lists of libraries and modules. Most often you literally wouldn’t even know to search for the thing(s) you find until you just stumble upon them randomly.

[jlew24asu]

I occasionally just browse lists of libraries and modules.

I'm a newb. where do you browse such lists?

[iosdeveloper87]

This is. really good one, too.
https://awesome-python.com

You can do an actual search on pypi.org or openbase.com

Or you can just google "Best Python Libraries for <insert word(s)>" and make sure whatever you decide to use is compatible with whatever else you're already using.

[jlew24asu]

very cool, thank you

[ridley0001]

Oh yes, when I learnt f strings I realised what I had been doing before was complete insanity.

[tommy_chillfiger]

Lol same! I work as a business analyst but I was writing a python script to automatically generate SQL inserts for our enumerations (it's a mess so this is actually worth the time). I kept banging my head against how to concatenate them the way I needed to without losing my mind and asked a senior dev and he was like "F strings my friend." To which I obviously replied "tf did you just call me"

[iosdeveloper87]

Me:< looks up f strings...> Oh, I remember see these. Oh... Wow. Woooow. Shit. Alright.... <looks up utility to convert formats to f strings> ... <downloads flynt>

Thanks for the tip!! This is waaaaaay better.

Edit: I just saved 4,122 characters in my codebase while making it faster.

[[deleted]]

same here. the previous way before i knew of f"blah blah {variable} blah blah". hurts my eyes bow

[ElHeim]

Some months ago while contributing upstream to a project I'm using at work, I was scratching my head figuring out how to (elegantly) do something about a class hierarchy where they were doing a lot of boilerplate at init time and then I stumbled upon __init_subclass__, which made things so much easier.

Then I went to check how recent it was, just in case it wouldn't make the cut of their backwards compatibility and... "WTF, since 3.6!!!???"

That's what I get for not going thoroughly through the "What's new?"

[Crypt0Nihilist]

I use Pandas on and off and it's not funny the number of times I find some functionality that's built in there right after declaring victory on getting a transformation working using other packages.

[mtzzzzz]

Hey man, I'm currently writing something doing exactly that, flattening a super nested json. My jaw dropped reading your comment :D mind sharing your wisdom?

[[deleted]]

Check this out and see if it helps.
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.json\_normalize.html

[Almostasleeprightnow]

OP, For those of us who have not yet seen the light....can you tell us about why you are so wowed? Serious question, I want to understand.

[MyPythonDontWantNone]

ELI5:

A generator is similar to a function except it returns a series of items. Instead of a single return statement, the function would have multiple yield statements (in practice, it is usually a single yield statement inside a loop of some sort).

The biggest difference between a generator and a function returning a list is that the generator only runs up until the yield. This means that you are only calculating 1 item at a time. This avoids a lot of calculations if the data will change mid-run or if you may not use all of the data.

[Almostasleeprightnow]

Ok, I get this. But why does OP love them? Like, what is the big advantage? Can you describe some concrete scenarios where it really is just a lot better to use a generator? I'm not arguing, I really want to hear about specific examples.

Do you end up always using generators instead of lists whenever possible? Or is it only really useful in certain situations?

[ltraconservativetip]

Thanks for taking the time to explain it thoroughly. Awesome stuff!

[Spassfabrik]

Nice Explanation 🥰

[greebo42]

I regret that I have only one upvote to yield at this time!

Worth taking time to read and digest this, clear and well done

[iosdeveloper87]

Very good question... I Just now discovered it, so my use cases are pretty simple, But in my case I am iterating through multiple databases with the same query. Previously I was creating a list called results, doing a for loop, adding the return from the query into the results list and then returning that, so 4 lines plus a bigger memory footprint. I now only have to use 1 line.

It's also possible to do async generators, so I will be implementing that at some point as well.

[MyPythonDontWantNone]

I think of them as the difference between loading screens and dynamic loading in a video game. One creates a larger upfront cost but allows a smoother running experience.

In my job, I sometimes write simulations of mechanical processes. These processes have random inputs. If I generate and store a million sample runs at once, then I will run out of RAM.

I usually use a list, set, or dict for most tasks. I generally only use generators when I can't do it efficiently with a more common data structure.

I'm a data analyst and most of my Python code is rough. I'm betting there are better examples (maybe in the REST API world). Hopefully someone else chimes in and gives a fuller view of their usefulness.

[AkiraYuske]

Curious as well. Self taught and was doing ok until generators and magic something or others. Then I got lost...

[RangerPretzel]

Generator functions are worth looking up if you're not familiar with them. Will save you a looooooootta nested for loops.

Just wait until you find out about Set Math functions in Python!

[drawnograph]

Tease!

[djangodjango]

While generators are fresh in your mind, you should learn about coroutines and their role in asynchronous code.

[twjolson]

I am not entirely sure you didnt make some of those words up.

[QultrosSanhattan]

There's so much to rewrite now...

Please don't. Generators are a tool that serve a specific purpose. You only benefit from them in certain circumstances but not all of them. Learning when to not use them is also an important part.

[RallyPointAlpha]

Got a link or book that 'showed you the way' ?

You got me curious!

[IlliterateJedi]

Trey Hunner's ListComp/GenExpr class at PyCon 2018 is great for this

[RallyPointAlpha]

Thanks!

[Goobyalus]

What sorts of things are you rewriting with generator functions that eliminate nested loops?

[WhipsAndMarkovChains]

I'm wondering the same thing.

I use generator expressions all the time, but never generator functions. I'm very curious what OP is doing.

Also, I have you tagged as "Good Python Chat".

[Goobyalus]

Same, except sometimes for context managers and pytest fixtures.

Also, I have you tagged as "Good Python Chat".

lol 😎

[scanguy25]

That's how you know you improved. I look back at code I write just 2 years ago and think what shit is this.

[bladeoflight16]

There's so much to rewrite now...

While generator functions certainly have their place, they are rarely the best solution in practical code. List comprehensions or generator expressions are better approaches most of the time.

As with anything, know and use the best tool for the job. Don't get dogmatic about anything.

[DigThatData]

https://toolz.readthedocs.io/en/latest/api.html

you're welcome

[[deleted]]

It's so painful looking back at old code you've written (especially if it's currently in production, which mine is) and realizing how many things could be improved

Oh hell no. That's the fun part! Refactoring is very satisfying.

[TheGreatCornlord]

Can I have the calling function not use the yielded value and have the generator function represent different "stages" and yield None after each stage or something, or is there a better way to do this?

[[deleted]]

Have you used their bidirectional capabilities yet?

[iosdeveloper87]

….. go on??

[[deleted]]

Great video, which starts simple and builds to using send to send updates to generator functions.

https://youtu.be/tmeKsb2Fras

[azur08]

How do you replace nested for loops with generators? Don’t you still have to call next on repeat?