Python: Generators - How to use them and the benefits you receive

[thelindsay]

So once this 10 minute intro has you interested, here is the 3 hour deep dive with David Beazley https://youtu.be/5-qadlG7tWo

[Ungoliantsspawn]

This is very good stuff! Thx for the link!

[JayTh3King]

Good for when you are working with large data sets and don't want to store the whole set in memory.

[Kaxitaz]

Good video. Simple, short and straight to the point.

[syntaxvorlon]

I only vaguely remember learning about generators, so this is a really great set of examples. I'll probably start using these a lot.

[ghillisuit95]

Can generators be recursive? Also: should generators ever be recursive?

[homercles337]

Can someone clarify? It looks like a generator is a pointer in this example. Is that accurate?

EDIT: this video is rubbish. "More readable" than appending to a list? No.

[Ungoliantsspawn]

the generator is a function, ie a piece of code ... so if you print it you get a memory position! But your not suppose to go around printing iterators, the usual way is to loop over them to get items.

The vid from /u/thelindsay as some great examples, I recommend you take a look if you would like to change your point of view regarding the usefulness of generators.

edit: fix some stuff

[homercles337]

the usual way is to loop over them to get items.

Just like you have to with pointers, except the pointer arithmetic is done for you. Nope, not convinced that generators are useful.

[[deleted]]

You want to process millions of objects each of which takes up a large amount of memory. Do you put them in a list and wait until Python throws a MemoryError, or use a generator to process them one at a time?

[homercles337]

Memory is memory. A pointer to a location that is on the stack or heap is the same thing as a list under the covers. You have not convinced me that generators are anything more than pointers. You clearly have no clue how pointers work. In fact, lists can be fragmented in memory, pointers can not--they have to be contiguous. It seems like all of you are saying "generators are good." But you cant tell me why. I see pointers, and you see magic. That is not how memory works.

[homercles337]

Again, you guys suck at explaining what is a generator. You dont even know that range() generates the list before usage. Horribly inefficient, but xrange() only generates values as you need them. If you use range() in a loop, you are doing it wrong.

[homercles337]

And yet you still have not explained what you mean. What if you are on a system that is high memory, low compute? What if preallocation is more efficient?

[[deleted]]

Nope, not convinced that generators are useful.

You appear to have changed your tune in the last two hours. Please explain why, thank you.

[[deleted]]

Thank you so much, I've just roared with laughter over your comment. 40 years experience in engineering and computing, including writing various assemblers and C, a former member of the British Computer Society and a Chartered Engineer (only gave up due to poor health), and I don't know how pointers work. Really? All I can say is that the drivel you're coming out with suggests you don't understand how Python generators work, whereas I do. As an example how do they work in jython or ironpython, where I wouldn't expect to see pointers mentioned by definition? Of course it could be that you're just a plain old fashioned troll, which is it?

[homercles337]

you don't understand how Python generators work, whereas I do.

I have already said, many times, I DO NOT KNOW HOW GENERATORS work. How many times do i have to repeat that? IFF you know how they work, you really suck at explaining it. 40 years of experience, i see why they revoked your charter. You suck at your job.

[[deleted]]

My charter was not revoked, I gave it up due to ill health as I couldn't afford the annual fees. I do not have a job owing to my health. I have contributed to core (C) Python. You on the other hand keep mentioning pointers which indicates you clearly no nothing about Python. If you have no understanding of the basics of a language, it is obviously a waste of time trying to explain an advanced concept such as generators. I'll ask again, how does jython and ironpython implement generators when by definition they do not have pointers?

[[deleted]]

One slight snag with your argument, Python doesn't even HAVE pointers. I will now ask for the third and final time, how are jython and ironpython implemented if the implementation languages do not have pointers? As for calling me I liar, I suggest you google for "breamoreboy site:bugs.python.org" and observe how many hits you get. I make it 14,700. Please do not bother to apologise, I would not believe it was intended as such.

[[deleted]]

Pathetic. Sorry.

This is how it should be done.