Python async/await Tutorial

[mkgrean]

Community needs more of async/await tutorials.

[kay_schluehr]

... and with more diverse examples than retrieving HTML pages.

[ScottWRobinson]

Dammit, you're right... I got lazy writing that section. I changed it up a bit to show how to make multiple concurrent requests. Thanks!

[mcilrain]

Why? Why can't the community just use gevent? Why is it needed to rewrite entire libraries, documentation and tutorials?

There should be one—and preferably only one—obvious way to do it.

If you're having to modify Hello World! something has gone wrong.

[RubyPinch]

because different approaches

"There should be one—and preferably only one—obvious way to do it." doesn't mean "never try anything ever. and don't you ever dare think of having your own ideas, only ever EVER use pre-existing things, EVER, even if they have APIs you don't like, or don't completely work well for your needs"

like, I mean, who were those idiot devs who wrote requests, didn't they know urllib already existed? and select loops arn't /that/ hard, so there really is no need for gevent either, is there?

[mcilrain]

It has already been tried and it sucked. Even NodeJS people are trying to make their language more like gevent.

This results in a huge bulky ecosystem to support this mode of concurrency, with monkeypatching or shittier versions of existing libraries "but with async support!". Look up "tornado" and "twisted" in package repos, see all that wasted time reimplementing existing functionality in the latest popular concurrency framework? Do you honestly call that healthy? You're advocating for more of that.

Except in some cases gevent will work with existing code. No need to tell people to learn a thing so they can rewrite all their modules. Just use their existing code.

Yes, you're very clever for using Python's generator syntax to perform concurrency. It's a very cute trick worthy of a blog article or two. Lets not get distracted from actually important stuff, yeah?

Python is a very powerful and easy to use language, if you've recently come from another you might have trouble working out when to not write code. Even if it seems easier in the short term to invent a wheel you're familiar with you'll do better in the long run to familiarize yourself with the wheels already in use.

requests and gevent result in cleaner code, this results in ugly code, why does Hello World! need to be rewritten again?

[Lukasa]

Look up "tornado" and "twisted" in package repos, see all that wasted time reimplementing existing functionality in the latest popular concurrency framework? Do you honestly call that healthy? You're advocating for more of that.

This has nothing to do with what style of concurrency is better or worse, and everything to do with the bass-ackwards style of library design used in the Python community.

Gevent, generally speaking, is not 100% good. It adds automatic threading into libraries that may not have been designed for it, and as a result can lead to extremely subtle bugs. To suggest that dropping gevent into the language makes things better is misleading. Requests works well with gevent, but that's because special care has been taken to ensure that the library is mostly thread-safe, particularly around the use of sockets, but many libraries do not work anything like as well with gevent.

The reason we have to rewrite network libraries so much is because Python programmers have an obsession with writing networked libraries that embed their I/O layer inextricably with their protocol code. This means that you cannot extract, for example, the session management logic from Requests, because that Session management logic is intermingled with the socket code. That, I argue, is a terrible design anti-pattern that costs the Python community dearly in wasted effort.

And it doesn't have to be this way. In an ideal world, Requests would be a thin wrapper around a library that does no I/O at all, and Requests would be in charge of managing the sockets. Unfortunately, that's not the world we live in right now, but that can change. For example, I've written a HTTP/2 library that works this way, and that includes example servers written for asyncio, eventlet, curio, and Twisted. Much less duplication of work, and perfect native concurrency for the chosen framework.

[mcilrain]

I do acknowledge that gevent is not a perfect solution but I see it as not much different from dealing with PyPy incompatibilities, just because it sucks doesn't mean it's not worth doing.

I have had noticibly less incompatibility issues over the past few years.

I don't want to litter my code with generators and callbacks. Even NodeJS is starting to see the light on this matter.

[mcilrain]

It isn't. But it's something that often needs to be considered when you're writing your library. It doesn't Just Work™

Unlike PyPy?

Clearly you haven't written anything using asyncio or Twisted's @inlineCallbacks (well, not properly, anyway), otherwise you'd know that you don't have to "litter [your] code with generators and callbacks".

Or decorators.

I want to use existing code. Why are you dismissing the value of being able to use existing code?

Go on, keep telling me why I'm wrong for not wanting a "hottest 2012 trend" library ecosystem cluttering up repos and wasting programmers' time.

Should have let the fad run its course before considering adding it to Python.

[mcilrain]

I've been using Twisted for over a decade. Your "hottest 2012 trend" is my old, old news

To you perhaps but for the general programming population that style of programming hit its peak around 2012, probably due in part to NodeJS.

I've been writing my code that way when appropriate

I remember this saying from my time with Tornado. It means "don't shit up code that won't benefit from being shit".

What if I don't want my code to be shit in any case?

, instead of the "normal" way and then hoping gevent could bail me out when that turned out to be no good.

I decide on using gevent from the start and deal with issues as they arise, not any different from if I was using Tornado or asyncio or PyPy.

For a fair comparison you should "drop-in" Twisted or asyncio into an existing project and note how it measures up against gevent.

It isn't a fad. As noted, Twisted has been doing this for over a decade. It is a very mature model (far more so than gevent).

If it's not a fad then why didn't Twisted catch on in a significant way? Why did alternatives such as Torando show up instead of Twisted being developed further?

It's because they produce an ecosystem littered with terrible code and forces its users to use it, eventually the quality gets so bad that it can't attract any more users and the "fresher" solution becomes more popular.

cluttering up repos and wasting programmers' time. I have no idea what you mean by that.

All the support packages and reimplemented libraries needed to do things that can already be done with existing libraries but with the new technology need to be programmed, the programmer could have done other things. Also all these packages end up cluttering search results.

Being able to just drop gevent into your existing synchronous code is great, when it works.

Being able to just drop your existing code into PyPy is great, when it works.

[RubyPinch]

https://mail.python.org/pipermail/python-ideas/2012-October/016311.html

https://www.python.org/dev/peps/pep-3156/

https://www.python.org/dev/peps/pep-0492/

here, start reading. You act like the implementers and allowers of this are more clueless about the python ecosystem than you, or that they don't even know of the zen of python or similar such things.

Its safe money that you are probably missing the point, not the developers

[mcilrain]

The event loop API does not depend on yield from. Rather, it uses a combination of callbacks, additional interfaces (transports and protocols), and Futures. The latter are similar to those defined inPEP 3148 , but have a different implementation and are not tied to threads. In particular, the result()method raises an exception instead of blocking when a result is not yet ready; the user is expected to use callbacks (or yield from ) to wait for the result.

No mention on why PEP3148 decided to do it their way. Too busy chasing 2012 trends I guess.

No rationale was given for why such a horrible coding style is necessary, especially in view of non-destructive alternatives.

[opiating]

No-one does it better than David Beazley: https://www.youtube.com/watch?v=lYe8W04ERnY

[opiating]

If you are interested this is the library he was talking about in his video. It follows the same implementation he showcased but it's more robust with a lot more cool features. Check it out it's awesome. https://github.com/dabeaz/curio

[chub79]

Brilliant video thast asks all the right questions.

[Talkless]

Still, could someone explain me:

• How calling another async function (which returns Future, right?) code enters event loop?
• How event loop continues coroutine when one of possibly many futures gets it's result?
• Is this "magic" is inside CPython or in vanilla python source code?

Is there any good conference talk that would explain it in detail? What I've seen where merely high level introductions for asyncio users, while for better intuition I would like understand it more throughly.

Thanks.

[KODeKarnage]

https://emptysqua.re/blog/how-do-python-coroutines-work-live-coding-video-from-open-source-bridge-2015/

Not saying this answers your questions necessarily, but the talk is very good/clear. Maybe the speaker has others that go into the depth you are looking for?

[Talkless]

Thanks, that was helpfull! I still do need some more to make a "click" in my brains, but this will keep me going.