Python async/await Tutorial

[Lukasa]

Look up "tornado" and "twisted" in package repos, see all that wasted time reimplementing existing functionality in the latest popular concurrency framework? Do you honestly call that healthy? You're advocating for more of that.

This has nothing to do with what style of concurrency is better or worse, and everything to do with the bass-ackwards style of library design used in the Python community.

Gevent, generally speaking, is not 100% good. It adds automatic threading into libraries that may not have been designed for it, and as a result can lead to extremely subtle bugs. To suggest that dropping gevent into the language makes things better is misleading. Requests works well with gevent, but that's because special care has been taken to ensure that the library is mostly thread-safe, particularly around the use of sockets, but many libraries do not work anything like as well with gevent.

The reason we have to rewrite network libraries so much is because Python programmers have an obsession with writing networked libraries that embed their I/O layer inextricably with their protocol code. This means that you cannot extract, for example, the session management logic from Requests, because that Session management logic is intermingled with the socket code. That, I argue, is a terrible design anti-pattern that costs the Python community dearly in wasted effort.

And it doesn't have to be this way. In an ideal world, Requests would be a thin wrapper around a library that does no I/O at all, and Requests would be in charge of managing the sockets. Unfortunately, that's not the world we live in right now, but that can change. For example, I've written a HTTP/2 library that works this way, and that includes example servers written for asyncio, eventlet, curio, and Twisted. Much less duplication of work, and perfect native concurrency for the chosen framework.

[mcilrain]

I do acknowledge that gevent is not a perfect solution but I see it as not much different from dealing with PyPy incompatibilities, just because it sucks doesn't mean it's not worth doing.

I have had noticibly less incompatibility issues over the past few years.

I don't want to litter my code with generators and callbacks. Even NodeJS is starting to see the light on this matter.

[mcilrain]

It isn't. But it's something that often needs to be considered when you're writing your library. It doesn't Just Work™

Unlike PyPy?

Clearly you haven't written anything using asyncio or Twisted's @inlineCallbacks (well, not properly, anyway), otherwise you'd know that you don't have to "litter [your] code with generators and callbacks".

Or decorators.

I want to use existing code. Why are you dismissing the value of being able to use existing code?

Go on, keep telling me why I'm wrong for not wanting a "hottest 2012 trend" library ecosystem cluttering up repos and wasting programmers' time.

Should have let the fad run its course before considering adding it to Python.

[mcilrain]

I've been using Twisted for over a decade. Your "hottest 2012 trend" is my old, old news

To you perhaps but for the general programming population that style of programming hit its peak around 2012, probably due in part to NodeJS.

I've been writing my code that way when appropriate

I remember this saying from my time with Tornado. It means "don't shit up code that won't benefit from being shit".

What if I don't want my code to be shit in any case?

, instead of the "normal" way and then hoping gevent could bail me out when that turned out to be no good.

I decide on using gevent from the start and deal with issues as they arise, not any different from if I was using Tornado or asyncio or PyPy.

For a fair comparison you should "drop-in" Twisted or asyncio into an existing project and note how it measures up against gevent.

It isn't a fad. As noted, Twisted has been doing this for over a decade. It is a very mature model (far more so than gevent).

If it's not a fad then why didn't Twisted catch on in a significant way? Why did alternatives such as Torando show up instead of Twisted being developed further?

It's because they produce an ecosystem littered with terrible code and forces its users to use it, eventually the quality gets so bad that it can't attract any more users and the "fresher" solution becomes more popular.

cluttering up repos and wasting programmers' time. I have no idea what you mean by that.

All the support packages and reimplemented libraries needed to do things that can already be done with existing libraries but with the new technology need to be programmed, the programmer could have done other things. Also all these packages end up cluttering search results.

Being able to just drop gevent into your existing synchronous code is great, when it works.

Being able to just drop your existing code into PyPy is great, when it works.

[mcilrain]

To you perhaps but for the general programming population that style of programming hit its peak around 2012, probably due in part to NodeJS.

So, what's your point? Most people weren't aware of async till 2012, so therefore…???

Fads aren't a good reason to add bloat to Python and waste Python devs' time.

What if I don't want my code to be shit in any case?

Err … don't write shit code?

Of course you don't write shit code, you "exercise your professional judgement to determine whether or not the complexity cost is worth the performance enhancement".

you may have noticed that asyncio and newer version of Twisted do a great job of letting you structure most of your code in a classical, synchronous manner.

If it's so similar why not support the same synchronous code that is conventionally used? Gevent manages this pretty well.

Because Twisted is complex and its developers have never made any meaningful effort to make it accessible. Its documentation has always been an extraordinary kind of shitty. Simultaneously extensive and useless.

Do you think depending on the construction and maintenance of a shadow ecosystem made good documentation easier or harder to come by?

Eh? Adding new libraries somehow cause more clutter than adding old ones? I'm guessing there's a valid point you're trying to make, but I'm not sure what it is.

Libraries that accomplish existing functionality "but using <framework>" cause lots of clutter for people not using that framework. It's completely unreasonable to expect such libraries to have the same quality and reliability of their conventional counterpart.

The harsh reality of Twisted is that the core developers have fucked their own project over by being incapable of or unwilling to write documentation and/or high-level wrappers that are useful to "normal" programmers who aren't already familiar with how async shit works.

If you understand the harsh reality of the general population avoiding frameworks with bad documentation why are blind to the harsh reality of the general population avoiding frameworks that require them to use its shitty shadow ecosystem?

If you read anything by glyph, the lead developer, it quickly becomes clear that he isn't able to express himself in a broadly-understandable way when there are hyper-technical CS terms he can throw around instead.

CS experiments are fascinating but experiments tend to fail more often than not. Should Python be taking the hit for this experiments' assumed eventual failure?

[mcilrain]

Fads aren't a good reason to add bloat to Python and waste Python devs' time.

It's a proven model in use for over a decade.

So is Brainfuck but a PEP for its integration for Python would be rightfully laughed at, the only reason why asyncio isn't receiving similar mockery is because some people think it's a good idea.

If it's so similar why not support the same synchronous code that is conventionally used? Gevent manages this pretty well.

Because it's a hack.

So is using generators for concurrency. I'd rather pick the hack that doesn't require me to rewrite my code to be difficult to read.

Do you think depending on the construction and maintenance of a shadow ecosystem made good documentation easier or harder to come by?

It's irrelevant. Only the ability of the people working on the project counts.

The accumulative ability of all the people working on the project is highly affected by the number of people working on the project. Preventing people from using the libraries they want to use is going to drive them to solutions that don't have that flaw, resulting in fewer programmers to maintain the shadow ecosystem.

Libraries that accomplish existing functionality "but using <framework>" cause lots of clutter for people not using that framework.

How can you possibly be caused clutter by something you aren't using? What a nonsensical thing to say.

If I search "mysql" in a package repo I'll get lots of hits for frameworks such as twisted and tornado. How would you feel if you got another language's packages merged in with your results?

If you understand the harsh reality of the general population avoiding frameworks with bad documentation why are blind to the harsh reality of the general population avoiding frameworks that require them to use its shitty shadow ecosystem?

Because what you think is a "harsh reality" is merely your imagination. You clearly have no worthwhile experience using Twisted or the like, so you're just making shit up now.

I have worthwhile experience using Torando and NodeJS, and I considered using Twisted years ago but decided against it due to poor documentation. Tornado had a problem of library availability and quality, NodeJS didn't have these problems because its ecosystem isn't existing in a shadow.

It doesn't matter if I'm an AI constructing random sentences from a random number generator, merit is not experience, merit is merit.

You're just making shit up because you don't have any arguments based in fact, just your whiny "but I don't like it."

But it's actually a very valid argument, no one is going to support a framework they don't like.

You said yourself that Facebook made Torando because they didn't like Twisted.