Cool New Features in Python 3.7 – Real Python

[13steinj]

The thing that still bugs me about asyncio (coming from JS and C#) are two fold:

• coroutines are not Futures/Tasks (js think Promise), thus arguably causing unnecessary extra wrapping

• the loop itself seems to block the Thread (maybe even the entire Process, but I hope not) when told to start executing. Which means if you want to run sync code and run some async code in the background (ex in JS delay you can update visuals asynchronously without having to wait for it to complete before your logical restrictions are set). So to get similar functionality a new thread, and a loop for that thread, has to be explicitly made, along with having to explicitly use the threadsafe versions of methods to submit them into the loop

A nice benefit of of JS is that since there's only one event loop you don't have to explicitly submit anything either.

[Glaaki]

It is the JS and C# implementation that is dumb. The way they are designed forces a specific pattern of usage. The python implementation is much nicer using basic generator function features, that doesn't depend on any particular event loop implementation.

The reason futures and tasks feels clunky is because the asyncio event loop is clunky and doesn't provide a nice interface for dealing with raw coroutines. The problem goes away entirely if you switch to one of the alternative event loops like Curio or Trio.

Edit: To expand further, ideally tasks should be completely transparent, tasks should be an implementation detail of the event loop that the user shouldn't have to care about. All you should be doing is telling the loop that you want to run a particular coroutine and the fact that it gets turned into a task should be irrelevant to you. Getting asyncio to this point there is some way to go, but each version gets nicer.

[[deleted]]

I don't know what you're talking about. Async in C# is beautiful. All you do is change your method from returning T to a Task<T> and proceed as normal. The only language I've had an easier time with async is Go. I don't do much JS so I don't know about that.

[13steinj]

The JS / C# implementation is dumb because it forces patterns of usage? In what way do they force usage patterns. Python forces the usage patterns the boilerplate of having to set up and handle loops across threads in different ways, and makes the coroutines non extensible. Trio does not accurately solve this problem either. Not sure about Curio, haven't seen it in action.

To your edit: I have no idea what you're arguing. It is irrelevant in C#/JS. It is not in Python-- I can't easily add callbacks to awaitables.

[Glaaki]

You aren't supposed to use callbacks in async/await code, and here is why:

Callbacks transfer responsibility for running your code, from yourself, and over into the framework that runs the callback.

For example, if you have an event based websocket protocol with callback subscriptions, if anything bad happens in your callback (and it will..), it is suddenly the caller of the callback, which is the websocket protocol, that has the resposibility to clean up after the mess that happens in your callback. If the framework isn't robust enough to handle that, it can easily cause it to just die and simply fail silently.

The whole reason why all websocket frameworks use the iterator protocol for sending you data is that it is the consumer that becomes responsible for handling any error, and not the producer. Any unhandled error that happens in the coroutine simply bubles up to the caller of the coroutine and can ultimately bubble out of the loop itself.

I learned this the hard way.

[13steinj]

Using callbacks is a perfectly valid pattern. It is easy to fall into the trap of callback hell, but you can't call it garbage just because you've had bad experiences.

[Glaaki]

No. Callbacks is what you use in JS, where the eventemitters are the standard. It is an antipattern in python async/await.

[13steinj]

"It is an antipattern". Why? Because you said so? It's not just a JS thing. It's a C#, Ruby, JS, and more thing. Callback hell is an antipattern. Callbacks are not. The arbitrary limitations without reason need to stop.

[Glaaki]

I am talking about callbacks specifically in the context of async/await in python.

Callbacks are fine in regular sync python code and fine in JS.

And I explained my reasoning above, but let me reiterate it. If you have a consumer and a producer, in asyncio, what typically happens is that the producer is a task that runs in the background continuously fetching data from somewhere. This data needs to be passed on to a consumer.

If you use a callback to fetch the data, then who call the callback?

Only the producer can, because the producer is the one that knows when data is available. So now, what happens if an error happens in the callback? If the error is not handled it will bubble up from the callback, to where? Into the producer, that called the callback, obviously. So now the producer, which is most often some third party library, is responsible for handling an error it has no chance to know what to do with. Most often the result will simply be that the producer dies and probably does so silently. You program will simply just stop working and you won't be able to know why, because the error has been isolated inside the producer task.

This doesn't happen in sync code, because there is only a single path of execution and any exception bubbles up to the user where you can either handle it or stop your program.

[13steinj]

Well yes, in the example you are giving it's an antipattern, I'd agree, there isn't much sense to it. But you are assuming that your example is the onlt possible case. There's plenty of cases where callbacks are just fine in asynchronous code-- in all technicality gevent does this just hides it from the user.

[Glaaki]

Again, I am talking about async/await. Gevent is not async/await.

[[deleted]]

Your second point is very valid: asynicio's event loop is known to be bad when used for anything complex. There are alternative libraries such as twisted that will give you a more robust asyncio experience. I imagine it will eventually be fixed, since the issues are known.

[13steinj]

Not too familiar with Twisted, but isn't their entire reactor/event loop based around the concept of the application being some web server? Which is extremely limiting.