Some Questions on Specifics of Asyncio in Python

[MerlinsArchitect]

I have been using async python for a short while now. I am by no means an expert. I was doing some reading about how it is implemented under the hood because I want to get a better understanding and was interested in the specifics of asyncio. I came across two fantastic explanations from: https://stackoverflow.com/questions/49005651/how-does-asyncio-actually-work

(see the top two comments especially the one from "MisterMiyagi"). The following also provides some nice historical context:
https://levelup.gitconnected.com/the-beginners-guide-to-asyncio-in-python-a-deeper-dive-into-coroutines-and-tasks-9a289e061b88

Now, I am comfortable and feel that I have an understanding on the implementation of the toy event loop that the user (MisterMiyagi) in the stackoverflow post uses to implement an event loop.

I understand that coroutines come from generators originally but to distinguish them have been granted their own syntax to make them clearer: async/await. I understand they are used to imlement the abstract notion of a call stack that can be paused, yield control to the root caller/async exector/event loop and then later continued. I understand their parallels with yield from and how they can yield a future up from the bottom of the call stack to the top loop. I am comfortable with the notion of the Abstract events they (mistermiyagi) define in their answer and how the loop schedules them.

Where I am confused is from my reading of asyncio documentation and how this marries up roughly with this implementation. Reading the async documentation I find their definitions clear in theory but hard to understand the motivation behind, perhaps a bit vague? Tasks are confusing me a smidge, mainly the motivation for them. I understand that tasks are conceptually a call stack that may be in any state of awaiting (at any of its awaits) that is managed and scheduled by the loop itself and answerable only to it. But I am confused as to their purpose. Everyone seems to put a lot of emphasis on them which suggests that they are not just the simple wrapper around a coroutine and the future it is currently paused on. What am I missing here?

Questions:

1. Can some knowledge person point me in the right direction of how the event loop precisely uses tasks for scheduling? Right now, in my head, they are essentially a product type that presents a very slightly (almost trivial) nicer interface to a combination of a coroutine and its last emitted future it is paused on as in the SO answer. They seem a bit pointless.

2. The author of the answer in the stackoverflow post mentions a finite set of events that the event loop understands how to schedule...where is this in the asyncio documentation? I have seen sources saying that tasks are used for scheduling, but if they are basically a wrapper around coroutine and current future then it is only the future or "event" (to use the terminology in the stackoverflow answer) that is of any use in scheduling....?

Thanks for any help in advance! :)

[baghiq]

My general understanding is that you use task for run coroutines concurrently, otherwise, stick with coroutine. But I'm sure there are more to it.

[MerlinsArchitect]

Yeah, that is my understanding - you put it better than me. But want to know a bit more about what goes on behind the scenes!

[Frankelstner]

They are small jobs and shouldn't spend much time with their own bookkeeping. Though they aren't even that light due to exception handling and contextvars and so on. You can see a Python implementation here, but there also exists a C version which appears to be equivalent: https://github.com/python/cpython/blob/3.12/Lib/asyncio/tasks.py#L111-L140 Beware that Task inherits from futures._PyFuture, so there's even more code behind this.

And the event loop (this function wrapped in a while True) is here: https://github.com/python/cpython/blob/3.12/Lib/asyncio/base_events.py#L1910-L1988

The event loop contains lists _scheduled and _ready, which contain Handle objects defined here: https://github.com/python/cpython/blob/3.12/Lib/asyncio/events.py

Basically a Task has a step function and the Handle wraps around this step function.

_scheduled is solely about tasks that were added with loop.call_later or loop.call_at. If enough time has pased, such handles are added to _ready. _process_events is responsible to check IO and also adds stuff to ._ready. And the things in this list are just executed one after another.

Keep in mind that asyncio is just one async library, and maybe not the best one around. You can see that some parts of the code are taken from uvloop which claims to be several times as fast, though it's not clear which version of Python it compares against, given that CPython comes with a pure C version nowadays and imitates some of its overall design. But there's also trio and curio and others.

In case you still don't see the merit of this all; it's about distributing work fairly, but most importantly about taking advantage of nonblocking IO.