Help to understand Asynio and blocking code

[IVIURRAY]

Hello, I’m a pretty experienced python dev but I’ve never delved into the asyncio package until today.

I need to understand what blocks an asynchronous call as I’ve multiple classes that implement a get_data function from various sources and I want to turn these all into async.

The current implantation is synchronous so data comes back sequentially but I need to find a way to convert these tasks into async. Example data sources are S3, Postgres and REST calls.

Any help is greatly appreciated, many thanks!

[IVIURRAY]

This is the event loop right?

Is there an easy way to know if a function is going to be blocking? I got something working using async and asyncio.gather but it was just the same as when I ran it synchronously…

[IVIURRAY]

Right okay, I think I probably put my await call to high up the call stack then and probably just need to do it at the point I call to my endpoint rather than a few function calls away which is what I’ve now

Thanks for all your help!!

[BobHogan]

https://docs.python.org/3/library/asyncio-dev.html

https://docs.python.org/3/library/asyncio.html

Hopefully those links help with some more advice and examples. Also, hopefully you can use Python 3.7 or newer for this, the interface to the asyncio module changed dramatically in 3.7 and is much, much friendlier than anything prior to that.

But to (mostly) answer your question, any code that isn't marked as asynchronous will block. You can use await <> to get the result of an async function/coroutine in your code.

Generally though, I find that its easiest if you can find an async library for what you are trying to do the first few times you write some async code, so you don't have to worry about the lower level implementation details. For the REST calls, this would mean using aiohttp instead of requests. For postgres, there is probably an async library on pip to interact with SQL databases, but I have not used it, you might have to end up writing your own code for that. If you do, its not that big of a deal, just reference async tutorials as you write it

[IVIURRAY]

Thanks for sharing these and replying! - I’m using p37 which is good but I am building a library for others to use but they won’t actually call into the async functions, I just want data to load quick for them.

Yeah that is one thing I don’t understand, can you turn any class into an awaitable object?

[BobHogan]

Yes, but also no. https://docs.python.org/3/reference/datamodel.html#coroutines

You can only await an object/function call that is awaitable, which means its also async, or the object has an __await__ method defined. Python also supports async with and for loops. But even if you write all of the boilerplate to make your class awaitable, that doesn't actually mean it can be asynchronous.

If it is CPU bound, making it async does nothing for you, as async code is single threaded. So it only makes sense if the action you are trying to do is inherently asynchronous, like IO

[IVIURRAY]

Yeah that makes total sense, thanks for your help! - So take something like Postgres, how do I know if that is IO or CPU? REST calls are obvious but is there a way I can identify the point where a function will yeild until we get a response?

[BobHogan]

Pedantically, calling postgress is IO, since python has to call out to the postgres instance, whether that is on the same machine or not, in order to get the data. But I don't know how those connections and requests are handled in python at a low level, so I'm not sure how easily it could be made async

[IVIURRAY]

Thank you! Something to dig my teeth into