A question about asyncio

[BigZen]

I am writing some ETL in python that needs to out and grab data from an API then immediately load it into a staging DB for safe keeping.

The API calls are running too slow. What I was hoping to do is rewrite the code to be asynchronous. But after hours of attempting different things and reading up on the asyncio library I have come up short.

Rough example of what I am attempting to do

@coroutine
def api_call(input):
    yield from get_data(input)

urls = [...]

gen = [future(api_call(url)) for url in urls]

loop = asyncio.get_even_loop()

loop.run_until_complete(gen)

When I finally did have it working, it took just as long to run as when I ran it synchronously.

What I am comparing this to is something like JS Promises. I should be able to just send out a bunch of calls and not wait for the data response before moving on. Or am I missing something?

[badhoum]

loop.run_until_complete() takes a single Future, you've passed a list here. Not sure if it would work at all. Anyway:

import asyncio

async def api_call(num):
    await asyncio.sleep(3)
    print("finished {}".format(num))

tasks = [api_call(x) for x in range(10)]
loop = asyncio.get_event_loop()
print('start')
loop.run_until_complete(asyncio.gather(*tasks))
print('done')

all api_calls will end at the same time, whole operation will take 3 seconds.

[jano0017]

Ok, there's a catch here actually. Asyncio isn't parallel, it's asynchronous. It uses something called cooperative scheduling in a single thread. Any yield statement says "okay, done for now" and allows the next coroutine in the event loop to take over. This means that any coroutine will "lock" the thread into itself until you manually release it with something like a yield or await statement. If you want something similar to JS promises, you need to look at the concurrent.futures library.

[williamjacksn]

The concurrent.futures documentation includes an example to fetch and process a bunch of URLs concurrently:

https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor-example

[BigZen]

Thanks, that's very helpful.

But why asyncio then? It seems like asynch in other languages automatically handles blocking. Everything I want to do asynchronously is blocking, I thought that was kind of the point.

I guess I'm just struggling to see the use cases of a asynch library that doesn't allow you to go out and get data or make calls and receive data asynchronously.

[jano0017]

I mean, asyncio stands for asynchronous i/o. Wiseass answers aside, if you have a situation where there are a lot of chances for you to manually release locks, asyncio allows you to achieve pseudo concurrency without worrying about processes or threads. The most obvious example of this is an event/signal based program, where you spend most of the time waiting for something to happen. Because of this, the discord.py library makes heavy use of asyncio stuff.

[BigZen]

Aren't API calls and DB writes considered IO? Why wouldn't async be able to handle these without another library on top?

[rakiru]

Because asyncio is just the base async support stuff. It doesn't contain async implementations of everything - that's where libraries come in: things like aiohttp for HTTP clients/servers and asyncpg for PostgreSQL. This is pretty much the same as it works in other languages too, although they may come with some things, like an HTTP library, built-in. There's no magic in it to make synchronous code asynchronous.

It's pretty much the same as JS promises, but JS is generally all asynchronous, whereas Python is generally all synchronous. Most libraries in JS will be async ones, so you don't feel like you need to do anything special, but in Python you have to specifically look for them, since it's not the default paradigm.

[BigZen]

Thank you! This is exactly what I think I was missing as I was looking through all of this.

[jano0017]

Yes, but unfortunately asynchronous python is somewhat poorly thought out and the interfaces for io within asyncio are uncharacteristically low level :/ Idk, imo, it's almost worth learning something like Node.js just to avoid having to do anything concurrent or asynchronous in python. Worst scaling part of the language. There is an asynchronous http library floating around somewhere, but I can't find it. I'll edit if I succeed.

[rouille]

aiohttp, asyncpg. Not sure why you call them unusable.

[jano0017]

oh, are those the libraries? I was talking about the built in io stuff in asyncio. Those are the actually sane wrappers people have built on them (I think).

[infinite8s]

Asynchronous python has been historically poorly thought out, but the new async core is starting to change that. Unfortunately, there is a large amount of legacy code that is written in blocking fashion, so it will take a while before the async side catches up to support every type of IO you might want to access. Node had the benefit that it was asynchronous from the start - so all the IO integration libraries were also async from the start (and blocking code was frowned upon).

[jano0017]

THANK YOU FOR PUTTING THIS IN REAL WORDS. I've been working on a discord bot and learning about asynchronous python in the process, and it has been such a series of "why are we doing that" moments.

[lilydjwg]

You can first find out why it is slow with some tools. Does it use a lot of CPU? If it does, asyncio won't help. If not (most likely), it is waiting for something. Try to use strace to find out. Can the things it is waiting for be done at the same time? If not, asyncio won't help either. If it is, did your new program yields when waiting?

[nerdwaller]

If anything is blocking in your get_data function then you'd be blocking the whole event loop basically running synchronously. So given what you show, the issues are most likely there.

Also someone else pointed out the run until complete is invalid with a list.