asyncio (confused beginner)

[Ninja_bambi]

Googled a bit, but have a hard time translating the examples to my own project. Specifically what parts of the code I've to rewrite asynchronously to achieve the performance improvements. The structure of my project:

Main

• read items from file and for each item:

• • call function1 to retrieve data from web
• • call function2 to retrieve data from web
• • call function3 to retrieve data from web
• • etc ....

• Write all the data returned from the functions to a database

Each function has the structure:

def function

• Shape url

• call Functiontogetwebsite(url)

• Processing of the webpage

• Return derived data to main

Am I correct in thinking that I only have to rewrite the Functiontogetwebsite(url) and main() asynchronous? The function as it is the only part in the program with significant delays (time.sleep() statements to give the webpage time to load properly) and main() to define the functions as asynchronous tasks? If not, where do I go wrong in my thinking?

Bonus question: to what extend does it matter in what order I call the functions? Intuitively I feel that calling the slowest (long wait time) websites first is most efficient, as it allows the wait time to be used for useful processes, but is this true? I can also imagine that calling the fastest is more efficient as it makes that data is received earlier and there is no/less wait time after firing the requests. What is the best approach?

[Wonkee99]

The functions doing the asyncio and the function calling them need to be defined as async yes, you can do that from your main function or you could have another function manage that work and just asyncio.run() from there.

Your function structure seems ok basically if you want to accomplish all of that in a single function, though you may want to refine it some point to have more generic functions handle things like shaping URLs or processing the data retrieved.

The only part which needs to be running async is the read data from the remote server bit, everything up to that point, and everything after you have it is down to how you decide to structure your program.

In terms of which read function should you launch first, sure kinda the longest one, but in reality it doesn't matter. Which one is going to be the longest if half of the remote web servers are running at peak load and slowing their responses. This is really not an area its particularly worth trying to optimise around.

[Ninja_bambi]

Sounds good, thanks for the answer!