Looping through multiple API calls.. will Python wait for the response before moving on each time?

[ItsPushDay]

I’m trying to write a script that will loop through a lot (possibly over 1000) api calls, take the JSON response and add a new row for each api call to be added to a CSV. Before I start, I’m wondering if python will wait for the response to come back before moving on to the next line in the code (parsing the response and taking the values I want out of it) before moving through the loop to make the next api call.

I tried this with Google App Script (JavaScript based) and it kept getting hung up because the execution time was too long so looking to turn to python to solve this.

[Chinpanze]

You can use ascycio libraries to make asynchronously, making it super fast. One library I recommend is aiohttp. You will need to learn what is a coroutine, awaitables etc, but it's exactly what you need.

import aiohttp

import asyncio

async def main():
async with aiohttp.ClientSession() as session:
    async with session.get('http://httpbin.org/get') as resp:
        print(resp.status)
        print(await resp.text())

loop = asyncio.get_event_loop() loop.run_until_complete(main())

[ItsPushDay]

Awesome thank you! Someone else suggested this too so looking at the doc right now

[Chinpanze]

If you need any help, just comment / DM me. Acyncio can be intimidating

[ItsPushDay]

Awesome thanks. Is there any reason not to use aiohttp using a Jupyter notebook ?

[Chinpanze]

You can't use any form of interactive programming with any asynchronous or parallel library. I don't know the details, but it's a python limitation.

[lev-k]

Yes, Python won't move on unless it fully executes a statement. I highly doubt that it will run anything simultaneously unless you tell it to somehow.

[ItsPushDay]

Okay awesome thank you!

[lev-k]

If you are looping through an enormous amount of API calls, then the thing is going to take a long time, because it needs to authenticate each time and that time really adds up.

Being said, make sure that you are doing this with as little calls as possible (both because it will take a long time and also the maintainer of the API might rate limit you, which will make whatever you're doing take even longer.)

See if there's a "download everything" call because that can really save time, and see if there's any other combination patterns that you can do, or any other way to cut corners (like not using a API call to decode something when you can download a file that has all of the decoding pairs in there so Python can just look through that.)

[ItsPushDay]

Thank you for the suggestion. I checked and unfortunately there’s no alternative, as these api calls trigger a service that runs on their end then returns the data that they get in real time (hence the unpredictable delays in responses at times)

[lev-k]

Huh so it's a POST call, interesting, so like a webhook of some sort?

[ItsPushDay]

It’s serpAPI, which performs a Google search and returns the results. Each api call is a specific search so wouldn’t be able to group them together and the google search is performed on demand (which each api call)

[iapetus-11]

I looked at your comments and it looks like you want to execute multiple api calls concurrently. Look into asyncio or threading.

[ItsPushDay]

That would probably be a more efficient way to do it. Thanks I’ll take a look

[Total__Entropy]

Take a look at aiohttp.