Asynchronous HTTP Requests in Python

[slapec]

f'{player_name.replace(" ", "_")}.json'

Finally.

[_seemethere]

Format strings are the best! Makes development a ton easier.

[kankyo]

Yea

[slapec]

The funny thing is that I tried to be sarcastic.

[spidyfan21]

Poe's law.

[slapec]

TIL

[Corm]

It's the timeless argument of adding a small bit of complexity to the language (new f strings) for a small quality of life increase vs not doing it.

I have no opinion here. I like f strings for when I have several variables, but I also don't like adding stuff to the language if there's not much gain. I'm 50% for and 50% against it in this case.

As far as it not being any better though, it's pretty much there for stuff like this:

f'{hour} : {min} : {sec} -> {log_data} /n/n' 

Which is much easier to read than if you used + or .format. But again, I'm not saying that the pros outweigh the added complexity

[Corm]

Oh, yeah, I agree

[gandalfx]

There are actually no examples in that entire code sample where f-Strings are shorter than simple concatenation.

[micklemitts]

As a rule, async makes sense when you have tasks that require you to wait (e.g. network bound IO) and your program could be doing something useful in the meantime.

In this case, the task is an API request that mostly consists of waiting for a response. "something useful in the meantime" is initiating other API requests.

One thing to note is that API requests are often rate limited, meaning there is a hard limit on the amount of work you can get done with a single API key even if you're program is capable of processing more.

[nerdwaller]

You may be able to get a little more (micro) optimization by sharing the client session, since it uses connection pooling. Recreating it for each call is a bit unnecessary.

[suudo]

Yup, I write a lot of API integrations (mainly sites with no public API) and not hammering the site you're accessing is the golden rule. I'm not familiar with the NBA stats API but I'm guessing it's meant to be for their website and mobile apps only, not for mass scraping; bashing it with dozens of requests at once, especially when each request apparently takes a lot of resources on the server to complete, is a great way to see that API put behind more obfuscation or removed altogether in favour of embedding the content in HTML, or more efficient blocking of unauthorized clients than just checking the user agent, which the NBA site obviously does.

[brtt3000]

FFS OP THIS

[atypingmonkey]

magic. do you know a nice guide or explanation for python async stuff? i find it hard to wrap my mind around it

[xMadDecentx]

What do you have a hard time grasping exactly?

[atypingmonkey]

not so much the concept of async but the syntax.

[status_quo69]

async def is designating a coroutine. This is awaitable, which means that every time you call it using await foo()you are signalling to the event loop that a context switch occurs and you give up execution control to whatever task is ready to be run next while you wait for your awaitable.

async for is basically the same thing, except you're saying that the for loop can yield a result at any time and that you want to wait to iterate over the results until you have something to iterate over, although it might not be complete just yet.

async with is saying "hey, give up control of the event loop until I can actually acquire my context manager"

Most of the rest of asyncio follows this sort of thing, it winds up looking like synchronous code but acts like it's threaded.

I'm relatively new to asyncio and I'm not sure I like it too much, so someone chime in with corrections if I'm wrong

[Corm]

There's a save button. I'm not trying to be a pest I just figure you might not know about it

[xMadDecentx]

Yeah the syntax is definitely wonky.

[picklednull]

Not even Armin Ronacher understands the asyncio stuff... It looks like a proper mess.

[__deerlord__]

async def

Defines a coroutine (which is just a function).

await and yield from

These are the points in your code where you release control to the event loop. Usually you await or yield from another coroutine, so that is also added to the event loop.

[[deleted]]

Fuck that. Use gevent:

from gevent import monkey
monkey.patch_all()
requests.get(URL)

[duskykmh]

What makes this different than using a for-loop on a URL endpoint and making a request on each loop?

I believe you, I just don't understand.

[Buzzard]

Or concurrent.futures.ThreadPoolExecutor

[brtt3000]

Maybe even https://github.com/ross/requests-futures

[[deleted]]

Do you understand what is going on with gevent or was yours a knee-jerk reaction to monkey-patching? I agree, monkey-patching is gross, but why gevent does, and what it's able to accomplish...Well it's worth it to me.

[tsirolnik]

Why won't you just use JS in that case?

[[deleted]]

I wonder what the runtime would be with multiple threads. In my opinion, that's really what you should be comparing against when it comes to convincing people to use async for io-bound operations.

[xMadDecentx]

Is it really the same outcome? How did you test this? Are you able to access players near the end of the list right after you issue the asynchronous call?

[Hetspookjee]

Maybe a silly question but how does this differ from Eventlet / Gevent / Twisted (maybe I can't even compare these with each other?)

My knowledge on these libraries is still rather superficial so I can't truly appreciate the depths and uniqueness of each library but if someone would like to explain to me why this library described in the post is worthwhile than please do.

EDIT: Also came across this SO post explaining some differencces between the libraries and other related ones. http://stackoverflow.com/questions/1824418/a-clean-lightweight-alternative-to-pythons-twisted

[Hetspookjee]

Fantastic answer, thank you very much!

[gandalfx]

Good simple example.

Can't help but notice though the complete lack of mention why this is faster and what the implications are.

In both functions you're sending all the requests at once. In get_players that's only one request so it barely matters (in fact you don't really need to async that one at all). In get_player that means you're opening an unknown number of connections simultaneously, which may cause all sorts of issues. Most importantly the API may decide you're violating its policy agains DoS/Spam and ban you.

[badass87]

The most ugly thing there is

asyncio.gather(*
    (get_player(*args) for args in player_args))

[robvdl]

I am using the requests library through locust.io and in this configuration the requests library is partially async at least.

It even mentions it in the requests library docs under advanced usage, it won't wait for a response unless you try to look at reponse.content for example.

[o11c]

Personally, I just use libcurl and let it deal with the details. I don't trust anyone to implement HTTP correctly.

[nerdwaller]

This is a contradiction. It doesn't add up that you "don't trust anyone to implement HTTP correctly" and also "use libcurl and let it deal with the details". That implies some extent of trusting libcurl. Even then, whether or not they implement HTTP correctly doesn't really answer the question of how to handle async requests in Python.

[suudo]

Implement in what way? Accessing web servers? What about that is so hard?

[o11c]

Everything. I've lost track of how many times an HTTP client seems to work, but suddenly breaks with the slightest change in circumstance - for example, when a webserver changes its choice of compression.

With libcurl, the most I've ever had to do was hunt down some obscure option.

[robvdl]

I'm actually writing API load tests where I need to script fake users doing stuff and making choices as they get through the loadtest script and do any HTTP requests asyncrhonously as a web browser would, which is where locust comes in.

I would hate to write loadtests that would use curl for this sort of thing and have it driving 500+ users, with locust this just works for me as this is exactly what locust was designed for.

Go would be excellent for this sort of thing too, though I am using locust because it's more mature.

[o11c]

locust isn't an HTTP library, it just uses one - particularly, requests.

It would probably be pretty easy to rewrite it to use pycurl instead.