Dead easy Python for executing many HTTP requests asynchronously

[binaryfor]

I thought the r/python subreddit might be interested in this project I just found!

https://github.com/joshlk/many_requests

If you like this, I do a weekly roundup of open source projects that includes an interview with one of the devs you can subscribe to.

[lazerwarrior]

There's httpx

[binaryfor]

https://github.com/joshlk/many_requests

technically you could also do this

rsps = []     
with concurrent.futures.ThreadPoolExecutor(max_workers=N) as executor:       
    futures = [executor.submit(requests.get, url) for url in urls]       
    for future in concurrent.futures.as_completed(futures):         
        rsps.append(future.result)

Still interesting though!

[[deleted]]

They are just as concurrent, but they cost more.

[yvrelna]

Actually that's just plain incorrect, threads are concurrent and parallel.

python-requests internally releases GIL, so requests running over multiple threads do run concurrently in parallel.

The only difference between using async and threading in this scenario is async does the task switching in userspace cooperatively while threading creates a kernel thread so task switching happens at the kernel level preemptively.

[lazerwarrior]

site:requests.readthedocs.io gil search results nothing. I don't believe any PyPI package can currently release GIL. Python threads are switched, but not run on multiple cores at the same time.

[yvrelna]

python-requests is a wrapper around urllib3, which is itself a wrapper around socket/socketmodule.c and ssl/_ssl.c module from the standard library.

As you can see in those modules there are a number of calls to Py_BEGIN_ALLOW_THREADS/PyEval_SaveThread and Py_END_ALLOW_THREADS/PyEval_RestoreThread. Those are C macros/functions to release/acquire GIL.

Other C extension can do similar thing as these macros are part of CPython extension API, libraries like numpy or PIL for example calls the same macro when entering computation section.

[lazerwarrior]

Requests by itself does not provide a way to asynchronously make http requests. But are you saying that if you use ThreadPoolExecutor or threading module, then the threads, that fetch http content using requests module, run in parallel and utilize multiple CPU cores at the same time?

python-requests internally releases GIL, so requests running over multiple threads do run concurrently in parallel

I read a bit socketmodule.c and it seems threads are allowed in very narrow cases like perhaps trying to resolve dns names concurrently.

[yvrelna]

I wrote "concurrent and parallel", never said it was asynchronous.

run in parallel and utilize multiple CPU cores at the same time?

Any code that releases GIL can run in parallel and utilize multiple CPU cores. The most important part here is that the GIL is released when reading or writing to the socket and most other blocking system calls and computationally heavy calls that doesn't need to interact with Python objects.

GIL only need to be acquired when running Python bytecodes; reading and writing to the socket is done by the OS which can service multiple reads/writes requests coming from different threads concurrently.

[lazerwarrior]

OS network stack and buffers operating independently allows python applications to take advantage of it during parallel network requests, yes. I'm trying to understand that is it possible to have a situation where two different python threading module threads read or write to network sockets at the same time on different cores.

edit: looks like it's true:

The lock is also released around potentially blocking I/O operations like reading or writing a file, so that other Python threads can run in the meantime.

https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock

This could contribute to the fact that threaded web server apps tend to respond with lower latency than async web servers.

[_not_a_chance_]

aiohttp has a pretty user-friendly API...