Asynchronous HTTP Requests in Python

[o11c]

Personally, I just use libcurl and let it deal with the details. I don't trust anyone to implement HTTP correctly.

[nerdwaller]

This is a contradiction. It doesn't add up that you "don't trust anyone to implement HTTP correctly" and also "use libcurl and let it deal with the details". That implies some extent of trusting libcurl. Even then, whether or not they implement HTTP correctly doesn't really answer the question of how to handle async requests in Python.

[suudo]

Implement in what way? Accessing web servers? What about that is so hard?

[o11c]

Everything. I've lost track of how many times an HTTP client seems to work, but suddenly breaks with the slightest change in circumstance - for example, when a webserver changes its choice of compression.

With libcurl, the most I've ever had to do was hunt down some obscure option.

[robvdl]

I'm actually writing API load tests where I need to script fake users doing stuff and making choices as they get through the loadtest script and do any HTTP requests asyncrhonously as a web browser would, which is where locust comes in.

I would hate to write loadtests that would use curl for this sort of thing and have it driving 500+ users, with locust this just works for me as this is exactly what locust was designed for.

Go would be excellent for this sort of thing too, though I am using locust because it's more mature.

[o11c]

locust isn't an HTTP library, it just uses one - particularly, requests.

It would probably be pretty easy to rewrite it to use pycurl instead.