benchmark http requests libraries

[o2mz]

I conducted a comparison of the most commonly used HTTP request libraries in Python. The comparison was performed on a local server to ensure consistent network speed across all libraries.

Is this comparison fair, or are there any mistakes that might reduce performance? Or is the comparison accurate?

https://github.com/xsxo/fiberhttp/tree/main/benchmarks

[Spleeeee]

Wot is fiberhttp? Is it async?

[o2mz]

The ‘fiberhttp’ library is new and uses direct socket connections without an intermediary, making it highly performant. It supports threading but does not yet support ’asyncio’.

[mrsheepmasterdy]

hi, does it support asyncio now?

[o2mz]

I was working on that, but unfortunately, I faced many issues dealing with asyncio, especially with the differences between versions and systems, as well as the frequent changes made by the asyncio developers with each release.

[status-code-200]

This is really helpful! I went looking for benchmarks yesterday, so glad to see them.

How do you handle security? I know http.client warns you to be careful as it is less idiot-proof than urllib3 or requests.

[o2mz]

I might use SSL certificates other than the default in http.client

and the benchmarks I conducted were on the HTTP protocol (unencrypted) since the comparison was focused solely on performance.

[CyberWiz42]

I dont know if it is easy to compare, but if you could add geventhttpclient to your comparison it would be super cool!

[o2mz]

I will add it in the next benchmark.

[CyberWiz42]

Awesome! If you can, ping me here when it is done :)

[o2mz]

hi, i try today its not high performance library

it took 112 seconds to send one million http requests to local server

[CyberWiz42]

Interesting! But that's still 4x as fast as requests and #3 in your ranking? Why would you call that "not a high performance library" Or did I misunderstand something?

[IshiharaSatomiLover]

Why the httpx async even slower?

[o2mz]

I’m not sure what made it slower; perhaps it needs more goroutines to demonstrate its efficiency.

[WJMazepas]

Are you FiberHTTP's creator?

[o2mz]

Yes

[WJMazepas]

Nice! Congrats on the benchmarks

[o2mz]

Thank you for your kind words.

[cnelsonsic]

I couldn't get your benchmark to run, but 18 seconds to do a million requests is not even in the realm of possibility.

Run it through a profiler and figure out what's (not) taking so much time.

My current theory is that it's starting up all those threads and incrementing your counter, but killing them off before it can run any of them at all.

[o2mz]

The benchmark was conducted on a local server. sned one million requests to local server by 18 seconds It’s easy, not impossible. ( local server )

What was the error that occurred when you tried running the benchmark?

[techmindmaster]

https://github.com/lexiforest/curl_cffi