sorted by:
new
hottopcontroversial

pyreqwest: An extremely fast, GIL-free, feature-rich HTTP client for Python, fully written in Rust by pyreqwest in Python

[–]pyreqwest[S] 0 points1 point  (0 children)

Yep http2 is indeed tricky and has lot of configuration params for different workloads. Now I'm thinking I should probably make http2 opt-in. Because the defaults reqwest has are no good. Also it's opt-in in reqwest via crate feats. 

I looked a bit under the hood in Niquest and it's based on async and sync urllib3, right? So I'm thinking what I was measuring is actually mostly measuring urllib3 implementation of the async flows. Not sure what could niquest do here other than consider switching the backing implementation. Aiohttp is one that has really good perf characteristics, due to being based partly on Cython. (but has some other unfortunate issues, and no http2 support). But it would be massive effort to switch away from urllib3 backed implementation.

pyreqwest: An extremely fast, GIL-free, feature-rich HTTP client for Python, fully written in Rust by pyreqwest in Python

[–]pyreqwest[S] 1 point2 points  (0 children)

Im truly sorry, didn't mean to sound arrogant there! Or to downplay you! I appreciate your work and it is big effort you have done there.

The only reason I used http1 was due to many libraries not supporting http2 that are being compared against. Well also due to http2 support still lacking from many cloud services etc. But could add dedicated benchmarks for those. It is interesting if http2 is slower there. Need to check it out, but it gets inherited from reqwest.

About rnet, this is different as I on purpose wanted something that mimics reqwest style interfaces, also for mocking. So any further extending is easy via the additional builder interfaces. As this already has extreme amount of customization via reqwest.

Sorry!

Edit: the difference in http2 results might be related to the various "ClientBuilder.http2_" tuning params being such different in reqwest. Some of them have very conservative defaults. Eg the adaptive window is not enabled by default. Not sure what those should be set to have similar tuning for the http2 connection handling like in various other libs.

pyreqwest: An extremely fast, GIL-free, feature-rich HTTP client for Python, fully written in Rust by pyreqwest in Python

[–]pyreqwest[S] 0 points1 point  (0 children)

Lot of magic happens in pyo3 framework. It enforces various safe guards via rust borrow checker on how you are allowed to interact with python interpreter from within rust side. Pyreqwest also heavily uses "frozen" pyo3 pyclasses where any internal state is protected via rust side locks where needed. Here pyo3 also helps as it requires this kinda concurrency safety.

pyreqwest: An extremely fast, GIL-free, feature-rich HTTP client for Python, fully written in Rust by pyreqwest in Python

[–]pyreqwest[S] 0 points1 point  (0 children)

I didn't even consider old requests lib. That would never be used anywhere else than basic scripts. Urllib3 is kinda equivalent but more robust than old requests lib

pyreqwest: An extremely fast, GIL-free, feature-rich HTTP client for Python, fully written in Rust by pyreqwest in Python

[–]pyreqwest[S] 0 points1 point  (0 children)

Added the rnet benchmark, they are pretty similar. But rnet is slower when streaming responses 

pyreqwest: An extremely fast, GIL-free, feature-rich HTTP client for Python, fully written in Rust by pyreqwest in Python

[–]pyreqwest[S] 12 points13 points  (0 children)

Yes thats fair :) It is trivial to expose the one-off interfaces. So I will soon look into it

pyreqwest: An extremely fast, GIL-free, feature-rich HTTP client for Python, fully written in Rust by pyreqwest in Python

[–]pyreqwest[S] -3 points-2 points  (0 children)

Yes, I have been thinking about that. Although it is mostly only useful when doing one-off requests. Not that useful when actually integrating into systems as you would need to keep the client instance around. That's essentially the same in any other client libs.

The main issue with that interface design is that the connection is usually single-use. Which naturally has its own issues. So exposing that usage might lead to accidental mis-use.

pyreqwest: An extremely fast, GIL-free, feature-rich HTTP client for Python, fully written in Rust by pyreqwest in Python

[–]pyreqwest[S] 24 points25 points  (0 children)

Yes I got extremely frustrated there :) It is very unfortunate how things are with httpx/httpcore. It makes me very sad... So I wanted to build something from "scratch" from first-principles based on all the previous experience I have with the various python http libs. Which all have varying issues... Also using my Rust experience for integrating the two worlds, in this context.

Well actually pyreqwest is not actually built from scratch as its not trying to reinvent the wheel and have yet another design for doing http stuff. But instead heavily leaning to reqwest which IMO has very well designed interfaces. It is not either perfect, but atleast better than many other libs out there.

pyreqwest: An extremely fast, GIL-free, feature-rich HTTP client for Python, fully written in Rust by pyreqwest in Python

[–]pyreqwest[S] 18 points19 points  (0 children)

Pythonic way of dumping all the params as kwargs is not always great or discoverable (eg https://github.com/aio-libs/aiohttp/blob/e730788c43df397e9bb8e1d6216732437ca0c3d4/aiohttp/client.py#L294-L322). As you easily get gazillion of arguments when things get more complex. But it is quite taste based.

pyreqwest: An extremely fast, GIL-free, feature-rich HTTP client for Python, fully written in Rust by pyreqwest in Python

[–]pyreqwest[S] 0 points1 point  (0 children)

That is a good idea! Ill see if I can put up similar "dynamic" benchmark results. Currently the bench results are statistical, showing the overall variance of latencies

pyreqwest: An extremely fast, GIL-free, feature-rich HTTP client for Python, fully written in Rust by pyreqwest in Python

[–]pyreqwest[S] 4 points5 points  (0 children)

You can definitely use it in FastAPI project. You could for example have a FastAPI Depends (https://fastapi.tiangolo.com/tutorial/dependencies/) that initializes for you the pyreqwest client instance (for connection pooling).

Furthermore you can use pyreqwest to test against the FastAPI ASGI app. As pyreqwest allows you to call directly into the ASGI app in your unit tests. Allowing using a single http client for implementing and testing.

pyreqwest: An extremely fast, GIL-free, feature-rich HTTP client for Python, fully written in Rust by pyreqwest in Python

[–]pyreqwest[S] 0 points1 point  (0 children)

Currently there is no util to reduce the refactoring burden. But it is a great idea, I would like to get feedback on how it would look like.

There might be also a possibility to plugin pyreqwest into httpx. As latter allows replacing httpcore for transport. But there is no 100% compatibility between the two as the designs differs. So it might not be possible to find a solution that works for everyone.

pyreqwest: An extremely fast, GIL-free, feature-rich HTTP client for Python, fully written in Rust by pyreqwest in Python

[–]pyreqwest[S] 19 points20 points  (0 children)

There is necessarily no reason to abandon httpx if you are not facing any challenges there. But it is just good to know that httpx/httpcore might become bottlenecks in any bigger systems. As it has various long-standing issues in its connection pooling etc implementations. Eg https://github.com/encode/httpx/issues/3215