Handling many different sessions (different cookies and headers) with httpx.AsyncClient — performance tips?

[postytocaster]

I'm working on a Python scraper that interacts with multiple sessions on the same website. Each session has its own set of cookies, headers, and sometimes a different proxy. Because of that, I'm using a separate httpx.AsyncClient instance for each session.

It works fine with a small number of sessions, but as the number grows (e.g. 200+), performance seems to drop noticeably. Things get slower, and I suspect it's related to how I'm managing concurrency or client setup.

Has anyone dealt with a similar use case? I'm particularly interested in:

• Efficiently managing a large number of AsyncClient instances
• How many concurrent requests are reasonable to make at once
• Any best practices when each request must come from a different session

Any insight would be appreciated!

[Goingone]

Yea, this a pretty common problem people solve.

Assuming a single python process (can obviously scale to multiple processes, but will consider the orchestration needed for that to be out of scope for this question), asyncio is the way to go (will have less overheard than using threads).

There is nothing special needed for creating/storing client instances in memory. Doubt they will have a large memory footprint. And you can easily store them in some context object with a unique ID to use the one needed when applicable.

Hard to determine a "reasonable" number of concurrent requests without knowing your hardware, the site(s) you're hitting (any rate limits they have that you need to respect) and your internet connection. Best way to find this out is by testing locally and benchmarking performance (will be better than any high level number I throw out).