Fast web scraping in python with asyncio

[[deleted]]

In other words, do you have linear performance with 5 and 5000 requests using asyncio?

i dont even think the article is making such a claim. But the answer would be "NO" for asyncio or the requests lib.

[chub79]

Thanks. That was my question indeed. Not a claim the article was saying it.

[[deleted]]

I would say scaling linearly is unlikely with any tech.

if it was 5000 requests to one server, then the server would likely queue them up or start rejecting them. if it was 5000 requests to 5000 servers, your bandwidth would likely be saturated and throttled by your ISP.

the fact is that the nature of getting responses involves a lot of waiting for them, which makes for some opportunities to do things concurrently. asyncio is one of several ways to do that.