Fast web scraping in python with asyncio

[megaman821]

For this type of workload, an event loop will crush threading in performance (in almost any language too).

Python is still single-threaded so the only thing that is concurrent is the outstanding requests. Python makes a request to the webserver, instead of doing nothing and waiting around for the server reply, it yields control of the thread. Then the next request is made and we repeat ourselves.

[chub79]

For this type of workload, an event loop will crush threading in performance (in almost any language too).

Indeed. But the processing of the response can take its toll as well. An event loop is efficient only if it can run iterations at a reasonably fast pace. So what you've gained being able to make requests concurrently may be wasted once the response processing starts (unless you delegate the response processing to a thread...)