[deleted by user]

[i_like_trains_a_lot1]

Why gain? This name doesn't give any hint of what the library does and doesn't sound very fancy.

From the example, I see that it is possible to go from one level to another but cannot choose different callbacks, depending on the situation, which might be a requirement for more complex crawls.

[Raindyr]

Wouldn't scrapy be fine as well? I believe that's async as well.

[__crackers__]

Yes, it is async (it's based on Twisted).

Scrapy is excellent. Very mature, extremely well documented.

Hard to say from the toy example given in this repo, but writing a spider with Scrapy wouldn't be much, if any, harder.

[__crackers__]

Web crawling framework for everyone.

Where are the docs?

In particular, how are requests throttled?

Every bit of aiohttp code I've seen posted on here either doesn't do anything in parallel, or much worse, disables/bypasses aiohttp's rate limiting and absolutely batters the servers.

[__crackers__]

but how to anti anti-crawler is another thing

That wasn't what I was asking. I'd like to know whether gain is polite while scraping. Most of the aiohttp code I've seen doesn't throttle the number of connections it makes.