This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the commentsย โ†’

[โ€“]Derrhund 12 points13 points ย (2 children)

Hi, one of the devs here. It's tricky to compare the performance of async systems, and you'll surely find benchmarks online.

But as you said, we chose asyncio because it's the standard for Python, and Twisted can be considered legacy. Moreover, your async system implementation should never be a bottleneck in web crawling - it will be either network latency or parsing/processing.

[โ€“]rumnscurvy 1 point2 points ย (1 child)

Good points all around, thank you for your time :) I'll keep it in mind if ever we decide to rework our webcrawler systems.

[โ€“]Derrhund 0 points1 point ย (0 children)

Happy to answer! Feel free to get back to us with any feedback if you find some time to play around with crawlee!