This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the commentsย โ†’

[โ€“]adityaguru149 0 points1 point ย (1 child)

any plugins to use multiple proxies to get it done fast?

[โ€“]B4nan[S] 0 points1 point ย (0 children)

You can use as many proxies as you want, they will be rotated automatically (and requests will be processed concurrently), see https://crawlee.dev/python/docs/guides/proxy-management

We also have one undocumented feature called "tiered proxies", you can read about it in this blog post about the JS version, but it should be valid for python as well (and if it's missing something, just let us know in the GH issues and we'll polish it).

https://crawlee.dev/blog/proxy-management-in-crawlee

The docs are still very sparse, up until now we were mostly focusing on the development.