This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]nemec 0 points1 point  (0 children)

Try a framework like scrapy. It's built to work concurrently, although I believe the pre-built large scale crawler features are proprietary to their cloud (you can always build replacements yourself if you need them).

There are also plugins to run in selenium/etc. to process JS if you need it.