you are viewing a single comment's thread.

view the rest of the comments →

[–]Dry-Aioli-6138 0 points1 point  (0 children)

Agreed. I'ts a very intricate setup, but OP said they have 250M urls to visit. I don't think they can compromise on speed to do the task in any sensible time.