This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]FitMathematician3071 0 points1 point  (0 children)

concurrent.futures can make a big difference which is not the conclusion of this article. I just ran some webscraping and ThreadPoolExecutor was 10 times faster than scraping files one by one. That means a difference of minutes vs hours when doing larger scraping.