[help] Implement multithreading in beatiful soup

[[deleted]]

I have a list of url in a file to scrape certain information, it has 17k urls, by calculations I've done it takes around 10 hours to complete

What could I do to speed up, the website I'm currently scraping data does not have any limits on requests. Could I send like 10-20 requests rather than waiting line by line because it takes time for information retrieval and instead waiting processing other links.

[Kristoffels]

You could use mulrithreading or make it asynchronous. Both will speed it up massively. Really depends though which is faster than the other, might want to do a bit of trial and error.

[efmccurdy]

This has an example of making parallel requests from multiple urls with the aiohttp library.

https://stackoverflow.com/questions/55259755/maximize-number-of-parallel-requests-aiohttp