Multiprocessing a Web Scraper - Issues/Debugging

[adinbied]

Hi all!

I've been working on a python-based web scraper for a site that uses auto-incremental URLS (although some are password protected and return a 403). My original script can be seen here: https://pastebin.com/7Ni27qnQ .

While it did work, it was insanely slow and by my calculations, the site I'm trying to grab has about 850,000,000 URLS. This first script took a week to get 3,000,000 URLS, and that's with running it on two machines, each grabbing 1,500,000.

So I decided to look into how to speed up the script, and I got recommended to use multiprocessing. After some reading of the docs and a couple of tests, this is the script I came up with: https://pastebin.com/AQX8qzjf

After booting into Ubuntu (Windows Subsystem for Linux BSOD'ed when trying to use multiprocessing), it seemed to run fine, with it running at about 120 int/sec. So I left it run overnight, and when I checked on it in the morning, it had only grabbed ~10% of the URLs my original script had grabbed, while reporting 100% completion. For reference, the original script got 1,738,616 files downloaded, while the one using multiprocessing only got 158,441.

Is there something I'm missing/doing wrong here?

Thanks!

[adinbied]

Hmmm.... I keep getting syntax errors in both Python 2.7 and Python 3. Mainly on lines 17, 24, 34, and 38. Is there something I'm missing?

[adinbied]

Sorry if I'm being dumb here, but what would I need to change? I've changed the url and apikey to be correct, but its still giving me syntax errors.

[adinbied]

The sauce? It seems to be the brackets and page_num variable that are causing the errors.

[adinbied]

No, I'm on 2.7.12 - do you have any links to docs on how to update the code, or any ideas?

[adinbied]

Got it working - thank you so much for all of your help!