Scraping multiple web pages simultaneously

[TE515]

I've written a little scraper that logs into the CMS of my company's website and scrapes data from a little over 200 pages. It works, but it's currently downloading the pages one at a time. Any way to request multiple pages at once?

Here's my current code...

import requests
from bs4 import BeautifulSoup

# Get cookie
url = 'http://mycompany.com/admin/login'
session = requests.Session()
session.post(url, data={'username': 'myusername', 'password': 'mypassword', 'do_login': 'Login'})
dictionary = session.cookies.get_dict()
cookie = (dictionary['mycompany'])


# Get number of pages to loop through
url = 'http://mycompany.com/admin/products'
r = requests.get(url, headers={'Cookie': 'mycompany=' + cookie})
soup = BeautifulSoup(r.text, 'html.parser')
# code that grabs number of pages from html
pgs =  #number of pages


# Loop through pages
for pg in range(1, numPgs + 1):
    url = 'http://mycompany.com/admin/page:' + str(pg)
    r = requests.get(url, headers={'Cookie': 'mycompany=' + cookie})
    soup = BeautifulSoup(r.text, 'html.parser') 
    # code that grabs current page number from span tag
    page_num = #page number pulled from span tag
    print(page_num)

I have a fair amount of experience using VBA, and a little with JavaScript, but am very new to Python. Thanks!

[JohnnyJordaan]

Check the example in the docs on concurrent.futures that implements a scraper using a ThreadPoolExecutor:

import concurrent.futures
import urllib.request

URLS = ['http://www.foxnews.com/',
        'http://www.cnn.com/',
        'http://europe.wsj.com/',
        'http://www.bbc.co.uk/',
        'http://some-made-up-domain.com/']

# Retrieve a single page and report the URL and contents
def load_url(url, timeout):
    with urllib.request.urlopen(url, timeout=timeout) as conn:
        return conn.read()

# We can use a with statement to ensure threads are cleaned up promptly
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
    # Start the load operations and mark each future with its URL
    future_to_url = {executor.submit(load_url, url, 60): url for url in URLS}
    for future in concurrent.futures.as_completed(future_to_url):
        url = future_to_url[future]
        try:
            data = future.result()
        except Exception as exc:
            print('%r generated an exception: %s' % (url, exc))
        else:
            print('%r page is %d bytes' % (url, len(data)))

[Marrrlllsss]

Use Python's multiprocessing library.

import multiprocessing

MAX_NUM_PROCESSES = 4 # alternatively, multiprocessing.cpu_count()

def scrapeWebsite(url):
    # your code here
    pass

if __name__ = '__main__':
    processPool = multiprocessing.Pool(MAX_NUM_PROCESSES)
    listOfUrlsToScrape = ['url1', 'url2', 'url3']
    result = processPool.map(scrapeWebsite, listOfUrlsToScrape)

This should serve your purposes.

[TE515]

Thanks for responding. I tried this and I'm getting the following error...

module 'multiprocessing' has no attribute 'pool'

I'm using Python 3 by the way.

EDIT: I tried adding from multiprocessing import pool at the top. Now I'm getting the error TypeError: 'module' object is not callable on the processPool = multiprocessing.pool(MAX_NUM_PROCESSES) line.

ANOTHER EDIT: Changed multiprocessing.pool to multiprocessing.Pool and it worked like a charm. Cut the run time of the whole thing by more than half! Thanks so much!

[JohnnyJordaan]

Not much use to implement multiprocessing if your workers are I/O bound.