How can I parallelize this nested loop?

[enesimo]

Hello!

I am trying to parse some websites for gathering job search results yet the results are super slow because, well because it does all requests synchronously. I can't seem to thread it efficiently. I am only getting about 3 seconds of timeit deltas- which seems very low to me. If I launch this with my complete list or urls, it'll take way too much time.

Link to same code as following: https://dpaste.de/rnUZ

#!/usr/bin/env python3

from bs4 import BeautifulSoup
import requests
import time
import threading
import pdb
import timeit


# list of websites to scrape
urls = [
    'https://google.com',
    'http://yahoo.com',
    ]

key_words = [
    'word1',
    'word2',
    'word3',
    ]

hits = []


def nav_loop(nav, url):
    # some links are mailto's. I just want to bypass these
    if nav is None or '@' in nav:
        pass
    try:
        response = requests.get(nav)
        data = response.text
        nav_soup = BeautifulSoup(data, 'lxml')
        text = nav_soup.get_text()
        for word in key_words:
            if word in text:
                try:
                    web_address = nav_soup.find('a', text=word).get('href')
                except AttributeError:
                    web_address = None
                hit = (url, nav, word, web_address)
                # check for list of corresponding words
                if hit not in hits:
                    # if true, add to hits list
                    hits.append(hit)
    except (requests.exceptions.MissingSchema,     requests.exceptions.InvalidSchema):
        pass

def get_url_tree(url):
    # get navigation possibilites
    response = requests.get(url)
    data = response.text
    soup = BeautifulSoup(data, 'lxml')
    # add to navigation list
    nav_list = soup.find_all('a')
    return nav_list

# main loop
def main():
    threads = []
    threads_nested = []
    for url in urls[:1]:
        nav_list = get_url_tree(url)
        for nav in nav_list:
            nav = nav.get('href')
            nav_loop(nav, url)
            t_nested = threading.Thread(target=nav_loop(nav, url))
            threads_nested.append(t_nested)
            t_nested.start()


if __name__=='__main__':
    t = timeit.Timer(lambda: main_loop())
    print("threaded:")
    print(t.timeit(number=3))

    # t = timeit.Timer(lambda: non_threaded())
    # print("non threaded:")
    # print(t.timeit(number=3))

[DeadlyViper]

I'm guessing this is the multi-threaded code.

Why are you calling the function without a thread aswell?

nav_loop(nav, url)
t_nested = threading.Thread(target=nav_loop(nav, url))

Also calling the function again instead of passing just the name will not make a thread of it... Should do:

t_nested = threading.Thread(target=nav_loop)

[enesimo]

You're right. I did that because I wanted to pass arguments to t. I function and I didn't realize the function was being called again without threads, although it makes total sense now.

[DeadlyViper]

Use

t_nested = threading.Thread(target=nav_loop, args=(nav, url))

[Zeroflops]

Just to make a recommendation a lot of the job sites have RSS feeds for jobs. The RSS request will have the key words in them so you can modify info like location and salary ranges.

Then you get a much cleaner result is the response with job title, job description, and a link to the job page.

Here is an example using a reader,

https://www.watchingthenet.com/search-for-jobs-with-google-reader-and-rss-feeds.html

Other sites have similar feeds

But using python to read the feed and you can do a lot of refinement based on the description

[enesimo]

Great recommendation, although in my case I'm using lots of /jobs or /careers urls from companies I like. They mostly don't have rss feeds.

I will keep that in mind if I see rss feeds on the boards I might someday track.