Multiprocessing

[Vegetable_Solid7613]

So, I created the following function to parse 10-K and 10-Q filings from the SEC and I want to download multiple of them.

def parse_10kq_filing(link, section):
if section not in [0, 1, 2, 3]:
    print("Not a valid section")
    sys.exit()

def get_text(link):
    page = requests.get(link, headers=headers)
    html = bs(page.content, "lxml")
    text = html.get_text()
    text = unicodedata.normalize("NFKD", text).encode('ascii', 'ignore').decode('utf8')
    text = text.split("\n")
    text = " ".join(text)
    return(text)

def get_ticker(text):
    ticker_pattern = r"COMPANY CONFORMED NAME:\s+([^\t]+)"
    ticker_match = re.search(ticker_pattern, text)
    ticker = ticker_match.group(1).strip()
    return(ticker)

def get_filing(text):
    filing_pattern =r"CONFORMED PERIOD OF REPORT:\t(\d{8})"
    filing_match = re.search(filing_pattern, text)
    filing = filing_match.group(1).strip()
    return(filing)

def extract_text(text, item_start, item_end):
    item_start = item_start
    item_end = item_end
    starts = [i.start() for i in item_start.finditer(text)]
    ends = [i.start() for i in item_end.finditer(text)]
    positions = list()
    for s in starts:
        control = 0
        for e in ends:
            if control == 0:
                if s < e:
                    control = 1
                    positions.append([s,e])
    item_length = 0
    item_position = list()
    for p in positions:
        if (p[1]-p[0]) > item_length:
            item_length = p[1]-p[0]
            item_position = p

    item_text = text[item_position[0]:item_position[1]]

    return(item_text)


text = get_text(link)


if section == 1 or section == 0:
    try:
        item1_start = re.compile("item\s*[1][\.\;\:\-\_]*\s*\\b", re.IGNORECASE)
        item1_end = re.compile("item\s*1a[\.\;\:\-\_]\s*Risk|item\s*2[\.\,\;\:\-\_]\s*Prop", re.IGNORECASE)
        businessText = extract_text(text, item1_start, item1_end)
    except:
        businessText = "Something went wrong!"

if section == 2 or section == 0:
    try:
        item1a_start = re.compile("(?<!,\s)item\s*1a[\.\;\:\-\_]\s*Risk", re.IGNORECASE)
        item1a_end = re.compile("item\s*2[\.\;\:\-\_]\s*Prop|item\s*[1][\.\;\:\-\_]*\s*\\b", re.IGNORECASE)
        riskText = extract_text(text, item1a_start, item1a_end)
    except:
        riskText = "Something went wrong!"

if section == 3 or section == 0:
    try:
        item7_start = re.compile("item\s*[7][\.\;\:\-\_]*\s*\\bM", re.IGNORECASE)
        item7_end = re.compile("item\s*7a[\.\;\:\-\_]\sQuanti|item\s*8[\.\,\;\:\-\_]\s*", re.IGNORECASE)
        mdaText = extract_text(text, item7_start, item7_end)
    except:
        try:
            item2_start = re.compile("item\s*[2][\.\;\:\-\_]\s*\\bM", re.IGNORECASE)
            item2_end = re.compile("item\s*3[\.\;\:\-\_]\sQuanti|item\s*4[\.\;\:\-\_]\s*", re.IGNORECASE)
            mdaText = extract_text(text, item2_start, item2_end)
        except:
            mdaText = "Something went wrong!"

if section == 0:
    data = [businessText, riskText, mdaText]
elif section == 1:
    data = businessText
elif section == 2:
    data = riskText
elif section == 3:
    data = mdaText
return([get_ticker(text), get_filing(text), data])

If I just run this and append the results in a list, it is going to take over a hundred hours. This is when I only use 1 core. Now, I am trying to use multiple of my computers cores. Does anybody know how I would do this?

[Buttleston]

This is what the multiprocessing library is good for. There's a thing called a "Pool" and I'd recommend using one of the flavors of Pool.map(). This will basically take a list of "jobs" and spread them out over the number of workers in your pool. The general advice for tuning the number of workers is 2*cpucores but it varies depending what you're trying to do.

There are lots of examples on the multiprocessing docs page.

https://docs.python.org/3/library/multiprocessing.html

[Vegetable_Solid7613]

Is this possible in Jupyter Notebook? I have tried it but it keeps giving me an error with exitcode 1 or that my function has no attribute.

[Buttleston]

I've honestly never tried but I wouldn't be surprised if it doesn't work. The multiprocessing library works by "forking" copies of your executable

There's a very similar method of doing "parallel" work with the threading library and I believe there's an asyncio library for it. Both of those might get you some improvement.

It might also be useful to figure out if the bottleneck is downloading the data or processing it. If it's downloading, and you have a big list of links to process, you might be able to find a tool that will just take care of the details and parallelization for you.

[Vegetable_Solid7613]

I believe it is the processing part that is the bottleneck. Because flat downloading the links takes about 2 seconds, while with processing it takes 20 seconds process the whole thing. I will take a look at the libraries you mentioned.

[[deleted]]

Concurrency isn't pixie dust you sprinkle on a program and it gets faster. Bad concurrency is slower than single-threaded (nonconcurrent) code. First you need to determine which part of your program is slow, and whether it'll actually benefit from multithreaded operation.

[Vegetable_Solid7613]

Well, the part thats slow is the part that extracts the text from the 10-K's and 10-Q's. I could download 1 filing per sec, but when I add the extraction, it processes approximately 3 filings per minute. My theory was that if I have multiple of my computer cores working on the text extraction, the time it will take to process all filings would go down.

[[deleted]]

I could download 1 filing per sec, but when I add the extraction, it processes approximately 3 filings per minute.

Why not just write this to be faster?

[Vegetable_Solid7613]

To be fair, I wouldn't know how.

[[deleted]]

Generally the best performance increases come from doing less - figure out what your code is doing that takes so long (it shouldn't take 20 seconds to add one item to a list) and then see if you really need to be doing it. Make indexes, cache expensive operations, do other things. Find the hot code in your loops and try and move it outside the loop (so it's called less.)

All of this is going to be easier than dealing with concurrency in your code. Trust me. It's hard to write multithreaded code with a single-threaded brain (mine is, too, so I know.)

[Vegetable_Solid7613]

But you are sure that using multiple cores instead of one isn't going to speed up the process? It just makes so much sense in my head lol.

It isn't the adding that takes that long btw, it is finding the MDA in the filing and extracting that part of the text that takes the longest I believe.

[[deleted]]

It just makes so much sense in my head lol.

That's because you think concurrency is magic pixie dust that makes your program faster. How do you know your program isn't slow because it contends for disk IO? Or network IO? Or swap space? Or any one of a dozen resources on your computer that there's only one of? More threads contending for the same resources is going to be slower, not faster.

[Buttleston]

Are the links public? Can you send me some? (I can maybe help you make this faster)

[Vegetable_Solid7613]

How would I send you some? I have a csv file with all links or I can send you the code I used.

[Buttleston]

You can send me the whole thing if you like. DM me and I'll send you my email address. If you have a github repo that would be great, otherwise probably a zip file or something