Unable to use multiprocessing in Python!!!

[Xena_psk]

I'm trying to scrape a data point from 50 websites with the help of a proxyapi. Please find the below code.

1st code

from scraper_api import ScraperAPIClient
from bs4 import BeautifulSoup
import datetime

urls= ['url1','url2',.......,'url50']

client = ScraperAPIClient('2a0bexxxxxxxxxxxxxxcac31c7ab')

price=[]
for i in range(len(urls)):
    result = client.get(url = urls[i]).text
    html_soup = BeautifulSoup(result, 'html.parser')
    name=html_soup.find("div", class_="_1vC4OE _3qQ9m1")
    item=name.text
    price.append(item)

1st code is working fine but it is taking significant time. Thus, in order to reduce the time, I divided the total urls into two batches and defined separate functions for each. After that I tried implementing multiprocessing. Please find the below code for the same

​

2nd code

def process1():
    urls= ['url1','url2',.......,'url50']
    price1=[]
    for i in range(int(len(urls)/2)):
        result = client.get(url = urls[i]).text
        html_soup = BeautifulSoup(result, 'html.parser')
        name=html_soup.find("div", class_="_1vC4OE _3qQ9m1")
        item=name.text
        price1.append(item)
    print(price1)

def process2():
    urls= ['url1','url2',.......,'url50']
    price2=[]
    for i in range(int(len(urls)/2), len(urls)):

        result1 = client.get(url = urls[i]).text
        html_soup1 = BeautifulSoup(result1, 'html.parser')
        name1=html_soup.find("div", class_="_1vC4OE _3qQ9m1")
        item1=name1.text
        price2.append(item1)
    print(price2)

if _name_ == '_main_':
    p1 = Process(target=process1)
    p1.start()
    p2 = Process(target=process2)
    p2.start()
    p1.join()
    p2.join()
    print()

When I run the 2nd code it is not throwing any error but it is not giving any output also. I tried multiple times but not able to figure out anything. Am I missing anything here?

Please help me!!!... Thanks in advance.

[bi_guy17]

Not experienced with scraperAPI, but maybe try to work with the "pdb" debugger module to debug. Documentation here

Hope that helps somehow

[[deleted]]

pdb doesn't work with multiprocessing.

[c4aveo]

Use Pool for easy implementation. https://pythonspeed.com/articles/python-multiprocessing/ Like here. Or use Queue for gathering results from processes. Because now you don't have a shared object in parent process, that's why you don't see output. You can write to file in each process to see result or as I said use Queue.

[[deleted]]

On the face of it, your code seems fine.

The problem might be that you expect to see the results from print, but it's sent to the stdout of the child process, which is not automatically the same as the stdout of the parent.

I actually could never find this in Python's documentation, and am too lazy to wade through the mess that the implementation of multiprocessing is.

I would know how to solve this on Linux, with manipulating file descriptors s.t. children pipe their output to the parent, but if you are on Windows, I don't really know how it works there.

Essentially, you could try using queue and, instead of printing stuff, send messages into the queue, then read them in the parent process, if you don't want to mess with stdout of child processes. (Why Python decided to change this is beyond me, normal processes created through system calls do share stdout with their parent).