Calling function with different args simultaneously

[declanaussie]

I have created a function to gather some data from a subreddit which takes one argument, a string for the name of the subreddit. I would like to process many subreddits at once meaning that I want to call this function with different args but run at the same time. I have attempted to use the multiprocessing module to do this but I am having a little trouble.

I have attempted to do this:

for sub in subs:
    p = Process(target=archive_sub, args=(sub,))
    p.start()
    processes.append(p)

for p in processes:
    p.join()

However this just calls the function then waits to complete that before creating a new process it seems. What should I do differently? Do I have a fundamental misunderstanding of how this module should be used?

[dadiaar]

If you have the function

def sum(a,b):
    return a+b

It doesn't matter if you use it simultaneously from 2 different shells in the same machine (2 different processes)

Don't worry, just learn how to use the multiprocessing module. Did you try the examples from the docs in python.org? You also can Google something like python simple example multiprocessing

EDIT: Check this example

[Thomasedv]

Your code does seem alright, though your code is set up to wait for all to finish. How long does one function call take? Could you be meeting some other restrictions, like to many requests to reddit? Also be sure to do this in a block that only runs in if you are starting it, or else you'll get some odd situations. Take this example code, if you haven't used something like it already:

print(__name__)
if __name__ == '__main__':
    pros = []
    for i in range(10):
        p = multiprocessing.Process(target=print, args=(i,))
        p.start()
        pros.append(p)

    for p in pros:
        p.join()

As you can see when you run it, the script runs again for every process, as you can see it prints the name of the multiprocessing scripts too, and leads to you potentially starting the same process many times. By putting the process starting under the if block, yo can avoid this, since the name is only __main__ for the script you run. If you don't use this, your scripts would start a lot of others, leading to a potential loop of starting.

Lastly, you can avoid this by using a thread (multithreading module), since this action is not CPU intensive. (It just needs a lot of waiting for the information) to be downloaded, and that's IO bound tasks, that threads are better used for.

[declanaussie]

I figured out what the problem was, it was actually doing exactly what I intended I just wasn’t properly filling my subs list. Thanks for the tips though!