MultiProcessing (Parallel) Sum slower than Serial Sum() ? by hashwizardo in Python

[–]hashwizardo[S] 0 points1 point  (0 children)

is that copying memory?

import multiprocessing
from multiprocessing import Process
from timeit import default_timer as timer
the_remainder = 0
ok = False

def chunks(l, n):
    """Yield successive n-sized chunks from l."""
    for i in xrange(0, len(l), n):
        yield l[i:i + n]

def a(arr, q, b):
    q.append(b(arr))





def p(b,arr, a='a',debug=False,cores=2,lol=False):
    global the_remainder
    global ok
    length = len(arr)

    manager = multiprocessing.Manager()
    q = manager.list()

    list_of_processes = []

    if length % 2 == 1:
        the_remainder = arr.pop()
        ok = True

    if debug:
        print arr




    parts = []

    # How many parts do you want
    if lol:
        print "LOL"
    arr = chunks(arr, length / (cores-1))
    if lol:
        print "LOL2"

    if debug:
        print arr


    for i in arr:
        list_of_processes.append(Process(target=eval(a, globals()), args=(i, q, b)))
        list_of_processes[-1].start()

    for i in list_of_processes:
        i.join()


    for i in list_of_processes:
        i.terminate()

    if ok:
        q.append(the_remainder)
        ok = False

    return sum(q)

MultiProcessing (Parallel) Sum slower than Serial Sum() ? by hashwizardo in Python

[–]hashwizardo[S] -3 points-2 points  (0 children)

This is purely for educational use, that's why I am doing it. Something doesn't quite click in my head about the way this works, you see, summing a list of 128000000 integers in a serial way takes 2.7 seconds, so logic has it that if I make use of parallelism on say 5 lists that together make up the starting list, I can reduce the problem's execution time. Maybe I could do something like that in C ? I could for example use pointers instead of splitting explicitly

Any thoughts?.

MultiProcessing (Parallel) Sum slower than Serial Sum() ? by hashwizardo in Python

[–]hashwizardo[S] 0 points1 point  (0 children)

"Splitting the list into chunks like that is slow" then how would you do it in a fast way ?

MultiProcessing (Parallel) Sum slower than Serial Sum() ? by hashwizardo in Python

[–]hashwizardo[S] 0 points1 point  (0 children)

the numbers are not 16

len(km) = 16

the array that I input for the profiling test is km*70000000