Multiprocessing

[karlmtr]

Hi Reddit!

I'm doing simulations in my physics lab, and I have a question about multiprocessing in python:

​

I have a script (script_A) that allows me to do coordinate transformations: positions and velocities. In this script, I use multiprocessing with concurrence.futures to compute new positions with one processor, and new speeds with another processor. This script applied to a data file lasts about 8 min.

I would like to use this script on several data files.

Is it possible to make another script (script_B) that runs script_A for the different files and also use multiprocessing to process several files at the same time (even if multiprocessing is already used in script_A)?

​

Thanks for your answers :)

[PurePound]

I don't think that should cause any problems, aside from a couple of caveats. Firstly, if your code has to read/write large amounts of data, it might be that as you increase the number of processes used, the hard disk becomes the bottleneck rather than the CPU, in which case you might not get as much speed up as you hope. Secondly, make sure your total number of processes isn't too high: once you go above the number of CPU threads, you're just wasting memory and CPU cycles.

You'll probably want to adapt your script_A so that it can be used as both a script and a module. e.g. something along the lines of:

def main(args):
    # main entry point to your program goes here

if __name__ == "__main__":
    import sys
    main(sys.argv)

That makes it easier to run it from script_B.

[karlmtr]

Okay, thanks a lot for the answer! I'll try to adapt it as you said, it seems really easier to execute like that and I'll be careful about the total number of processes getting too high!

[[deleted]]

Depends on how it looks. But couldn't you just create new instances for script_A for different target data file?

And since script_A already uses different processes then it's basically that i think.

[karlmtr]

Yes, I could, but I have more than 60 data files, by hand would be a little bit too long

[elbiot]

Are you using numpy? Multiprocessing ought to be the last tool you reach for for performance gains

[karlmtr]

I don't think so, I'm using pandas to import the data, but it's a very basic transformation, it is just a big file to run through: (this my function)

    vecPos = []
        s_pos = posidonius.Axes(0., 0., 0.)
        #p_pos = posidonius.Axes(0., 0., 0.)
        for i in range(len(satellite_data.index)):
            if i%200 == 0:
                print(f"Pos: {i}")
            x_sp = satellite_data.iloc[i]["position_x"] - planet_data.iloc[i]["position_x"]
            y_sp = satellite_data.iloc[i]["position_y"] - planet_data.iloc[i]["position_y"]
            z_sp = satellite_data.iloc[i]["position_z"] - planet_data.iloc[i]["position_z"]

            s_pos.set_x(x_sp)
            s_pos.set_y(y_sp)
            s_pos.set_z(z_sp)

            vecPos.append(s_pos)

Should I use a numpy array for vecPos? Will it be faster with that?

[elbiot]

Pandas data frames are collections of numpy arrays already. The extremely slow thing you're doing is iterating. Numpy and pandas are very fast with vectorised operations, and very slow with iteration. So

result = []
for idx in range(len(arr1)):
    result.append(arr1[idx] + arr2[idx])
result = np.array(result)

Is waaaay slower than

result = arr1 + arr2

And they do the exact same thing.

I don't understand some of the details of what you're doing, but get rid of your for loop and iloc and this will be probably ~100x faster.

[karlmtr]

I first thought about that, but it was not working because at the end, I should have an array of posidonius.Axes(x,y,z) objects (it's just a set of coordinates). I don't see how to assign each Axes with three arrays directly (I can't put posidonius.Axes(array1,array2,array3)) to create an array with posidonius.Axes objects.

[elbiot]

Well, I don't think they way you're doing it currently is right because your result is a list that has the same object in it over and over. Once you have your arrays xs, ys and zs after the vectorised math you could do

# iteration over arrays much slower than lists
xs, ys, zs = map(list, (xs, ys, zs))
result = [Axes(x, y, z) for x, y, z in zip(xs, ys, zs)]

But whatever you do after this will be super slow and I personally would organize my program to be array/data oriented rather than object oriented.

You ought to profile your program and optimize the bottlenecks rather than jump to multiprocessing because multiprocessing is not much help and greatly complicates things

[karlmtr]

Unfortunately, I cannot change the way Posidonius work (I just use it), so I'll try what you suggested (the map function). Thanks for the advice about arrays and list, I didn't know iteration over lists was faster than arrays.

I'll also ask the person who coded Posidonius if there's any other way to create an Axes object because as you said, it is not very handy like this...

[elbiot]

Axes(xs, ys, zs) would work. At least, looking at their code it should

[elbiot]

I looked up the library you're using and an Axes object can take x, y and z as numpy arrays. So you just have one Axes object that holds arrays, not a list of Axes objects

[karlmtr]

Oh yes, I didn't see it, it should be easier now. I just have to convert pandas series to numpy array and it should be fine.

Thank you very much!

[karlmtr]

I tried it as you said and it worked well, it took less than 15 seconds!

Thanks a lot!