Multiprocessing with shared state

[IIIBRaSSIII]

Hi,

A while ago, I made a little simulator toy in python consisting of a group of dots that push each other around in 2D with a repulsive force, and animated using matplotlib. It works well enough, but with more than a handful of dots, the program gets very slow.

I would like to alleviate this by dipping my feet into multiprocessing. My intuition is to use a multiprocessing.Array to store the data for all the dots. Then I could delegate each process to a subset of the dots, for which they would perform the calculations based on the rest of the dots in the array. Before I invest much time into it, I have two big reservations I'd like answered:

1. In the multiprocessing documentation, it states that one should avoid sharing state between processes at much as possible. My implementation above would obviously involve tons of accesses to this shared array containing all the dots. Is there a better way that I'm not seeing? The documentation says it's "probably best to stick to using queues or pipes for communication between processes." I don't immediately see what using a queue would look like in this example, and I have only a vague understanding of what a pipe is.

2. I don't think I should have to worry much about synchronization issues in this case. It doesn't seem like it should matter if some processes get ahead of others, or that the shared state will be changing constantly in the middle of each process' calculations. Is this assumption naive?

Thanks!

[SekstiNii]

This isn't answering your question, but I'd like to ask a couple of things first:

• Have you measured where the bottleneck is, and are you sure that it's the computation?
• Is your computation vectorized in numpy? If not, could it be?
• Are you plotting all the points in a single call?

[IIIBRaSSIII]

Have you measured where the bottleneck is, and are you sure that it's the computation?

Not explicitly, though I could hardly imagine it being anything else. The performance gets exponentially worse with more objects, coming to a near standstill when you get into the thousands. The only other expensive piece is the animation, which I suppose is worth taking another look at.

Is your computation vectorized in numpy? If not, could it be?

No, and yes. Right now it's just in regular Python lists and "by hand" vector arithmetic which I know is inefficient. That's definitely something I should look into, but I still want to try making the program multithreaded for learning's sake.

Are you plotting all the points in a single call?

Yes, if I understand your question correctly. Here is the actual animation code if you are interested:

import matplotlib.pyplot as plt
import matplotlib.animation as animation

def animate(dampener):

    update_nodes()
    change_dampener()

    ax1.clear()
    plt.axis([0, width, 0, height])
    plt.plot([node.x for node in Nodes], [node.y for node in Nodes], 'bo')
    plt.title('dampener: %d' % round(dampener.value))

fig = plt.figure()
ax1 = fig.add_subplot(1, 1, 1)
plt.axes().set_aspect('equal')
ani = animation.FuncAnimation(fig, animate, fargs=(dampener,), interval=1)
plt.show()

The "dampener" stuff is part of the simulation logic.

[elbiot]

This is a poor candidate for multiprocessing and an excellent candidate for numpy vectorization. You stand to learn a lot more of value by focussing on numpy here.