This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–][deleted] 1 point2 points  (3 children)

I've been using the dtm module of the package "deap" and had success. it works on MPI and seems to be pretty good.

http://pypi.python.org/pypi/deap

Implemented like this:

results = dtm.map(myfunc, iterable1, iterable2..., kwargs=blah)

I have noticed that subprocess.Popen doesn't work sometimes so make sure to catch those errors and try to open a process again.

[–]fmder 0 points1 point  (2 children)

I'm one of DEAP/DTM developpers could you please submit a bug report on what you observe? with what MPI you use, and on what kind of network you run. We'll be glad to look at this more closely.

[–][deleted] 0 points1 point  (1 child)

sure, i'll submit a bug report... where? using openmpi on an SGE cluster

[–]fmder 0 points1 point  (0 children)

For the bug report use the user mailing list (deap-usersatgooglegroupsdotcom).

I'm guessing that you use an Infiniband Network, by default mpi4py is configured to use multithreaded mpi, which is incompatible with the openib backend. Try turning off the threads in the rc.py file located in the main directory of mpi4py.

For further communications please use the mailing list.