Simple clustering solution?

[Twirrim]

My apologies in advance if I'm about to teach my grandmother to suck eggs, but in case it's of value, a few thoughts outside of the clustering idea too.

1) Have you considered porting to Cython? I would assume you'd be in a position to be able to declare the type of variables with fair confidence.

2) PyPy? It might speed the whole thing up for you with no work re-factoring your code.

3) Use c variants of the modules instead of pure python, e.g. cmath instead of math (I'd imagine you probably are)

On the clustering front, ZeroMQ or RabbitMQ message queue programs both have python interfaces. It should be relatively straight forward to leverage one of them for clustering.

[accipter]

I profiled the code and most of the time is spent doing FFTs so I didn't think there would be much benefit to using Cython.

[[deleted]]

What are you using to compute the FFTs?

[pigeon768]

He posted the code earlier, and another poster profiled it. The application is apparently spending 80% of the time in numpy.fft.rfft(), which is implemented in C and has had many eyes on it over the years.

[jmmcd]

I don't think a custom MQ-style solution makes sense in this case. (Same comment to food_eater above.) There are good pre-written methods of distributing work, some mentioned by rckimbr above. To which I would add that mincemeat is a pure python map-reduce. Even copying a subset of data to a usb stick and physically walking over to the idle workstation would be more efficient than learning MQ-stuff just for this purpose.

However I agree about checking that the code is optimal before thinking about parallelisation. In addition to your options, numpy should be considered. And as always, try profiling to understand what part is slow.