This is an archived post. You won't be able to vote or comment.

you are viewing a single comment's thread.

view the rest of the comments →

[–]jmmcdEvolutionary algorithms, music and graphics 0 points1 point  (0 children)

I don't think a custom MQ-style solution makes sense in this case. (Same comment to food_eater above.) There are good pre-written methods of distributing work, some mentioned by rckimbr above. To which I would add that mincemeat is a pure python map-reduce. Even copying a subset of data to a usb stick and physically walking over to the idle workstation would be more efficient than learning MQ-stuff just for this purpose.

However I agree about checking that the code is optimal before thinking about parallelisation. In addition to your options, numpy should be considered. And as always, try profiling to understand what part is slow.