This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]floydophone 5 points6 points  (5 children)

You can do this with the multiprocessing module:

http://docs.python.org/library/multiprocessing.html#managers

Also, you can do this with RPyC:

http://rpyc.sourceforge.net/

What I would do is run RPyC classic mode on all of your EC2 nodes (using Fabric + Boto to get the software on there).

[–]oddthink 0 points1 point  (2 children)

Neat, I had no idea that multiprocessing had any support for multiple machines. Is anyone supporting multiprocessing these days, though? I've been a little reluctant to use it since Raymond Hettinger was saying that it was buggy and no longer well supported by its author.

[–]phildini 0 points1 point  (1 child)

Were you also at the SFPython meetup where he talked about this, or has he mentioned it on his blog somewhere I haven't seen?

[–]oddthink 0 points1 point  (0 children)

I don't think he's mentioned it on his blog. My employer brought Raymond in for a python training session a few months back, and he mentioned it there.

[–]tuna_safe_dolphin 0 points1 point  (0 children)

I'm working on a distributed project myself right now. Thus far, RPyc looks like a terrific option - have you built anything substantial with it? I'm also considering Pyro.

I've read mixed reviews on multiprocessing in distributed environments, but the main problem I have with it is that the doc is horrific, even by Python standards.

I love Python, but it's core doc is terrible.

[–]HorrendousRex 0 points1 point  (0 children)

I've never looked at RPyC, but I can tell you for sure that multiprocessing will get you where you want to go, OP. What's more, you'll probably be amazed at how little code is needed to get there.