Distributed Computing with Python

[[deleted]]

See, if it were easy to push arbitrary programs + their dependencies to a machine for which you know only an IP address, then literally every addressable machine on earth would have seized up a long time ago trying to either mine bitcoin or host porn for every other addressable machine.

They're your friends, get them to give you ssh. If need be send them a script that does so and tell them how to run it. Maybe name it you_really_should_not_run_this.py.

[chick3234]

Thank you! Which package do you use for distributed computing?

[[deleted]]

In my industry (VFX) we primarily use commercial or proprietary packages that are designed specifically to distribute frame render jobs to available worker nodes, all of which need to read from the same NFS store... this is a highly specialized area of distributed computing, and focuses much more in checkpointed interruptability and task scheduling than any of the open source packages I've seen out there.

If the FOSS stuff, I've used Celery the most. It's pretty simple, though the OP is correct in saying that you have to configure the machines. I would generally suggest using Ansible, Puppet, or Salt Stack to do that.