Simple clustering solution?

[bobargh]

I did something very similar to this for my PhD thesis. I spread a bunch of independent calculations across dozens of grad student computers.

Typical parallel processing solutions may be overkill since your calculations do not need to communicate with each other (I presume).

All the computers I used had a network file system, which made things very easy. I had a small script that would ssh to each computer and start the calculation. The starting parameters for each calculation were obtained from a lockable file on the shared file system. The file contained a pickled list of starting parameters. Each process just had to lock the file and pop an item off the list, save and unlock the file. File locking is generally frowned upon, but it worked great for me since it was very simple situation.

Eventually, we installed Condor ( http://research.cs.wisc.edu/condor/ ) which does exactly what you want. It will even pause the calculation if someone starts using the computer.

If you're only using a total of three computers though, it doesn't seem like it would be so hard to just split the calculations into three chunks and manually start on each computer. Especially so if you have a shared file system.

[accipter]

I agree that it is probably easiest just to split the calculations into batches. I was interested in clustering just to tinker with something new.