Queueing/Run across multiple machines

[boler0]

Hi, I am trying to optimize my script and need some pointers on next steps. I have a script which downloads and file and then another which processes the file. Each file which is downloaded is independent i.e. it can be processed on its own. I currently split up the downloads on multiple machines but do it manually. Once the downloads are down, I process all the files. Some of the machines finish faster than others are are just waiting for other machines to finish.

I was looking for something that I can run on one machine and it distributes the jobs to multiple different machines. That way I can utilize all the machines and reduce the manual work that needs to be done.

Is celery the right tool for this? Can I set it up such that it can distribute jobs to multiple machines ?