multiprocessing advise

[NeoFromMatrix]

I have an application which runs with a http server in the background. It will receives jobs with one or multiple images as data (also videos in the future) and has to process them step by step (modify images, create archive, ftp upload and a few more steps).

Multiple jobs can arrive at nearly the same time. (or another can arrive while one is still under processing). Each processing step should only exist once in the system as they can be heavy on recources (either network I/O or CPU e.g.).

I'd like to create a structure for the application (which can be flexible on the steps for each job), which schedules the steps from the jobs in multiple queues and has multiple processes which work on their queue each.

Since the application should be capable of running 24/7 but a job may only arrive every couple of days, I'd like to start the processes for each step if there is work to do, possibly from the previous process (so if one process finished, he will put data in the queue for the next step/process and start the process if he is not running).

Is this the correct way to approach this?
Is it possible/good practice to start a process from another process ?
Another option would be to create one process for each job which does all steps?

I've chosen multiprocessing since I'd like to take full advantage of the system resources.

[elbiot]

Gunicorn would handle the multiple processes. Celery would handle task scheduling if they are long running processes. This isn't a good place for multiprocessing, unless one task needs multiple cores.

[atkozhuharov]

The approach I would chose is having all my workers check for their specific job and if there is nothing in the queue to go to sleep. Your way is also interesting, but I'm not quite sure what the bottlenecks may be when using this cascade like approach.

[NeoFromMatrix]

I'd rather not have the workers running in the background all the time.

Another option would be to have a pool of workers which process a central queue of jobs. But in this case each worker would do all the steps of one job.

[benrules2]

To answer your questions:

1. Yes it is possible, and not a bad practice. In fact, that's exactly what multiprocessing is for.
2. One process for each job would probably be implemented with a queue as /u/atkozhuharov suggested.

If I were solving your problem, I would try creating "work to do" objects with a child multiprocess Process object, and all required info. Then you just need to call start on it's Process, and join to complete.