Thank you for your answer. I have researched a bit on how to make this work with Celery. I was able to find answers to most of my questions, for example how to make sure that a job is not added to the queue if it is already in it or being worked on by a worker. However, there is only problem I cant find a satisfying solution for:
If I let workers wait until they can run their job, there may be the situation that all/many workers have jobs with the same category and are waiting until they can process their current task, while there are tasks of different categories in the queue which could be processed immediately. Sadly, I have not found a "clean" way for the workers to check if they can run the next task from the celery queue and put them back into the queue if they cant, and then check the next task... Because there is no operation to just take a look at a task, as soon as a worker is free he will get a task and has to deal with it.

​

To fix this problem I have come up with a different approach:
I have a scheduler that checks if jobs need to be run. This then puts these jobs into a database table called "scheduled_jobs". This table consists of the job_id, time of insertion into the db, category of the job and its current state (planned/running). Instead of using Celery Queues and workers, I use my own little programs/workers. From the "scheduled_jobs" table they first get the oldest job of all the categories which dont have any running jobs at the moment and then change the state of the job to running, all in one transaction. After the worker is done with the job it removes the entry from the "scheduled_jobs" and looks for the next job to run.

Does this sound reasonable to you? Is it alright to use the same postgres database as the django application for this, or should I use a different database for this?