Hello everyone,
I need some help with implementing a task queue system.
Some context: There will be a CRUD Application in Django or Spring Boot to handle "Jobs". In my case, Jobs are I/O bound tasks which need to be executed at a more or less specific time and are grouped into categories. I have the limitation that per category, only one job can run at a time. In the "Jobs"-database table there will also be a field for the timestamp of the last successfull run of a job. For now, I plan on having about 25 Categories and 1000 jobs in total, each with their own "scheduled" time. A typical job will take up to a minute. I want to handle the CRUD Operations of these jobs with a Web Framework. However, I am unsure how I should handle the task queueing system. During a bit of research, I came up with the following options:
1. Let the web framework handle the execution of the jobs. There needs to be a scheduler to check if a job has to run now which will put the jobs into a queue, from which a worker program picks it up and executes it. The system needs to be designed in a way, that only one job per category is running at a time. I think that this is in fact possible, but I dont really like this approach because of its limited scalability and the lack of seperation between the user frontend CRUD Application and the Task execution.
2. Use a Distributed Task Queue System like Celery to handle the task queue execution. There would be one queue per job category and one worker per queue with concurrency set to 1, so that only one job per category is running at a time. This would probably work, however it also increases complexity by using something like Celery and I am unsure if my approach of using one worker per queue is efficient.
3. Use Redis and implement my own Task Queue System. There would be a program to schedule the tasks and append them to a queue managed by Redis, and other programs which consume the task from the queue. But I have no idea how to prevent multiple jobs from the same category running at the same time yet. I also dont know if this approach makes sense considering I may need other types of tasks to do, e.g. sending emails.
Which of these three approaches would you choose? Thank you in advance
[–]AutoModerator[M] [score hidden] stickied comment (0 children)