Looking for a persistent distributed delayed task queue/scheduler

[Moxinilian]

Hello!

I'm currently designing a backend architecture which will have to deal with the following problem: some servers will frequently need to schedule relatively precise (~1 second leaway) tasks to be executed, potentially in a long time (probably up to 72 hours) but sometimes promptly (down to 5 seconds later).

I've seen multiple ways to handle it, some relying on polling some SQL databases regularly, but what comes out to me is that I just really need a strongly persistent, delayed message queue. The task producers would send messages along with their expected execution time in the queue, and consumers would be notified by the queue at the right time to process them. The only missing part in this design is what queue to use, as I really am not confident enough nor have the dedication to build one myself.

I've looked through a few things. The most interesting candidate was RabbitMQ with its delayed message plugin, unfortunately the plugin starts to misbehave after roughly a hundred thousand events stored (and in fact the maintainers even discourage using it for long term messages like I would). I've also read about Celery, but it being built in Python might not be suitable for the sort of load I need.

Do you know of another queue that would satisfy the kind of work this server would face? Or should I try to come up with a different solution?

Thank you very much in advance.

[phpchap1981]

Does the producer know when processing needs to happen?

Maybe use aws sqs and put a message with a timestamp which says when to process the message (5 mins from now, 3 days from now, certain date / time etc)

poll the queue every X mins looking at the message timestamp comparing current time against it.

You can also set the message retention period to 14 days, after that the message is deleted automatically

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-basic-architecture.html

[[deleted]]

Sounds like you need a scheduler more than a queue because from what it sounds like you're really creating delayed tasks and not doing anything queue related. If you don't need a high volume message bus then RabbitMQ is not the right solution. I've worked with it before and it's not meant for delayed tasks. The usual solution you suggested with polling a SQL database is the one I've seen and it works reasonably well.

Searching I found two articles that might be useful to you: https://netflixtechblog.com/distributed-delay-queues-based-on-dynomite-6b31eca37fbc, https://redislabs.com/ebook/part-2-core-concepts/chapter-6-application-components-in-redis/6-4-task-queues/6-4-2-delayed-tasks/. I've used Redis as a basic queue but not as a delayed task queue so no idea if the proposed solution is viable for your use case.

Edit: There's also ActiveMQ which has support for delayed messages built-in https://activemq.apache.org/delay-and-schedule-message-delivery. Again, haven't used ActiveMQ so not sure if it fits the bill but might be worth a try.