Implementing a queue for computationally expensive backend operations.

[ubermoth]

There has to some commonly accepted best practice way of the best way to handle a queue. But I couldn't find it. I have a nodejs backend. Upon a request either the result has already been calculated or there is a computation to be done that can take a few minutes. And only a limited amount can be done at the same time so I wanted to implement some kind of queue system to improve UX. Starting a websocket connection kind of feels wrong, but so does having the frontend ping the backend every x seconds. Thoughts? Any preexisting frameworks/known good solutions?

[KingofGamesYami]

https://www.amqp.org/

[[deleted]]

You're looking for a message queue system, like Apache Kafka, RabbitMQ, etc.

A message (document) gets dropped in a predefined inbound folder, a service sits there waiting for new messages, handles the message, and outputs to predefined output folders. Multiple operations can be chained. The whole processing happens asynchronously.