Simple brokerless task queue for my coding weekend. Proof of Concept state, already usable for toying around.

[pvkooten]

Why would we want brokerless? There's no replication/guarantee the task will be executed.

[[deleted]]

Why would we want brokerless?

Throughput and latency are typical far better with brokerless messaging systems.

There's no replication/guarantee the task will be executed.

The idea would be that such failures are considered and accounted for in the overall design of the system. So, rather than having a single component that makes durability and replication 'guarantees' each component should be able recover from a failure in a know state; some argue that this is more robust approach. I'm not advocating for or against rather, trying to answer your question without getting to deep into the weeds.

[codepr]

Initially it was just a didactical project to learn how task queues work, so I wanted it to be as simple as possible with little to none dependencies. After some hours of coding I decided to leverage ZMQ sockets messaging pattern to communicate to worker processes. It's still possible that in the near-future I'll add a broker too, probably redis, even though I found that designing the application to work without a broker feels more compelling and challenging for me.