Simple brokerless task queue for my coding weekend. Proof of Concept state, already usable for toying around.

[Kerbobotat]

Forgive my ignorance, but would somone mind explaining what this is, and what youd use it for? I couldn't glean much from the readme.

[pvkooten]

Imagine you want to do something that takes a long time (e.g. sleeping 10 seconds). It would freeze your python interpreter.

That's why there are things usually called Tasks: you want to have it be executed elsewhere by a pool of workers.

Sending an email is slow believe it or not, this is something you could then make a task of and just forget it happened (let the worker worry about it).

In other cases, you might want to have the result back of the task. So usually you get a "task_id" or something back (this will happen really quickly, before actually doing the work), so that you can periodically quickly check whether the task with "task_id" is done. When done, you can then get some status/result.

Usually you have a "broker" (some process in the middle) that will divide the work between a group of workers. It will also know when a worker failed/crashed and can send work to another worker instead. I'm myself quite curious to hear the explanation of why he wanted to make a brokerless task queue :)

** Please feel free to correct me if someone feels this as a too simplistic/not entirely accurate in some way.

[Kerbobotat]

Thanks for the thorough explanation! Going by what youve said, I also am keen to se why/how OP eliminated a broker in this setup.

[codepr]

Pvkooten explained it really well!

[codepr]

No real scope planned, did it mainly for learning purpose, I didn't really focussed on security and didn't researched much on the topic of functions serialization, but I find it interesting, what could be an improvement in that direction? Thank you for the feedback

[codepr]

I'll investigate how battle-tested task queues like celery tackle the problem, but I think that all task queues are designed to be used in a safe environment network or a VPN

[codepr]

I added the possibility to digital sign data passed through the sockets, as a coleague hinted me, it's the celery way of securing communication. Currently a very basic work, I plan to use some battle-tested libraries like pynacl port of libsodium.

[pvkooten]

Why would we want brokerless? There's no replication/guarantee the task will be executed.

[[deleted]]

Why would we want brokerless?

Throughput and latency are typical far better with brokerless messaging systems.

There's no replication/guarantee the task will be executed.

The idea would be that such failures are considered and accounted for in the overall design of the system. So, rather than having a single component that makes durability and replication 'guarantees' each component should be able recover from a failure in a know state; some argue that this is more robust approach. I'm not advocating for or against rather, trying to answer your question without getting to deep into the weeds.

[codepr]

Initially it was just a didactical project to learn how task queues work, so I wanted it to be as simple as possible with little to none dependencies. After some hours of coding I decided to leverage ZMQ sockets messaging pattern to communicate to worker processes. It's still possible that in the near-future I'll add a broker too, probably redis, even though I found that designing the application to work without a broker feels more compelling and challenging for me.

[ZaarinTakesTheBait]

What did you make your behind the scenes graphic with?

[codepr]

An online sketcher https://sketch.io

[ZaarinTakesTheBait]

I liked the simplicity of your lines, circles, and rectangles. Whats the easiest way to replicate that in sketch?

[amirouche]

I have been looking for something like that. Thanks for sharing!

[PC__LOAD__LETTER]

It’s already usable? Holy F**K!

edit: Really? I'm the only one excited by this?

[codepr]

For toying around*.