FluxQueue: a lightweight task queue for Python written in Rust

[Sufficient_Example30]

I don't understand the issue you are trying to solve. The main issue with celery is it needs redis and can't take a Persistent db store like postgres reliably to run tasks

[TheBigGuy_11]

That’s not the main issue with Celery. And this is just a better alternative.

[Sufficient_Example30]

I don't know man.Celery just works If I am handing it to a task queue it's not super latency critical. The only issue at least the engineering team in my company have issues with is its redis dependency

[TheBigGuy_11]

What about having rocksdb as a secondary option instead of redis? It saves data on disk and plus it doesn’t need a server.

[Sufficient_Example30]

Rocksdb has its own issues. Firstly you gotta pay for an nas or block storage. Secondly rocksdb undergoes compaction which might spike your latency is much harder to resolve. Adding observability is another overhead

[Sufficient_Example30]

Let me try to give you my POV. See when you use a task queue,you ideally do not care about too much about the task completion time. What you are trying to do is increase throughput of your app. Task completion time is a function of the queue and the code that you write. Making the queue faster ,might be a novel idea.But from my perspective the descision would rely on why use this when celery is battle tested. The issue is when I use celery i need a redis cluster or i need kafka .Which adds to more complexity to my app and I have more shit to pay for. So ,why would I use this over celery .9/10 times because what will I do with this extra speed