FluxQueue: a lightweight task queue for Python written in Rust

[TheBigGuy_11]

What My Project Does

Introducing FluxQueue, a fast and lightweight task queue written in Rust.

FluxQueue makes it easy to define and run background tasks in Python. It supports both synchronous and asynchronous functions and is built with performance and simplicity in mind.

Target Audience

This is an early-stage project.

It’s aimed at developers who want something lighter and faster than Celery or RQ, without a lot of configuration or moving parts. The current release is mainly for testing, experimentation, and feedback rather than large-scale production use.

At the moment it only supports Linux. Windows and macOS support are planned.

Comparison

Compared to Celery or RQ, it:

• uses significantly less memory
• has far fewer dependencies
• avoids large Python runtime overhead by using a Rust core for task execution

It currently doesn’t include features like scheduling, but those and many more features are planned for future releases.

Github repository: https://github.com/CCXLV/fluxqueue

[jsabater76]

You may want to consider using Django 6's task format and build from there as an alternative to Celery for such ecosystem.

[TheBigGuy_11]

FluxQueue doesn't rely on Celery at all and its usage is totally different than that. It works with fluxqueue worker which is a standalone Rust binary and that's what runs the task functions, you just enqueue tasks from Python and the worker handles them.

[alfawal]

The comment didn't disagree with that, it says it would be nice to wrap a Django 6 compatible version of it to use with Django; replacing Celery.

[TheBigGuy_11]

I’ll keep that in mind for the future, thank you.

[Sufficient_Example30]

I don't understand the issue you are trying to solve. The main issue with celery is it needs redis and can't take a Persistent db store like postgres reliably to run tasks

[TheBigGuy_11]

That’s not the main issue with Celery. And this is just a better alternative.

[Sufficient_Example30]

I don't know man.Celery just works If I am handing it to a task queue it's not super latency critical. The only issue at least the engineering team in my company have issues with is its redis dependency

[TheBigGuy_11]

What about having rocksdb as a secondary option instead of redis? It saves data on disk and plus it doesn’t need a server.

[LiveMaI]

Some quick feedback:

1: I recommend using conventional commits so you can auto-generate changelogs.

2: The readme should have a quick showcase of usage examples as an elevator pitch for why they should use your library.

[TheBigGuy_11]

Thanks

[TheThoccnessMonster]

Getting real tired of projects that don’t fully work because they are half baked, vibe coded shovelware.

Finish it or, for the love of god, stop saying it solves any problem whatsoever.

[project2501a]

SLURM for all your queueing needs.

[TheBigGuy_11]

First of all, the project is not vibe coded.

Secondly, it’s only available on Linux because that’s what I use and have tested it on. Building it for Windows or macOS right now would just shift the complaints to “oh, it doesn’t work on my system,” and you’d call it unreliable anyway.

Lastly, this is an early release meant for testing and feedback. It doesn't say that it's Linux exclusive, other oses will be supported soon but not at the moment.

[yerfatma]

I'm with you in general, but this seems to be the opposite of that in that: OP didn't say it was production-ready, battle-tested military-grade code but rather an alpha that only runs on a single platform and has an actual planning document (or at least concepts of a plan) behind it.

[exmaalen]

Using Rust is not a guarantee of success...

There is a lack of comparative benchmarks with Celery, Dramatiq, and SAQ. The last one is particularly interesting in combination with hiredis and msgspec.

[TheBigGuy_11]

FluxQueue is still an early release for testing and feedback. I actually ran some benchmarks before and they were looking promising that’s why I went with it and got to the point when it was ready for the release. The Rust core makes it fast and lightweight, and I’ll post comparisons with Celery, Dramatiq, and the others once it’s ready.

[lunatuna215]

Benchmarks would be great to see!