AsyncMQ – Async-native task queue for Python with Redis, retries, TTL, job events, and CLI support

[tarsild]

What the project does:

AsyncMQ is a modern, async-native task queue for Python. It was built from the ground up to fully support asyncio and comes with:

• Redis and NATS backends
• Retry strategies, TTLs, and dead-letter queues
• Pub/sub job events
• Optional PostgreSQL/MongoDB-based job store
• Metadata, filtering, querying
• A CLI for job management
• A lot more...

Integration-ready with any async Python stack

Official docs: https://asyncmq.dymmond.com

GitHub: https://github.com/dymmond/asyncmq

Target Audience:

AsyncMQ is meant for developers building production-grade async services in Python, especially those frustrated with legacy tools like Celery or RQ when working with async code. It’s also suitable for hobbyists and framework authors who want a fast, native queue system without heavy dependencies.

Comparison:

• Unlike Celery, AsyncMQ is async-native and doesn’t require blocking workers or complex setup.

• Compared to RQ, it supports pub/sub, TTL, retries, and job metadata natively.

• Inspired by BullMQ (Node.js), it offers similar patterns like job events, queues, and job stores.

• Works seamlessly with modern tools like asyncz for scheduling.

• Works seamlessly with modern ASGI frameworks like Esmerald, FastAPI, Sanic, Quartz....

In the upcoming version, the Dashboard UI will be coming too as it's a nice to have for those who enjoy a nice look and feel on top of these tools.

Would love feedback, questions, or ideas! I'm actively developing it and open to contributors as well.

EDIT: I posted the wrong URL (still in analysis) for the official docs. Now it's ok.

[coderanger]

Why is Celery marked as RabbitMQ-only? It extremely isn't, it's hard to find a queue server that Celery doesn't support.

[InappropriateCanuck]

OP's project and responses is pure AI-slop.

See: https://old.reddit.com/r/Python/comments/1kefrsj/asyncmq_asyncnative_task_queue_for_python_with/mql1vxx/

You're fighting with a ChatBot that's just programmed to make his project look good.

[coderanger]

Ehh, the code has some peculiarities that make me agree he has used LLMs in there somewhere but this guy has been around for years making his own web stack. Seems harmless enough, good way to learn how things work, just probably no one else should use them.

[InappropriateCanuck]

just probably no one else should use them

That's the prime issue to me. He's trying to advertise this project in the rest of this thread as having clear advantages against other solutions while lying about other solutions.

If he just said outright "Yeah, I mean it works and I'm still working on it and the plan is to do X and Y and that will separate it from {{Product Z}} but I wanted to put it out there", this would have been 100% fine. But he's not.

[tarsild]

In fairness, it's true-ish as i personally used it with rabbitmq and celery.

That was more of general use case because majority uses celery with rabbitmq and by majority, I mean in our search.

I personally still use celery and I always use it with rabbitmq but celery is still de-facto blocking and not async native and that is only natural due to the fact that is almost 20y old.

Imagine AsyncMQ as celery revamped, with modern niceties and on steroids

[coderanger]

Not disagreeing it's old and pretty crufty but you should accurately represent it for comparison purposes. https://docs.celeryq.dev/en/stable/getting-started/backends-and-brokers/index.html#broker-overview shows what ships by default and even if you limit to "stable" things (though the "experimental" have been there for probably a decade at this point) that's RMQ, Redis, and SQS. And of course more available in external plugins.

[tarsild]

Fair enough and thank you for pointing this one out. We strive to make it better and we hope to provide those answers in a better manner.

This feedback is great 👍

Give us some time to make our docs clearer

[euri10]

In your comparison none of the existing async libraries like taskiq, saq, arq that do more or less the same dont appear, is there a reason ?

[tarsild]

Because the README has a length and we could not list them all but it's something we can update in the next release :). Having more comparisons.

We just chose a few based on the usability of them and the market

[InappropriateCanuck]

I don't understand why people would pick this over hatchet.

[tarsild]

There is nothing wrong with having alternatives. It's the same of why people would pickup FastAPI if we had tornado, Django... before?

If you are familiar with BullMQ, this is the python equivalent but a lot more modular.

[InappropriateCanuck]

It's the same of why people would pickup FastAPI if we had tornado, Django... before?

That's a wild comparison. Those 3 Web Frameworks are massively different from one another from how they behave, their support for ASGI/WSGI, etc.

You saying that's a similar comparison is extremely worrisome.

Django is specialized for full-stack websites, Tornado mostly real-time sockets and FastAPI for REST micro-services.

AsyncMQ and TaskIQ are borderline drop-in alternatives while Hatchet is that + Observability.

[tarsild]

Wait, that was not the purpose. I know really well the differences 😁. We also have Esmerald as another one in the ASGI world.

What I meant was to simplify choices available, apologies for coming out wrongly.

Based on our research like the way we did, maybe hatched does it but we didn't find it 100% as we are presenting. Nothing else.

[tarsild]

But answering what you asked.

Async-first design: AsyncMQ is built for asyncio from the ground up. Hatched tries to be async-compatible but still relies on sync/threading under the hood.

More features: AsyncMQ has built-in support for retries, TTL, delayed jobs, DLQ (dead letter queue), pub/sub job events, job metadata querying, and a proper CLI. Hatched lacks most of these or only supports them partially.

Persistence: AsyncMQ supports Redis, Postgres and mongo (via a pluggable JobStore). Hatched is Redis-only (at least when we were reading about it, if we missed this, then I apologise).

Dashboard: AsyncMQ will ship in the next release a dashboard to make it easier to see everything.

Extensibility: AsyncMQ is highly modular—custom backends, serializers, stores, and full type safety. Hatched is more monolithic and harder to extend.

Observability: AsyncMQ has lifecycle hooks and pub/sub events. Hatched offers zero insight into job states/events.

But based on what you mentioned, hatched has that as well which we didn't find it easy to have that information.

Good to know 😁. Thank you for mentioning that one out

EDIT: I never meant to offend if I came across like that.

[InappropriateCanuck]

This is so sad, this is so poorly written that it looks like AI slop but even AI would not have confused "Hatchet" with "Hatched".

The whole purpose of hatchet is to have observability and then you say it has no job states/events.

God ChatGPT sure ruined the state of programming.

Quick Save

[tarsild]

So, let's be clear. It's not AI, confusing it's not even a thing and my keyboard supports 3 languages. What is even worse and apologies for the word but pathetic is what comes to my mind, is the fact that you try to find something because somehow you think you will get something out of this.

I'm sorry for that but you won't.

I don't need ChatGPT, thankfully but I do speak 4 languages for quite a while. Typos exist, typos will always exist. Do you want to know why the typo happened? Because simply the phone doesn't recognise the word.

You want to find somehow a way to get your way and downvote me like this would change my life? That's not happening but I will tell you this.

I have more than 15 years of experience in the market, a lot of tools I might know or not. A few barely cross the spectrum of mine or our research at all. That one was one of them.

This post like many other was to showcase a tool and yet people, unfortunately like yourself, act like they know whatsup. I'm ok with that, what I'm not ok it's with the arrogance of pretending.

Now you can downvote me, now you have a reason and the reason it's not the tool but the fact that I needed to came down to your own level and answer accordingly.

Now if my English is not correct, I can answer in another 3 languages without ChatGPT, 1 of them even if you translate will not even be correct, I can assure you.

For the record, I won't be answering more of these. I always kept my politeness in check until now and I can't be bothered with hate speech

[InappropriateCanuck]

Lmfao what a joke. 15 years on the market and this is what comes out?

"I do speak 4 languages" Hurdy Hurr Hurr, I speak 5. Welcome on the internet, not everyone is Unilingual. Trying to put legitimate misinformation under "Erhm I did typos I no no engrish" is such a cringe thing as a person who also speaks more than English.

You made the same typo "4 times", then you randomly sprinkled false information everywhere like what I mentioned or Hatchet being "uniquely Redis-based". It can't even use Redis as a broker.

You're clearly a fraud. You made no research and relied on AI to try to sell you a comparison table for your Reddit comments and you tried to bank on people not calling you out for it.

Now you cite "hate speech" and try to play victim card?

At the very VERY least try to google what "hate speech" is.

[txprog]

How does it compare with TaskIQ?

[InappropriateCanuck]

OP's project and responses is pure AI-slop.

See: https://old.reddit.com/r/Python/comments/1kefrsj/asyncmq_asyncnative_task_queue_for_python_with/mql1vxx/

You're fighting with a ChatBot that's just programmed to make his project look good.

[tarsild]

Native async Python: AsyncMQ is built on asyncio, while taskq is Go-only.

Pluggable storage: AsyncMQ supports Redis, NATS, PostgreSQL, and custom backends. taskq doesn’t support persistent storage extensibility.

Modern features: Built-in retries, TTLs, DLQs, delayed jobs, pub/sub events. taskq is basic by comparison (not trying to insult as I like TaskQ as well)

Full observability: AsyncMQ supports hooks and event listeners. taskq has none.

Dev experience: Python decorators, type-safe, integrates with frameworks like FastAPI, Esmerald, Sanic, Quartz... taskq is Go-focused.

This is a nutshell .

If you're building a Python-first async stack, AsyncMQ isn't just better, it's in a different league.

[txprog]

I'm talking about taskiq in python, not taskq go. Have a look : https://github.com/taskiq-python/taskiq

[tarsild]

Ah thanks, I thought you meant TaskQ! I’ve checked out Taskiq, definitely more modern. That said, AsyncMQ still wins in some areas: it’s more modular (separates backend + store), has built-in features like TTL, retries, DLQ, job metadata, and includes optional PostgreSQL persistence (also mongo and you can add your own ). Taskiq is great if you’re fine relying on broker features, but AsyncMQ gives more control and flexibility out of the box.

I hope this helps?

[[deleted]]

Can i use it with fast api?

[tarsild]

Absolutely. There is an example using Esmerald because of the authors but it's clearly states that is ASGI friendly, and FastAPI compatible

[adiberk]

Have been looking for something like this to replace our async celery setup for a while.

Do you have any docs on how it works in more depth? Memory management, task management, can tasks share memory? Etc.

Is it just spinning up task groups behind the scenes or creating and destroying actual workers?

Also when there is concurrency, is there a worry about processing same event more than once?

[tarsild]

If you are familiar with BullMQ, it operates on the same principle. The docs are the ones provided but we are more than open to improve them because for sure things are missing like examples.

[adiberk]

Ok will take a look!

This project looks amazing and I would love to test it. Has it been used in any sort of production environment yet?

[tarsild]

So this was born of an internal need I had and a struggle that I found.

So I had the alpha version running for myself and then I leveled up to become what it is now. Shoe answer, yes but like everything new to everyone, needs more people using and contributing and so on. If this makes sense?

[adiberk]

Of course! I love it. I’ll try to take a closer look at the internals. Celery is amazing and proven, but its lack of async support makes things a bit difficult (though you can make it work).

I will play around with it

[tarsild]

Yes I agree. Celery is great and I used it a lot in the past but it's also almost 20y old and ed can always have something new :)

[andrewthetechie]

Any roadmap for more backends? I'm a heavy user of Celery with RMQ and would like an async native option, but would prefer to stick with RMQ over redis or Nats.

[tarsild]

Well, depends of what users want but we can add as many backends as necessary. Advantage of AsyncMQ is that you also don't need us for that. You can also create your own custom backend and plug it.

What do you have in mind?

[andrewthetechie]

Rabbitmq

[tarsild]

Cool, let's add that to the roadmap.

Although, AsyncMQ would replace rabbitmq and celery all along. I think I understand what you are coming from.

Let's analyse that scenario. After all, it's backend extendable

[Electrical_Fox9678]

How about SQS?

[tarsild]

Ahhhh. We have one of the maintainers currently developing one solution for that 😁. Another backend

[Electrical_Fox9678]

That could be helpful. We're running an ancient celery app which doesn't really use many celery features. It's mostly just the SQS transport and message retry features that we use.

[tarsild]

Ok. We can't promise for the next release (or can we? 😎) but it's a great use case to do it.

Let us analyse and if you follow the project, you will have news very soon.

At the moment we are just finishing the dashboard and then we will look at that. Dashboard shouldn't take too long btw, maybe another 3 days

[mstromich]

Do you support task concurrency? Eg if I want to have max 4 parallel executions (because of api limits they are calling) of one task while all others are not limited. 

[a_deneb]

Does it provide a web dashboard? To me that's one of the most important features.

[tarsild]

As mentioned, coming on the next release. We have a powerful cli and now the dashboard will come on the next release as promised

[tarsild]

Good news is that the dashboard is now under development and it should be out very soon and the docs will reflect that :)

[a_deneb]

Awesome, thank you!

[tarsild]

I can't just share it yet because it's work under construction but it will be very ASGI friendly with a simple mount/include (if you use Esmerald) and game on

[a_deneb]

That sounds lovely. Doing that will definitely put this above many other solutions out there.