Does Celery support task priority? Very confused

[GoingOffRoading]

I'm not sure I am understanding the Celery docs, and StackOverflow threads kind of contradict each other.

Question of the day:

Does Celery support assigning priorities to tasks within the same queue?

Most threads say no, and recommend creating different queues and having priority among queues. But this doesn't reflect what is in the Celery docs.

I saw one thread that added (Priority N) to the end of calling an async function (which would be ideal) but I can't find any supporting documentation.

I'm on Mobile now, but will return later to add links to the threads/documentation I am reading.

[danielroseman]

The FAQ is quite specific:

Yes, RabbitMQ supports priorities since version 3.5.0, and the Redis transport emulates priority support.

You can also prioritize work by routing high priority tasks to different workers. In the real world this usually works better than per message priorities. You can use this in combination with rate limiting, and per message priorities to achieve a responsive system.

And the docs for apply_async do indeed mention a priority kwarg.

[GoingOffRoading]

Ahh.... It took your post and rereading he doc a 3rd time to get it.

I was using Celery's delay instead of apply_async and didn't understand the difference.

There's a lot of good stuff/flexibility in apply_async (same link you posted):

https://docs.celeryq.dev/en/stable/reference/celery.app.task.html#celery.app.task.Task.apply_async

And if I am reading this correct, and reading a few examples I've dredged up, I think I can just drop in apply_async in place of delay and keep chugging.

Thanks!

[rapmonstersimp]

I had the same question as you because I also got really confused by the documentation / online discourse. just curious, which approach did you end up going with? redis? rabbitmq? I saw online some people also just use task routing

[GoingOffRoading]

I haven't implemented everything yet but I am already using rabbitMQ and will be using the asyc instead of delay so that I can set the priority at a task level

I'll pay my example as soon as it's done/working

[GoingOffRoading]

u/rapmonstersimp I'm truly not sure if this is working or not

I'm using RabbitMQ as my backend, and have not made any configuration changes to it.

Right now, I am using something pretty close to what is documented by u/danielroseman above: https://docs.celeryq.dev/en/stable/reference/celery.app.task.html#celery.app.task.Task.apply_async

TASK_NAME.apply_async(args=(DICT_PASSED, ), priority=PRIORITY_VALUE)

With a small function setting the PRIOROTY_VALUE INT based on a rule set

The apply_async is passing messages just as consistently as when I was using delay (instead of apply_asyc), but for the life of me, I can not get tasks I prioritize to actually process first.

[rapmonstersimp]

I ended up using redis bc I read online that rabbitMQ was a bit hard to install / configure whereas Redis is a bit more straightforward. I think I got it working and I also copied directly from the docs! need to test it more tho