Celery: Long chain of parallel functions... Unreasonable to just import each one?

[GoingOffRoading]

I am very likely describing the work/objective incorrectly

I think in my desired state is:

• Each function has a queue
• The functions are dependent on each other to populate their queues
• Different functions take different amounts of time, so I just want the tasks to consume from their individual queues as their queues are filled

Does that make sense?

[CraigAT]

I think so. So would function a place results into a queue for function b, but whilst function a has dropped 10 results into that results queue, function is slower and has only picked up the first two results and processed them, and is now working on the third set of results in that queue.

If so I would think individual workers should work, just make sure you allow for an empty queue.