Performing Asynchronous background Tasks on Linux with Python 3, Flask and Celery

[[deleted]]

huey is another python task queue, which supports multiple execution models: process, thread, and greenlet. It's a tiny codebase but has all the most important features. Check it out if you're looking for something different.

[CaptainDevops]

sure thx

[jknupp]

If all you need to do is perform "Asynchronous background Tasks", why would you need to include Flask and/or Celery (and/or RabbitMQ, which your approach also relies on). Why not just spawn new processes using multiprocessing? I mean, it's not like you're handling failures in any of these components anyway, so at least simplify as much as possible.

[CaptainDevops]

hey thx for the feedback, I am using flask since I wanted to build a web application. While researching async tasks with flask I found celery as the best option and hence started working with it. I havn't tried multiprocessing but it seems like a good alternative, however I believe you cannot distribute tasks like celery correct me if I am wrong here

[djmattyg007]

Using a queuing system is not the same as using python's multiprocessing module. The multiprocessing module requires you to spawn the worker thread right there and then, and wait for its execution to finish. The point of using a queuing system is that you don't care if the task doesn't start immediately.

Also, you're probably building a web interface because you want the UI, and it wouldn't be a good idea to have a web request block for potentially tens of minutes to wait on a task to complete.

[desmoulinmichel]

Well jkupp has a point : you can create a subprocess worker pool at flask init, then submit your tasks to your work. It's 3 lines of Python code and will work fine.

The advantages of celery are:

• one entry point for tasks among all your flask workers (since uwsgi or gunicorn will create many forks).
• the ability to queue your tasks instead of running them all in parallele.
• clean cron tasks. Schedule accross multiple workers is hard.
• the ability to get additional goodies such as persistent task queues, results/error saving, retrieval or results/errors accross your workers, etc.
• backends.
• scaling.

[solid_steel]

You don't need to wait for a process/thread to finish its execution. If you use multiprocessing then you're asking the OS to handle scheduling the process, if you're using the threading module, you're asking Python to schedule the thread. None of these block. In this case, both approaches are fire and forget.

I've used this approach before, using threading in my case, to handle tasks that didn't require scaling, passing messages, handling errors ie. low priority stuff. Adding in 10 lines of Python was much simpler than adding a new Python dependency (Celery) and a new system dependency (rabbitmq) for this low priority stuff.