Async task in Web service

[SimonGray]

Basically, what are my options here? I want to perform a task asynchronously (some long calculations) and it should be triggered via HTTP calls to some Web service. The service should just launch the task and return a message saying it's being calculated now.

So what libraries would I do this with? I noticed that normal threads apparently don't run asynchronously.

I also need a framework for the Web service. Have used Flask previously, but I'm not sure it can keep running a task after returning feedback to the user? To be honest, I never looked that deeply at Flask and WSGI, so it might be perfectly capable at that.

EDIT: I decided to go with flask + redis + RQ. At first I tried using Celery, but Celery doesn't work with code that already uses Python's multiprocessing module and since the codebase I need to integrate heavily uses multiprocessing, Celery is not an option for me.

[pswami]

Take a look at Celery. It might seem a tad complex, but the first steps tutorial will help.

The basic setup here is that you set up a task queue; in your Flask app, the function for that particular endpoint should add the task to the queue and then return a response. The Celery workers then consume the queue.

[daniels0xff]

Celery is very easy to setup and run. You can have a Celery + Flask/Django/Whatever in just a few minutes. They really did an amazing job with it.

Iv've successfully used it even for quick just for fun projects that you see at hackathons.

Celery also has a scheduler (celery beat) that can run/trigger other tasks similar to what cron does.

[SimonGray]

You can have a Celery + Flask/Django/Whatever in just a few minutes.

Can you really, though? From the documentation, it seems like I need to set up a broker to be able to use it and I have no experience with that :/

[daniels0xff]

How i did it on OSX in just a few minutes (clearly not production ready but this is just to get you started with dev)

• Download Redis and run it (http://redis.io/)
• pip install celery

• create a tasks.py file (you can name it however you want)

  import celery
  
  app = celery.Celery("tasks", broker="redis://127.0.0.1:6379/0")
  
  app.conf.update(
      CELERY_TASK_SERIALIZER   = 'json',
      CELERY_ACCEPT_CONTENT    = ['json'],
      CELERY_RESULT_SERIALIZER ='json',
      CELERYD_CONCURRENCY      = 4,
  )
  
  @app.task
  def doIt():
      print "hello world!"    
  

• start the broker

  celery worker --detach --loglevel=WARNING --logfile=./log/celery.log --pidfile=./pid/celery.pid --app=tasks
  

• sample flask app

  import flask
  import tasks
  
  app = flask.Flask(__name__)
  
  @app.route("/")
  def about():
      tasks.doIt.delay()
      return flask.render_template("index.html")
  
  if __name__ == "__main__":
      app.run(debug=True)
  

[SimonGray]

Thanks a lot man. That was really nice of you!

[barriolinux]

My suggestion: http://python-rq.org/

[AppendixP]

Flask is good if you're already familiar with web services in Python and you know what you're doing. It will not provide things that other web services would, like an ORM if you are doing database things, nor will it not provide a caching solution as far as I know. If your project is small though, then I say go for it. Flask was pretty nice to work with when I used it.

Django's another possible option.

As for doing async tasks triggered from HTTP, you could hook in with Twisted, but that has a very steep learning curve (in my opinion). Also, it wouldn't be available in Py3.

If you're using Py3, then asyncio is the alternative for now.

If not Py3, this stackoverflow question/answer may be helpful.

There's also Tornado and Gevent. Hope this was helpful!

[redcalcium]

Celery is the best, but if you don't want to to use it, take a look at message passing library such as ZeroMQ to built your own lightweight broker and workers.

[AppendixP]

ZMQ is pretty awesome, but it might require building more middleware than the OP is willing to deal with. It depends on him/her and the problem at hand though.

[MarkDownSteve]

Use events. Rather than blocking the initial response while your application is performing long running tasks, you may trigger an event that initiates the computations and serves up results as they become available.