Mark Mark - 2 months ago 6
Python Question

How do I run a long-running job in the background in Python

I have a web-service that runs long-running jobs (in the order of several hours). I am developing this using Flask, Gunicorn, and nginx.

What I am thinking of doing is to have the route which takes a long time to complete, call a function that creates a thread. The function will then return a guid back to the route, and the route will return a url (using the guid) that the user can use to check progress. I am making the thread a daemon (thread.daemon = True) so that the thread exits if my calling code exits (unexpectedly).

Is this the correct approach to use? It works, but that doesn't mean that it is correct.

my_thread = threading.Thread(target=self._run_audit, args=())
my_thread.daemon = True


The more regular approch to handle such issue is extract the action from the base application and call it outside, using a task manager system like Celery.

Using this tutorial you can create your task and trigger it from your web application.

from flask import Flask

app = Flask(__name__)
celery = make_celery(app)

def add_together(a, b):
    return a + b

Then you can run:

>>> result = add_together.delay(23, 42)
>>> result.wait()

Just remember you need to run worker separately:

celery -A your_application worker