I have a web-service that runs long-running jobs (in the order of several hours). I am developing this using Flask, Gunicorn, and nginx.
What I am thinking of doing is to have the route which takes a long time to complete, call a function that creates a thread. The function will then return a guid back to the route, and the route will return a url (using the guid) that the user can use to check progress. I am making the thread a daemon (thread.daemon = True) so that the thread exits if my calling code exits (unexpectedly).
Is this the correct approach to use? It works, but that doesn't mean that it is correct.
my_thread = threading.Thread(target=self._run_audit, args=())
my_thread.daemon = True
The more regular approch to handle such issue is extract the action from the base application and call it outside, using a task manager system like Celery.
Using this tutorial you can create your task and trigger it from your web application.
from flask import Flask app = Flask(__name__) app.config.update( CELERY_BROKER_URL='redis://localhost:6379', CELERY_RESULT_BACKEND='redis://localhost:6379' ) celery = make_celery(app) @celery.task() def add_together(a, b): return a + b
Then you can run:
>>> result = add_together.delay(23, 42) >>> result.wait() 65
Just remember you need to run worker separately:
celery -A your_application worker