Jamie Forrest Jamie Forrest - 22 days ago 33
Python Question

How to dynamically add / remove periodic tasks to Celery (celerybeat)

If I have a function defined as follows:

def add(x,y):
return x+y


Is there a way to dynamically add this function as a celery PeriodicTask and kick it off at runtime? I'd like to be able to do something like (pseudocode):

some_unique_task_id = celery.beat.schedule_task(add, run_every=crontab(minute="*/30"))
celery.beat.start(some_unique_task_id)


I would also want to stop or remove that task dynamically with something like (pseudocode):

celery.beat.remove_task(some_unique_task_id)


or

celery.beat.stop(some_unique_task_id)


FYI I am not using djcelery, which lets you manage periodic tasks via the django admin.

Answer

No, I'm sorry, this is not possible with the regular celerybeat.

But it's easily extensible to do what you want, e.g. the django-celery scheduler is just a subclass reading and writing the schedule to the database (with some optimizations on top).

Also you can use the django-celery scheduler even for non-Django projects.

Something like this:

  • Install django + django-celery:

    $ pip install -U django django-celery

  • Add the following settings to your celeryconfig:

    DATABASES = {
        'default': {
            'NAME': 'celerybeat.db',
            'ENGINE': 'django.db.backends.sqlite3',
        },
    }
    INSTALLED_APPS = ('djcelery', )
    
  • Create the database tables:

    $ PYTHONPATH=. django-admin.py syncdb --settings=celeryconfig
    
  • Start celerybeat with the database scheduler:

    $ PYTHONPATH=. django-admin.py celerybeat --settings=celeryconfig \
        -S djcelery.schedulers.DatabaseScheduler
    

Also there's the djcelerymon command which can be used for non-Django projects to start celerycam and a Django Admin webserver in the same process, you can use that to also edit your periodic tasks in a nice web interface:

   $ djcelerymon

(Note for some reason djcelerymon can't be stopped using Ctrl+C, you have to use Ctrl+Z + kill %1)