Async Tasks in Django with Celery

Why slow work goes to a worker

A web request must return fast. The user waits for the response, so the view must not run slow work.

Slow work includes email, image resize and API calls. Send this work to a worker. The view returns at once and the worker does the job in the background.

Celery is the task queue. Redis is the broker. The broker holds the task messages until a worker takes them.

Install Celery and Redis

Install the Celery package and the Redis client in your project environment. Start a Redis server on the default port.

Create the Celery app file next to your settings file. The app file sets the Django settings module and finds tasks in each app.

Add the broker and result backend settings. The broker URL points to Redis. The result backend stores task results.

pip install celery redis
redis-server

# proj/celery.py
import os
from celery import Celery

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "proj.settings")
app = Celery("proj")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()

# proj/settings.py
CELERY_BROKER_URL = "redis://localhost:6379/0"
CELERY_RESULT_BACKEND = "redis://localhost:6379/1"

Write a task and call it from a view

A shared task is a plain function with the shared_task decorator. The decorator connects the function to the Celery app.

The view calls the task with .delay(). This call sends a message to the broker and returns at once. The view does not wait for the task.

Start the worker in a second terminal. The worker reads the broker and runs the task. The task runs only when a worker is active.

# app/tasks.py
from celery import shared_task

@shared_task
def send_welcome_email(user_id):
    user = User.objects.get(pk=user_id)
    send_mail("Welcome", "Hello", "a@b.com", [user.email])

# app/views.py
def signup(request):
    user = User.objects.create(username=request.POST["name"])
    send_welcome_email.delay(user.id)
    return HttpResponse("ok")

# terminal
celery -A proj worker -l info

Retry a failed task

A task can fail for a short time, such as a network error. Use autoretry_for to retry the task on a set of exceptions.

Set retry_backoff to True. The wait time grows after each failure. Set max_retries to stop the retries.

The code below retries a request error three times. The worker waits longer between each try.

from celery import shared_task
import requests

@shared_task(autoretry_for=(requests.RequestException,),
             retry_backoff=True, max_retries=3)
def fetch_price(url):
    response = requests.get(url, timeout=10)
    return response.json()["price"]

Schedule a task with beat

Beat sends scheduled tasks to the broker. Add a beat schedule to the settings file. The schedule uses a crontab or a number of seconds.

Run the beat process next to the worker. Beat sends the message and the worker runs the task.

The code below runs a cleanup task every day at 3 in the morning. Start beat with the command in the block.

# proj/settings.py
from celery.schedules import crontab
CELERY_BEAT_SCHEDULE = {
    "cleanup": {
        "task": "app.tasks.cleanup",
        "schedule": crontab(hour=3, minute=0),
    },
}

# terminal
celery -A proj beat -l info

Test with task_always_eager

A test must not need a worker. Set task_always_eager to True in the test settings. The task then runs at once in the same process.

The .delay() call returns a finished result. The test can check the result and the side effects.

Turn this setting off in production. Eager mode hides broker and worker problems.

# proj/settings_test.py
CELERY_TASK_ALWAYS_EAGER = True
CELERY_TASK_EAGER_PROPAGATES = True

Common mistakes and fixes

Mistake one: you call a task and no worker runs. The message waits in the broker and nothing happens. Fix: start the worker and check the broker URL.

Mistake two: you pass a model instance to the task. The instance is not safe to send through the broker. Fix: pass the primary key and load the object in the task.

Mistake three: a long task has no timeout. One stuck task can block the worker. Fix: set a soft time limit on the task.

Common questions

Why is my Celery task not running?

No worker is active, or the worker uses a different broker URL. Start the worker and check the broker setting.

What is the difference between delay and apply_async?

delay is a short form of apply_async with no options. apply_async lets you set options such as countdown, eta and queue.

Do I need a result backend?

You need a result backend only if you read task results or task states. Fire-and-forget tasks do not need one.