The first time we deployed Celery to production on a client project, we thought we had done everything right. We had workers running, tasks queuing, and Redis as the broker. Six weeks later, the task queue was backed up with 40,000 unprocessed jobs, the workers had silently died, nobody knew, and a batch of client invoices had not been generated for two weeks.
That was four years ago. Since then we have deployed Celery on dozens of projects and we have learned what actually goes wrong — not in development, where everything works, but in production, where things fail in ways you do not anticipate.
This post covers the configuration and operational patterns we now use on every Celery deployment.
Why tasks fail silently (and how to stop it)
The most dangerous thing about Celery is how quietly it can fail. A worker process dies, the task queue fills up, and your application keeps accepting work and sending it to a queue that nobody is processing. No exception is raised. No alert fires. Users notice eventually, or you notice when a daily report does not arrive.
The fix has two parts: monitoring and task acknowledgement configuration.
Task acknowledgement
By default, Celery acknowledges a task (removes it from the queue) as soon as a worker picks it up, before the task runs. If the worker dies mid-task, the task is lost.
# celery.py
app = Celery('myproject')
app.conf.update(
# Only acknowledge after the task completes successfully
task_acks_late=True,
# If a worker dies, reject the task back to the queue
task_reject_on_worker_lost=True,
# Limit memory — workers that leak memory will restart cleanly
worker_max_memory_per_child=200_000, # 200MB in KB
# Limit tasks per child process to prevent long-running workers
# from accumulating state
worker_max_tasks_per_child=1000,
)
With task_acks_late=True, a task that is picked up by a dying worker will be requeued and picked up by another worker. The task might run twice (more on that shortly), but it will not be silently dropped.
Monitoring with Flower
Flower is a real-time web UI for Celery. We deploy it on every project:
pip install flower
celery -A myproject flower --port=5555
More usefully, configure it to report metrics to a monitoring system:
# celery.py
app.conf.update(
# Enable events so Flower and monitoring tools can track tasks
worker_send_task_events=True,
task_send_sent_event=True,
)
We alert on two Flower metrics: queue depth (if a queue has more than 500 tasks, something is wrong) and worker count (if the number of active workers drops below the expected minimum, workers have died).
Retry strategy
Tasks fail. The database is momentarily unavailable, an external API returns a 503, a file is not there yet. The question is not whether tasks will fail but whether they will fail gracefully.
from celery import shared_task
from celery.utils.log import get_task_logger
logger = get_task_logger(__name__)
@shared_task(
bind=True,
max_retries=5,
default_retry_delay=60, # seconds
)
def send_invoice(self, invoice_id: int):
try:
invoice = Invoice.objects.get(id=invoice_id)
result = email_client.send_invoice(invoice)
invoice.mark_sent(result.message_id)
except Invoice.DoesNotExist:
# Don't retry — the invoice is genuinely gone
logger.error(f"Invoice {invoice_id} not found, not retrying")
return
except EmailServiceUnavailable as exc:
# Retry with exponential backoff
raise self.retry(
exc=exc,
countdown=60 * (2 ** self.request.retries), # 60s, 120s, 240s...
)
except Exception as exc: