Configuring Celery for the best performance

For some reason many Django developers choose the Django database as a Celery message broker and a result backend. Maybe they don't want to bring in more third-party software like RabbitMQ or Redis, but performance and stability suffer a lot.

The Django database is not a recommended Celery message broker. It's experimental and not suitable for production. Plus, it has serious limitations because it does not support remote control commands, events and the usage of more than a few workers.

Storing tasks results in Redis reduces the number of database connections and improves your application performance. Moreover, there are not much reasons for storing such data in your production database. So, let's use a faster key-value storage like Redis for that.

Install missing dependencies with $ pip install redis django-redis kombu flower

Change your Django settings to:

BROKER_URL = 'redis://localhost:6379/0'  
CELERY_ENABLE_UTC = False  # Not converting dates and times to the UTC timezone  
CELERY_TIMEZONE = TIME_ZONE  # Using the timezone from Django settings  
CELERY_TRACK_STARTED = True  # Having a “started” state can be useful for when there are long running tasks and there is a need to report which task is currently running.  
CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'  
CELERYD_CONCURRENCY = 8  # Defaults to the number of available CPUs, but I prefer doubling it.  
CELERYD_TASK_SOFT_TIME_LIMIT = 60 * 20  
CELERYD_TASK_TIME_LIMIT = 60 * 30  # The worker processing the task will be killed and replaced with a new one when this is exceeded.  
CELERY_SEND_TASK_SENT_EVENT = True  # Tasks can be tracked before they are consumed by a worker.  

We will be monitoring tasks with flower, a Celery monitoring tool. Let's define a Supervisor-controlled program:

[program:flower]
; Set full path to flower program if using virtualenv
command=/var/www/yourapp/.env/bin/flower -A yourapp --url_prefix=flower  
stdout_logfile=/var/log/supervisor/flower.yourapp.log  
stderr_logfile=/var/log/supervisor/flower.yourapp.log  

Re-read your Supervisor config and restart it with $ sudo supervisorctl reread && sudo supervisorctl reload.

Next, you should alter your server definition in your nginx config:

server {  
    [skipped]

    location /flower/ {
        rewrite ^/flower/(.*)$ /$1 break;
        proxy_pass http://127.0.0.1:5555;
        proxy_set_header Host $host;

        auth_basic "Restricted Content";
        auth_basic_user_file /etc/nginx/.htpasswd;
    }
}

We will be using a password-protected location for flower. You can easily create a .htpasswd file with standard tools from Linux.

$ sudo sh -c "echo -n 'username:' >> /etc/nginx/.htpasswd"
$ sudo sh -c "openssl passwd -apr1 >> /etc/nginx/.htpasswd"

Now, you should be able to access http://yourapp.com/flower/ for getting Celery tasks results.

Michael Samoylov

Python, JavaScript and Swift Expert with 12+ years of experience.

Vilnius, Lithuania https://monmar.tech

Subscribe to Michael Samoylov

Get the latest posts delivered right to your inbox.

or subscribe via RSS with Feedly!