Troubleshooting RabbitMQ and Celery Configuration Issues

Introduction

When using RabbitMQ as a message broker for Celery tasks, you may encounter issues where tasks stop executing, often accompanied by connection errors. This guide provides steps to troubleshoot and rectify these issues, including configuration tips for running Celery workers as daemons.

Common Connection Issues

If you experience the error Celery - errno 111 connection refused, it typically indicates that the RabbitMQ server is not accessible. This can happen for several reasons, including the server being down or misconfigured.

Steps to Resolve Connection Issues

Check RabbitMQ Status: Ensure that the RabbitMQ service is running. You can check its status with:
```
sudo systemctl status rabbitmq-server
```
Reinstall RabbitMQ: If the service is not functioning correctly, you may need to reinstall RabbitMQ. Use the following commands:
```
sudo apt-get --purge remove rabbitmq-server
sudo apt-get install rabbitmq-server
```

Verify Configuration: Ensure your Celery configuration is set correctly. Here’s a sample configuration:

BROKER_URL = 'amqp://'
CELERY_RESULT_BACKEND = 'amqp://'
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TIMEZONE = 'Europe/Oslo'
CELERY_ENABLE_UTC = True
CELERY_CREATE_MISSING_QUEUES = True

Daemonizing Celery Workers

If you want to run multiple Celery workers as daemons, you can configure them to start automatically without manual intervention. Here’s how to set it up:

Celery Configuration for Daemonization

Define Worker Nodes: Specify the nodes you want to run in your Celery configuration:
```
CELERYD_NODES="w1 w2 w3 w4"
```
Set Working Directory: Define the directory where your project is located:
```
CELERYD_CHDIR="/var/www/fractal/parser-quicklook/"
```

Configure Celery Options: Set additional options for your workers, including time limits and queues:

CELERYD_OPTS="--time-limit=300 --concurrency=8 -Q BBC,BGR,FASTCOMPANY,Firstpost,Guardian,IBNLIVE,LIVEMINT,Mashable,NDTV,Pandodaily,Reuters,TNW,TheHindu,ZEENEWS"

Logging and PID Files: Specify where to log output and store the process ID:

CELERYD_LOG_FILE="/var/log/celery/%n.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"

Environment Variables: Set any necessary environment variables for your project:
```
PROJECT_ENV="PRODUCTION"
CELERY_CREATE_DIRS=1
```

Dynamic Queue Management

To dynamically manage queues, ensure that CELERY_CREATE_MISSING_QUEUES is set to True. This allows Celery to create any queues that do not exist when tasks are sent to them. However, there is no built-in feature to automatically daemonize queues; you will need to define them in your worker configuration as shown above.

Conclusion

By following these steps, you should be able to resolve common RabbitMQ connection issues and configure your Celery workers to run as daemons effectively. If problems persist, consider checking RabbitMQ logs for more detailed error messages.