Celery v0.9.0 (unstable) documentation
This document describes the configuration options available.
If you’re using celery in a Django project these settings should be defined in your projects settings.py file.
In a regular Python environment using the default loader you must create the celeryconfig.py module and make sure it is available on the Python path.
This is an example configuration file to get you started, it should contain all you need to run a basic celery set-up.
CELERY_BACKEND = "database"
DATABASE_ENGINE = "sqlite3"
DATABASE_NAME = "mydatabase.db"
BROKER_HOST = "localhost"
BROKER_PORT = 5672
BROKER_VHOST = "/"
BROKER_USER = "guest"
BROKER_PASSWORD = "guest"
## If you're doing mostly I/O you can have higher concurrency,
## if mostly spending time in the CPU, try to keep it close to the
## number of CPUs on your machine.
# CELERYD_CONCURRENCY = 8
CELERYD_LOG_FILE = "celeryd.log"
CELERYD_PID_FILE = "celeryd.pid"
CELERYD_DAEMON_LOG_LEVEL = "INFO"
The number of concurrent worker processes, executing tasks simultaneously.
Defaults to the number of CPUs in the system.
The backend used to store task results (tombstones). Can be one of the following:
Use a relational database supported by the Django ORM.
Use memcached to store the results.
Use MongoDB to store the results.
Use Redis to store the results.
Use Tokyo Tyrant to store the results.
Send results back as AMQP messages (WARNING While very fast, you must make sure you only try to receive the result once).
Please see the Django ORM database settings documentation: http://docs.djangoproject.com/en/dev/ref/settings/#database-engine
If you use this backend make sure to initialize the database tables after configuration. When using celery with a Django project this means executing:
$ python manage.py syncdb
When using celery in a regular Python environment you have to execute:
$ celeryinit
CELERY_BACKEND = "database"
DATABASE_ENGINE = "mysql"
DATABASE_USER = "myusername"
DATABASE_PASSWORD = "mypassword"
DATABASE_NAME = "mydatabase"
DATABASE_HOST = "localhost"
Please see the documentation for the Django cache framework settings: http://docs.djangoproject.com/en/dev/topics/cache/#memcached
To use a custom cache backend for Celery, while using another for Django, you should use the CELERY_CACHE_BACKEND setting instead of the regular django CACHE_BACKEND setting.
Using a single memcached server:
CACHE_BACKEND = 'memcached://127.0.0.1:11211/'
Using multiple memcached servers:
CELERY_BACKEND = "cache"
CACHE_BACKEND = 'memcached://172.19.26.240:11211;172.19.26.242:11211/'
This backend requires the following configuration directives to be set:
Hostname of the Tokyo Tyrant server.
The port the Tokyo Tyrant server is listening to.
CELERY_BACKEND = "tyrant"
TT_HOST = "localhost"
TT_PORT = 1978
To install the redis package use pip or easy_install:
$ pip install redis
This backend requires the following configuration directives to be set:
REDIS_HOST
Hostname of the Redis database server. e.g. "localhost".
REDIS_PORT
Port to the Redis database server. e.g. 6379.
Also, the following optional configuration directives are available:
REDIS_DB
Name of the database to use. Default is celery_results.
REDIS_TIMEOUT
Timeout in seconds before we give up establishing a connection to the Redis server.
REDIS_CONNECT_RETRY
Retry connecting if an connection could not be established. Default is false.
CELERY_BACKEND = "pyredis"
REDIS_HOST = "localhost"
REDIS_PORT = 6739
REDIS_DATABASE = "celery_results"
REDIS_CONNECT_RETRY=True
CELERY_MONGODB_BACKEND_SETTINGS
This is a dict supporting the following keys:
- host
Hostname of the MongoDB server. Defaults to “localhost”.
- port
The port the MongoDB server is listening to. Defaults to 27017.
- user
Username to authenticate to the MongoDB server as (optional).
- password
Password to authenticate to the MongoDB server (optional).
- database
The database name to connect to. Defaults to “celery”.
- taskmeta_collection
The collection name to store task metadata. Defaults to “celery_taskmeta”.
CELERY_BACKEND = "mongodb"
CELERY_MONGODB_BACKEND_SETTINGS = {
"host": "192.168.1.100",
"port": 30000,
"database": "mydb",
"taskmeta_collection": "my_taskmeta_collection",
}
CELERY_AMQP_EXCHANGE
Name of the AMQP exchange.
The type of exchange. If the exchange type is direct, all messages receives all tasks. However, if the exchange type is topic, you can route e.g. some tasks to one server, and others to the rest. See Exchange types and the effect of bindings.
The default AMQP routing key used when publishing tasks.
The AMQP routing key used when consuming tasks.
The name of the AMQP queue.
Dictionary defining multiple AMQP queues.
The timeout in seconds before we give up establishing a connection to the AMQP server. Default is 4 seconds.
Automatically try to re-establish the connection to the AMQP broker if it’s lost.
The time between retries is increased for each retry, and is not exhausted before CELERY_AMQP_CONNECTION_MAX_RETRIES is exceeded.
This behaviour is on by default.
Maximum number of retries before we give up re-establishing a connection to the AMQP broker.
If this is set to 0 or None, we will retry forever.
Default is 100 retries.
If set to True, errors in tasks will be sent to admins by e-mail. If unset, it will send the e-mails if settings.DEBUG is False.
If this is True, all tasks will be executed locally by blocking until it is finished. apply_async and Task.delay will return a celery.result.EagerResult which emulates the behaviour of celery.result.AsyncResult, except the result has already been evaluated.
Tasks will never be sent to the queue, but executed locally instead.
Time (in seconds, or a datetime.timedelta object) for when after stored task tombstones are deleted.
NOTE: For the moment this only works for the database and MongoDB backends., except the result has already been evaluated.
A string identifying the default serialization method to use. Can be pickle (default), json, yaml, or any custom serialization methods that have been registered with carrot.serialization.registry.
Default is pickle.
The default filename the worker daemon logs messages to, can be overridden using the –logfile` option to celeryd.
The default is to log using stderr if running in the foreground, when running in the background, detached as a daemon, the default logfile is celeryd.log.
Worker log level, can be any of DEBUG, INFO, WARNING, ERROR, CRITICAL, or FATAL.
See the logging module for more information.
The format to use for log messages. Can be overridden using the --loglevel option to celeryd.
Default is [%(asctime)s: %(levelname)s/%(processName)s] %(message)s
See the Python logging module for more information about log formats.
Full path to the daemon pid file. Default is celeryd.pid. Can be overridden using the --pidfile option to celeryd.