Celery v0.9.0 (unstable) documentation
Default celeryd loglevel is now WARN, to enable the previous log level start celeryd with --loglevel=INFO.
Tasks are automatically registered.
This means you no longer have to register your tasks manually. You don’t have to change your old code right away, as it doesn’t matter if a task is registered twice.
If you don’t want your task to be automatically registered you can set the abstract attribute
class MyTask(Task):
abstract = True
By using abstract only tasks subclassing this task will be automatically registered (this works like the Django ORM).
Incidentally, this change also fixes the problems with automatic name assignment and relative imports. So you also don’t have to specify a task name anymore if you use relative imports.
You can no longer use regular functions as tasks. This change was added because it makes the internals a lot more clean and simple. However, you can now turn functions into tasks by using the @task decorator:
from celery.decorators import task
@task()
def add(x, y):
return x + y
See the User Guide: Tasks for more information.
The periodic task system has been rewritten to a centralized solution, this means celeryd no longer schedules periodic tasks by default, but a new daemon has been introduced: celerybeat.
To launch the periodic task scheduler you have to run celerybeat:
$ celerybeat --detach
Make sure this is running on one server only, if you run it twice, all periodic tasks will also be executed twice.
If you only have one worker server you can embed it into celeryd like this:
$ celeryd --detatch --beat # Embed celerybeat in celeryd.
The supervisor has been removed, please use something like http://supervisord.org instead. This means the -S and --supervised options to celeryd is no longer supported.
TaskSet.join has been removed, use TaskSetResult.join instead.
The task status "DONE" has been renamed to “SUCCESS”.
AsyncResult.is_done has been removed, use AsyncResult.successful instead.
This release (with carrot 0.8.0) enables AMQP QoS (quality of service), which means the workers will only receive as many messages as it can handle at a time. As with any release, you should test this version upgrade on your development servers before rolling it out to production!
If you’re using Python < 2.6 and you use the multiprocessing backport, then multiprocessing version 2.6.2.1 is required.
All AMQP_* settings has been renamed to BROKER_*, and in addition AMQP_SERVER has been renamed to BROKER_HOST, so before where you had:
AMQP_SERVER = "localhost"
AMQP_PORT = 5678
AMQP_USER = "myuser"
AMQP_PASSWORD = "mypassword"
AMQP_VHOST = "celery"
You need to change that to:
BROKER_HOST = "localhost"
BROKER_PORT = 5678
BROKER_USER = "myuser"
BROKER_PASSWORD = "mypassword"
BROKER_VHOST = "celery"
Custom carrot backends now need to include the backend class name, so before where you had:
CARROT_BACKEND = "mycustom.backend.module"
you need to change it to:
CARROT_BACKEND = "mycustom.backend.module.Backend"
where Backend is the class name. This is probably "Backend", as that was the previously implied name.
NOTE If you use the database backend you have to re-create the database table celery_taskmeta.
Contact the mailinglist or IRC channel listed in README for help doing this.
so if you change back to the database backend at some point, be sure to initialize tables (django: syncdb, python: celeryinit). (Note: This is only the case when using Django 1.1 or higher)
Now depends on carrot version 0.6.0.
Now depends on python-daemon 1.4.8
This means celery is no longer Django specific.
For more information see the FAQ entry Can I use celery without Django?.
Celery now supports task retries.
See Cookbook: Retrying Tasks for more information.
We now have an AMQP result store backend.
It uses messages to publish task return value and status. And it’s incredibly fast!
See http://github.com/ask/celery/issues/closed#issue/6 for more info!
This to not receive more messages than we can handle.
Now redirects stdout/stderr to the celeryd logfile when detached
the task supports.
by creating your own loaders. (see celery.loaders.default, celery.loaders.djangoapp, celery.loaders.)
Support for multiple AMQP exchanges and queues.
This feature misses documentation and tests, so anyone interested is encouraged to improve this situation.
celeryd now survives a restart of the AMQP server!
Automatically re-establish AMQP broker connection if it’s lost.
New settings:
Set to True to enable connection retries.
Maximum number of restarts before we give up. Default: 100.
which resulted in the [Errno 10] No child processes problem when detaching.
data to Django’s memcached cache backend.
Better Windows compatibility.
task_postrun, see celery.signals for more information.
Thanks Jerzy Kozera. Closes #31
Thanks to Jerzy Kozera. Closes #32
to timedelta to the class attribute instead of on the instance.
available in the previous module.
while setting task status with the database backend.
jail() refactored into celery.execute.ExecuteWrapper.
views.apply now correctly sets mimetype to “application/json”
views.task_status now returns exception if status is RETRY
or “RETRY”
Documented default task arguments.
Add a sensible __repr__ to ExceptionInfo for easier debugging
Thanks mikedizon
workers. So if you’ve had pool workers mysteriously dissapearing, or problems with celeryd stopping working, this has been fixed in this version.
Fixed a race condition with periodic tasks.
goes away or stops responding, it is automatically replaced with a new one.
"djangotwitter.tasks.UpdateStatusesTask". Very convenient. No idea why we didn’t do this before. Some documentation is updated to not manually specify a task name.
Tested with Django 1.1
New Tutorial: Creating a click counter using carrot and celery
startup instead of for each check (which has been a forgotten TODO/XXX in the code for a long time)
Time (in seconds, or a datetime.timedelta object) for when after stored task results are deleted. For the moment this only works for the database backend.
has been launched.
periodic task status. (MySQL only so far, seeking patches for other engines)
DEBUG loglevel (--loglevel=DEBUG).
Functions/methods with a timeout argument now works correctly.
With an iterator yielding task args, kwargs tuples, evenly distribute the processing of its tasks throughout the time window available.
Log message Unknown task ignored... now has loglevel ERROR
the task has an ETA (estimated time of arrival). Also the message now includes the ETA for the task (if any).
target, as it’s not pickleable (can’t share AMQP connection, etc)).
Added note about .delay hanging in README
Tests now passing in Django 1.1
Fixed discovery to make sure app is in INSTALLED_APPS
available, etc.) is now handled by multiprocessing.Pool itself.
Convert statistics data to unicode for use as kwargs. Thanks Lucy!
New arguments to apply_async (the advanced version of delay_task), countdown and eta;
>>> # Run 10 seconds into the future. >>> res = apply_async(MyTask, countdown=10);>>> # Run 1 day from now >>> res = apply_async(MyTask, eta=datetime.now() + ... timedelta(days=1)
Now unlinks the pidfile if it’s stale.
Lots of more tests.
Now compatible with carrot >= 0.5.0.
IMPORTANT The subtask_ids attribute on the TaskSetResult instance has been removed. To get this information instead use:
>>> subtask_ids = [subtask.task_id for subtask in ts_res.subtasks]
Taskset.run() now respects extra message options from the task class.
Task: Add attribute ignore_result: Don’t store the status and return value. This means you can’t use the celery.result.AsyncResult to check if the task is done, or get its return value. Only use if you need the performance and is able live without these features. Any exceptions raised will store the return value/status as usual.
Task: Add attribute disable_error_emails to disable sending error emails for that task.
Should now work on Windows (although running in the background won’t work, so using the --detach argument results in an exception being raised.)
Added support for statistics for profiling and monitoring. To start sending statistics start celeryd with the --statistics option. Then after a while you can dump the results by running python manage.py celerystats. See celery.monitoring for more information.
The celery daemon can now be supervised (i.e it is automatically restarted if it crashes). To use this start celeryd with the --supervised option (or alternatively -S).
views.apply: View applying a task. Example:
http://e.com/celery/apply/task_name/arg1/arg2//?kwarg1=a&kwarg2=b
NOTE Use with caution, preferably not make this publicly accessible without ensuring your code is safe!
Refactored celery.task. It’s now split into three modules:
celery.task
Contains apply_async, delay_task, discard_all, and task shortcuts, plus imports objects from celery.task.base and celery.task.builtins
celery.task.base
Contains task base classes: Task, PeriodicTask, TaskSet, AsynchronousMapTask, ExecuteRemoteTask.
celery.task.builtins
Built-in tasks: PingTask, DeleteExpiredTaskMetaTask.
IMPORTANT Now uses AMQP’s basic.consume instead of basic.get. This means we’re no longer polling the broker for new messages.
IMPORTANT Default concurrency limit is now set to the number of CPUs available on the system.
IMPORTANT tasks.register: Renamed task_name argument to name, so
>>> tasks.register(func, task_name="mytask")
has to be replaced with:
>>> tasks.register(func, name="mytask")
The daemon now correctly runs if the pidlock is stale.
Now compatible with carrot 0.4.5
Default AMQP connnection timeout is now 4 seconds.
AsyncResult.read() was always returning True.
Only use README as long_description if the file exists so easy_install doesn’t break.
celery.view: JSON responses now properly set its mime-type.
apply_async now has a connection keyword argument so you can re-use the same AMQP connection if you want to execute more than one task.
Handle failures in task_status view such that it won’t throw 500s.
Fixed typo AMQP_SERVER in documentation to AMQP_HOST.
Worker exception e-mails sent to admins now works properly.
No longer depends on django, so installing celery won’t affect the preferred Django version installed.
Now works with PostgreSQL (psycopg2) again by registering the PickledObject field.
celeryd: Added --detach option as an alias to --daemon, and it’s the term used in the documentation from now on.
Make sure the pool and periodic task worker thread is terminated properly at exit. (So Ctrl-C works again).
Now depends on python-daemon.
Removed dependency to simplejson
Cache Backend: Re-establishes connection for every task process if the Django cache backend is memcached/libmemcached.
Tyrant Backend: Now re-establishes the connection for every task executed.
NOTE This is a development version, for the stable release, please see versions 0.2.x.
VERY IMPORTANT: Pickle is now the encoder used for serializing task arguments, so be sure to flush your task queue before you upgrade.
IMPORTANT TaskSet.run() now returns a celery.result.TaskSetResult instance, which lets you inspect the status and return values of a taskset as it was a single entity.
IMPORTANT Celery now depends on carrot >= 0.4.1.
The celery daemon now sends task errors to the registered admin e-mails. To turn off this feature, set SEND_CELERY_TASK_ERROR_EMAILS to False in your settings.py. Thanks to Grégoire Cachet.
You can now run the celery daemon by using manage.py:
$ python manage.py celeryd
Thanks to Grégoire Cachet.
Added support for message priorities, topic exchanges, custom routing keys for tasks. This means we have introduced celery.task.apply_async, a new way of executing tasks.
You can use celery.task.delay and celery.Task.delay like usual, but if you want greater control over the message sent, you want celery.task.apply_async and celery.Task.apply_async.
This also means the AMQP configuration has changed. Some settings has been renamed, while others are new:
CELERY_AMQP_EXCHANGE
CELERY_AMQP_PUBLISHER_ROUTING_KEY
CELERY_AMQP_CONSUMER_ROUTING_KEY
CELERY_AMQP_CONSUMER_QUEUE
CELERY_AMQP_EXCHANGE_TYPE
See the entry Can I send some tasks to only some servers? in the FAQ for more information.
Forgot to add yadayada to install requirements.
Now deletes all expired task results, not just those marked as done.
Able to load the Tokyo Tyrant backend class without django configuration, can specify tyrant settings directly in the class constructor.
Improved API documentation
Now using the Sphinx documentation system, you can build the html documentation by doing
$ cd docs
$ make html
and the result will be in docs/.build/html.
delay_task() etc. now returns celery.task.AsyncResult object, which lets you check the result and any failure that might have happened. It kind of works like the multiprocessing.AsyncResult class returned by multiprocessing.Pool.map_async.
Added dmap() and dmap_async(). This works like the multiprocessing.Pool versions except they are tasks distributed to the celery server. Example:
>>> from celery.task import dmap >>> import operator >>> dmap(operator.add, [[2, 2], [4, 4], [8, 8]]) >>> [4, 8, 16]>>> from celery.task import dmap_async >>> import operator >>> result = dmap_async(operator.add, [[2, 2], [4, 4], [8, 8]]) >>> result.ready() False >>> time.sleep(1) >>> result.ready() True >>> result.result [4, 8, 16]
Refactored the task metadata cache and database backends, and added a new backend for Tokyo Tyrant. You can set the backend in your django settings file. e.g:
CELERY_BACKEND = "database"; # Uses the database
CELERY_BACKEND = "cache"; # Uses the django cache framework
CELERY_BACKEND = "tyrant"; # Uses Tokyo Tyrant
TT_HOST = "localhost"; # Hostname for the Tokyo Tyrant server.
TT_PORT = 6657; # Port of the Tokyo Tyrant server.
Introducing TaskSet. A set of subtasks is executed and you can find out how many, or if all them, are done (excellent for progress bars and such)
Now catches all exceptions when running Task.__call__, so the daemon doesn’t die. This does’t happen for pure functions yet, only Task classes.
autodiscover() now works with zipped eggs.
celeryd: Now adds curernt working directory to sys.path for convenience.
The run_every attribute of PeriodicTask classes can now be a datetime.timedelta() object.
celeryd: You can now set the DJANGO_PROJECT_DIR variable for celeryd and it will add that to sys.path for easy launching.
Can now check if a task has been executed or not via HTTP.
You can do this by including the celery urls.py into your project,
>>> url(r'^celery/$', include("celery.urls"))
then visiting the following url,:
http://mysite/celery/$task_id/done/
this will return a JSON dictionary like e.g:
>>> {"task": {"id": $task_id, "executed": true}}
delay_task now returns string id, not uuid.UUID instance.
Now has PeriodicTasks, to have cron like functionality.
Project changed name from crunchy to celery. The details of the name change request is in docs/name_change_request.txt.