This document is for Celery's development version, which can be significantly different from previous releases. Get old docs here: 2.5.

celery.concurrency.processes

class celery.concurrency.processes.TaskPool(limit=None, putlocks=True, **options)

Multiprocessing Pool implementation.

class Pool(processes=None, initializer=None, initargs=(), maxtasksperchild=None, timeout=None, soft_timeout=None, lost_worker_timeout=10.0)

Class which supports an async version of applying functions to arguments.

class Process(group=None, target=None, name=None, args=(), kwargs={}, daemon=None, **_kw)

Process objects represent activity that is run in a separate process

The class is analagous to threading.Thread

authkey

daemon

exitcode

Return exit code of process or None if it has yet to stop

ident

Return identifier (PID) of process or None if it has yet to start

is_alive()

Return whether process is alive

join(timeout=None)

Wait until child process terminates

name

pid

Return identifier (PID) of process or None if it has yet to start

run()

Method to be run in sub-process; can be overridden in sub-class

sentinel

Return a file descriptor (Unix) or handle (Windows) suitable for waiting for process termination.

start()

Start child process

terminate()

Terminate process; sends SIGTERM signal or uses TerminateProcess()

class TaskPool.Pool.ResultHandler(outqueue, get, cache, poll, join_exited_workers, putlock)

body()

exception TaskPool.Pool.SoftTimeLimitExceeded

The soft time limit has been exceeded. This exception is raised to give the task a chance to clean up.

class TaskPool.Pool.Supervisor(pool)

body()

class TaskPool.Pool.TaskHandler(taskqueue, put, outqueue, pool)

body()

class TaskPool.Pool.TimeoutHandler(processes, cache, t_soft, t_hard)

body()

TaskPool.Pool.apply(func, args=(), kwds={})

Equivalent of func(*args, **kwargs).

TaskPool.Pool.apply_async(func, args=(), kwds={}, callback=None, error_callback=None, accept_callback=None, timeout_callback=None, waitforslot=False, soft_timeout=None, timeout=None, lost_worker_timeout=None)

Asynchronous equivalent of apply() method.

Callback is called when the functions return value is ready. The accept callback is called when the job is accepted to be executed.

Simplified the flow is like this:

>>> if accept_callback:
...     accept_callback()
>>> retval = func(*args, **kwds)
>>> if callback:
...     callback(retval)

TaskPool.Pool.close()

TaskPool.Pool.grow(n=1)

TaskPool.Pool.imap(func, iterable, chunksize=1, lost_worker_timeout=None)

Equivalent of map() – can be MUCH slower than Pool.map().

TaskPool.Pool.imap_unordered(func, iterable, chunksize=1, lost_worker_timeout=None)

Like imap() method but ordering of results is arbitrary.

TaskPool.Pool.join()

TaskPool.Pool.map(func, iterable, chunksize=None)

Apply func to each element in iterable, collecting the results in a list that is returned.

TaskPool.Pool.map_async(func, iterable, chunksize=None, callback=None, error_callback=None)

Asynchronous equivalent of map() method.

TaskPool.Pool.restart()

TaskPool.Pool.shrink(n=1)

TaskPool.Pool.starmap(func, iterable, chunksize=None)

Like map() method but the elements of the iterable are expected to be iterables as well and will be unpacked as arguments. Hence func and (a, b) becomes func(a, b).

TaskPool.Pool.starmap_async(func, iterable, chunksize=None, callback=None, error_callback=None)

Asynchronous version of starmap() method.

TaskPool.Pool.terminate()

TaskPool.grow(n=1)

TaskPool.num_processes

TaskPool.on_start()

Run the task pool.

Will pre-fork all workers so they’re ready to accept tasks.

TaskPool.on_stop()

Gracefully stop the pool.

TaskPool.on_terminate()

Force terminate the pool.

TaskPool.requires_mediator = True

TaskPool.restart()

TaskPool.shrink(n=1)

TaskPool.terminate_job(pid, signal=None)

celery.concurrency.processes.WORKER_SIGIGNORE = frozenset(['SIGINT'])

List of signals to ignore when a child process starts.

celery.concurrency.processes.WORKER_SIGRESET = frozenset(['SIGHUP', 'SIGTERM', 'SIGTTOU', 'SIGTTIN', 'SIGUSR1'])

List of signals to reset when a child process starts.

celery.concurrency.processes.process_initializer(app, hostname)

Initializes the process so it can be used to process tasks.