Celery (celery) is a Python-based development of distributed task queue. It supports the use of task execution queue scheduling mode in the distribution of machine / process / thread.
Architecture 
Celery architecture consists of three parts, the middleware message (message broker), the task execution unit (worker) and task execution result storage (task result store) composition.
-
Messaging middleware
Celery does not provide messaging services, but can be easily and messaging middleware integration provided by third parties. Including, RabbitMQ, Redis, MongoDB (experimental), Amazon SQS (experimental), CouchDB (experimental), SQLAlchemy (experimental), Django ORM (experimental), IronMQ
2. The task execution unit
Worker是Celery提供的任务执行的单元，worker并发的运行在分布式的系统节点中。
3. The task results are stored
Task result store用来存储Worker执行的任务的结果，Celery支持以不同方式存储任务的结果，包括AMQP, Redis，memcached, MongoDB，SQLAlchemy, Django ORM，Apache Cassandra, IronCache
In addition, Celery also supports different methods of concurrency and sequence
1. Concurrent
Prefork, Eventlet, gevent, threads/single threaded
2. Serialization
pickle, json, yaml, msgpack. zlib, bzip2 compression， Cryptographic message signing 等等
Install and run
Celery installation process is slightly more complex, following the installation process is based on the Linux version of my AWS EC2 installation process, a different system installation process may vary. You can refer to the official documentation.
First, I select as RabbitMQ messaging middleware, so you must install RabbitMQ. As the installation preparation, first update YUM.
sudo yum -y update
RabbitMQ is based on erlang, so install erlang
# Add and enable relevant application repositories:
# Note: We are also enabling third party remi package repositories.
wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
wget http://rpms.famillecollet.com/enterprise/remi-release-6.rpm
sudo rpm -Uvh remi-release-6*.rpm epel-release-6*.rpm
# Finally, download and install Erlang:
yum install -y erlang
Then install RabbitMQ
# Download the latest RabbitMQ package using wget:
wget
# Add the necessary keys for verification:
rpm --import
# Install the .RPM package using YUM:
yum install rabbitmq-server-3.2.2-1.noarch.rpm
Start RabbitMQ service
rabbitmq-server start
RabbitMQ service is ready, and then install Celery, pip assumes that you use to manage your installation package python
pip install Celery
To test whether Celery work, we run a simple task, write tasks.py
from celery import Celery
app = Celery('tasks', backend='amqp', broker='amqp://guest@localhost//')
app.conf.CELERY_RESULT_BACKEND = 'db+sqlite:///results.sqlite'
@app.task
def add(x, y):
return x + y
In the current directory, run a worker, in addition to the implementation of this task
celery -A tasks worker --loglevel=info
Wherein -A is represented by the parameter name Celery App. Note that I'm using SQLAlchemy as a result of storage. Corresponding python package to be installed in advance.
worker logs we will see this information
- ** ---------- [config]
- ** ---------- .> app: tasks:0x1e68d50
- ** ---------- .> transport: amqp://guest:**@localhost:5672//
- ** ---------- .> results: db+sqlite:///results.sqlite
- *** --- * --- .> concurrency: 8 (prefork)
Among them, we can see that the default use prefork worker to perform concurrent, and set the number of concurrent 8
Tasks performed by the following client code:
from tasks import add
import time
result = add.delay(4,4)
while not result.ready():
print "not ready yet"
time.sleep(5)
print result.get()
This client code execution with Python, at the client, the following results
not ready
8
Work log shows
[2015-03-12 02:54:07,973: INFO/MainProcess] Received task: tasks.add[34c4210f-1bc5-420f-a421-1500361b914f]
[2015-03-12 02:54:08,006: INFO/MainProcess] Task tasks.add[34c4210f-1bc5-420f-a421-1500361b914f] succeeded in 0.0309705100954s: 8
Here we can see that each task has a unique ID, task executed asynchronously on the worker.
To note here is that if you run the example official documents, you can not get the results of the client, which is why I want to use SQLAlchemy to store task execution results of reasons. Examples AMPQ official use, it is possible to remove the Worker results of running task is displayed when the worker logs in the print log, but AMPQ as a message queue when the message is removed, there is no queue, then the client Total is the result of the implementation of the task can not be obtained. I do not know why the official documentation error turning a blind eye to this.
We recommend the python learning sites, click to enter , to see how old the program is to learn! From basic python script, reptiles, django, data mining, programming techniques, work experience, as well as senior careful study of small python partners to combat finishing zero-based information projects! The method has timed programmer Python explain everyday technology, to share some of the learning and the need to pay attention to small details
