Celery manually configure routing

1. Demand

We have a variety of different tasks with different priorities, for example we have tasks such as video uploading and compression, etc. tasks such as photo compression uploading and other less important tasks. These tasks take different time and need workerto be handled differently. Just using celerythe default queue will not meet our needs.
This requires us to taskroute different queues to different queues and let different ones workerhandle different kinds of things task.

2. Create queues and exchanges

For exchanges and queues, you can take a look at http://rabbitmq.mr-ping.com/

default_exchange = Exchange('dedfault', type='direct')

# 定义一个媒体交换机,类型是直连交换机
media_exchange = Exchange('media', type='direct')

# 创建三个队列，一个是默认队列，一个是video、一个image
CELERY_QUEUES = (
    Queue('default', default_exchange, routing_key='default'),
    Queue('videos', media_exchange, routing_key='media.video'),
    Queue('images', media_exchange, routing_key='media.image')
)
# 定义默认队列和默认的交换机routing_key
CELERY_DEFAULT_QUEUE = 'default'
CELERY_DEFAULT_EXCHANGE = 'default'
CELERY_DEFAULT_ROUTING_KEY = 'default'

3. Create`task`

from celery import Celery
import time

app = Celery()
app.config_from_object('celeryconfig')

# 视频压缩
@app.task
def video_compress(video_name):
    time.sleep(10)
    print 'Compressing the:', video_name
    return 'success'

@app.task
def video_upload(video_name):
    time.sleep(5)
    print u'正在上传视频'
    return 'success'

# 压缩照片
@app.task
def image_compress(image_name):
    time.sleep(10)
    print 'Compressing the:', image_name
    return 'success'

# 其他任务
@app.task
def other(str):
    time.sleep(10)
    print 'Do other things'
    return 'success'

We have defined three queues, now we want to taskroute the action videos and action photos to specific queues respectively.

4. Specify the route

CELERY_ROUTES = ({'tasks.image_compress': {
                        'queue': 'images',
                        'routing_key': 'media.image'
                 }},{'tasks.video_upload': {
                        'queue': 'videos',
                        'routing_key': 'media.video'
                 }},{'tasks.video_compress': {
                        'queue': 'videos',
                        'routing_key': 'media.video'
                 }}, )

By specifying the queue CELERY_ROUTESfor each task, if a task arrives, let the specified workerprocess be processed by the name of the task.

5. `task`Registration

For the name of the task, see this document http://docs.jinkan.org/docs/celery/userguide/tasks.html
Celery can automatically generate the name, if the task is not registered, an error will occur. After searching, I found that someone used the following method to solve it.

CELERY_IMPORTS = ("tasks",)

tasks is the name of the module where I save the tasks, so that workerthe tasks can be registered with the worker when they are created, like this:

The tasks below [tasks] in the picture are the tasks in my tasks file.

6. Complete code

tasks.py

from celery import Celery
import time


app = Celery()
app.config_from_object('celeryconfig')

# 视频压缩
@app.task
def video_compress(video_name):
    time.sleep(10)
    print 'Compressing the:', video_name
    return 'success'

@app.task
def video_upload(video_name):
    time.sleep(5)
    print u'正在上传视频'
    return 'success'

# 压缩照片
@app.task
def image_compress(image_name):
    time.sleep(10)
    print 'Compressing the:', image_name
    return 'success'

# 其他任务
@app.task
def other(str):
    time.sleep(10)
    print 'Do other things'
    return 'success'

celeryconfig.py

from kombu import Exchange, Queue
from routers import MyRouter

# 配置市区
CELERY_TIMEZONE = 'Asia/Shanghai'
CELERY_BROKER = 'amqp://localhost'
# 定义一个默认交换机
default_exchange = Exchange('dedfault', type='direct')

# 定义一个媒体交换机
media_exchange = Exchange('media', type='direct')

# 创建三个队列，一个是默认队列，一个是video、一个image
CELERY_QUEUES = (
    Queue('default', default_exchange, routing_key='default'),
    Queue('videos', media_exchange, routing_key='media.video'),
    Queue('images', media_exchange, routing_key='media.image')
)

CELERY_DEFAULT_QUEUE = 'default'
CELERY_DEFAULT_EXCHANGE = 'default'
CELERY_DEFAULT_ROUTING_KEY = 'default'
#
CELERY_ROUTES = ({'tasks.image_compress': {
                        'queue': 'images',
                        'routing_key': 'media.image'
                 }},{'tasks.video_upload': {
                        'queue': 'videos',
                        'routing_key': 'media.video'
                 }},{'tasks.video_compress': {
                        'queue': 'videos',
                        'routing_key': 'media.video'
                 }}, )

# 在出现worker接受到的message出现没有注册的错误时，使用下面一句能解决
CELERY_IMPORTS = ("tasks",)

Note that workera queue needs to be formulated at startup, and a terminal needs to be opened in the directory where the task is saved to start the worker. I have not yet figured out this problem!

# 启动worker1
celery worker -Q default --loglevel=info

celery worker -Q videos --loglevel=info

start processing imagesworker

celery worker -Q images --loglevel=info

In this way, we can route different types of tasks to different workers for processing.

pit encountered

When you modify the configuration file and need to restart the worker, remember to close and restart the python shell as well.
If the task is called but the worker gets an error like this

unregistered task of type

It means that your task is not registered, you need to add what I mentioned above CELERY_IMPORTS = ("tasks",)to the configuration file.
It is recommended to do more debugging by yourself, and take a look at the official documentation of celery, and the things in it will make us see it.


Author: hehe_Xiaoyu classmate
Link: http://www.jianshu.com/p/11b420aea529
Source: Jianshu
Copyright belongs to the author. For commercial reprints, please contact the author for authorization, and for non-commercial reprints, please indicate the source.