Celery use in Django's introduction

Celery Profile

celery is a simple, flexible and reliable, distributed processing system, a large number of messages, and to provide such a system must maintain tool.

It is a focus on the task queue is processed in real time, but also allow the task scheduler.

What is the job queue

Task Queue: a mechanism for the distribution of tasks between threads and machines.

Three major components of celery

worker

Task execution unit -> Worker Task Celery unit is provided to perform, worker operating in a distributed concurrent system node.

broker (deposit tasks of the warehouse)

Messaging middleware -> Celery itself does not provide messaging services, but can be easily and messaging middleware integration provided by third parties. Including, RabbitMQ, Redis, etc.

backend (save results of a warehouse)

ask result store for storing results of the executed task Worker, Celery stored in different ways to support the results of tasks, including AMQP, redis etc.

scenes to be used

Asynchronous tasks: submit time-consuming operational tasks to celery to execute asynchronously, such as sending text messages, e-mail, push messaging, audio and video processing.
Regular tasks: the timing of implementation of something, such as statistics day

Basic Commands

# 1. 启动celery服务：
#   非windows:
#   指令：celery worker -A celery_task(celery项目文件) -l info
#   windows: 需要先下载eventlet模块，pip install eventlet
#   指令： celery worker -A celery_task -l info -P eventlet

# 2. 添加任务：手动添加，需要自定义添加任务脚本；自动添加任务，在celery.py中配置

# 3. 获取结构：手动获取，需要自定义任务脚本

celery used in the Django project

celery directory structure

project
    |---celery_task
        |---celery.py # celery连接和相关配置，且名字必须是celery.py，如果要自动添加任务，那么相                         关配置也在celery.py里配置；
        |---tasks.py  # 所有任务函数
    |---add_task.py   # 手动添加任务：立即任务，延时任务，定时任务；
    |---get_result.py # 获取结果

The latter two documents can not add, look at demand.

use

celery.py

from celery import Celery
# 导入时间相关包，用法看下面
from datetime import timedelta
from celery.schedules import crontab

# 因为需要调用Django项目中的models，所有需要添加Django环境
import os,django
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'bookapi.settings.dev')
django.setup()

# 添加任务的仓库，这里使用了Redis
broker = "redis://127.0.0.1:6379/11"
# 接收处理结果的仓库
backend = "redis://127.0.0.1:6379/12"
# 指定需要处理的任务
include = ['celery_task.tasks']
app = Celery(broker=broker,backend=backend,include=include)

# 配置任务时区
app.conf.timezone = 'Asia/Shanghai'
app.conf.enable_utc = False

# 配置定时任务
app.conf.beat_schedule = {
    'recommend-task': {
        'task': 'celery_task.tasks.recommend_num',
        # 'schedule': timedelta(seconds=20),
        'schedule': crontab(hour=24),  
        'args': ()
    },
    'monthly-task': {
        'task': 'celery_task.tasks.monthly_num',
        # 'schedule': timedelta(seconds=60),
        'schedule': crontab(day_of_month=1,hour=0),
        'args': ()
    }
}

tasks.py

from .celery import app
from bookapi.apps.user import models


@app.task
def recommend_num():
    user_list = models.User.objects.all()
    # print(user_list)   ## <QuerySet [<User: admin>, <User: 18700022899>]>
    for user in user_list:
        models.User.objects.filter(username=user.username).update(recommend_nums=3)


@app.task
def monthly_num():
    user_list = models.User.objects.all()
    # print(user_list)   ## <QuerySet [<User: admin>, <User: 18700022899>]>
    for user in user_list:
        models.User.objects.filter(username=user.username).update(monthly_nums=2)