Chapter 1 celery
1.1 What is Celery ?
Celery is a simple, flexible, and reliable, distributed systems handle large messages
With python frame written to perform tasks and the timing of the asynchronous tasks
1.2celery consisting of
Intermediate message à Broker
Task execution unit à worker
Task execution results are stored à Store
1.3 usage scenarios
Asynchronous tasks: the time-consuming task to operate celery to execute asynchronously, such as sending SMS / e-mail, push messaging, audio and video processing, etc.
Regular tasks: the timing of implementation of something, such as daily statistics
Chapter 2 celery Installation and Configuration
pip install celery
Message middleware: RabbitMQ / redis , as used herein, redis
docker run -p 6379:6379 --name=redis -d redis:latest redis-server
Chapter 3 celery perform asynchronous tasks
3.1 Basic Usage
Defining the task:
from Celery Import Celery
Import Time
# produce stored task Broker = 'Redis: //10.211.55.8: 16379/1' # task execution result storage backend = 'Redis: //10.211.55.8: 16379/2' # first parameter is an alias, to just write App = Celery ( 'first_celery' , backend = backend , Broker = Broker) @ app.task DEF test_celery (X , Y ): the time.sleep ( 2 ) return X + Y
Add a task:
import task
if __name__ == '__main__':
# result不是函数的执行结果，是一个对象
result = tasks.add.delay(2,3)
print(result.id)
使用celery命令开启work进程
celery –A tasks worker –l info
3.2多任务结构：
当有多个任务需要执行时，会有多个任务函数，位了方便解藕，我们可以使用这种方式，方便管理
celery.py
from celeryimport Celery
broker = 'redis://10.211.55.8:16379/2'
backend = 'redis://10.211.55.8:16379/3'
# include包含两个任务的函数，去相应文件中找对应的任务函数
cel = Celery('test', broker=broker, backend=backend, include=['celery_task.task', 'celery_task.task2'])
# 时区,有时候我们想在固定的时间来执行相关任务，就需要设置时区相关配置
cel.conf.timezone = 'Asia/Shanghai'
# 是否UTC
cel.conf.enable_utc = False
task.py
from .celeryimport app
@app.task
def sum(x, y):
returnx+y
task2.py
from .celeryimport app
@app.task
def less(x, y):
returnx-y
在celery_task目录下创建添加任务的py文件
from celery_taskimport task
from celery_task import task2
task.sum.delay(1, 2)
task2.less.delay(3, 1)
第4章 celery执行定时任务
4.1设定时间让celery执行一个任务
from celery_taskimport task
from celery_task import task2
from datetime import datetime, timedelta
# # 方式一：
# #获取时间对象
# time = datetime(2019, 8, 3, 15, 0, 0)
# print(time)
# # 将时间转换成UTC时间
# utc_time = datetime.utcfromtimestamp(time.timestamp())
# print(utc_time)
# # 用apply_async，args是函数参数，eta是指定时间
# result = task.add.apply_async(args=[1, 2], eta=utc_time)
# 方式二：
ctime = datetime.now()
# 把当前时间转换成UTC时间
utc_time = datetime.utcfromtimestamp(ctime.utcfromtimestamp())
# 当前时间延迟十秒
time_delta = timedelta(seconds=10)
task_time = utc_time + time_delta
result = task.add.apply_async(args=[2, 3], eta=task_time)
# task.sum.delay(1, 2)
# task2.less.delay(3, 1)
4.2类似crontab的定时任务
多任务结构中celery.py文件如下
from celeryimport Celery
from datetime import timedelta
from celery.schedules import crontab
broker = 'redis://10.211.55.8:16379/2'
backend = 'redis://10.211.55.8:16379/3'
# include包含两个任务的函数，去相应文件中找对应的任务函数
app = Celery('test', broker=broker, backend=backend, include=['celery_task.task', 'celery_task.task2'])
# 时区
app.conf.timezone = 'Asia/Shanghai'
# 是否UTC
app.conf.enable_utc = False
app.conf.beat_schedules = {
#别名，可随意编写
'add_crontab': {
#执行task下的add函数
'task': 'celery_task.task.add',
#每隔两秒执行
'schedules': timedelta(seconds=2),
#固定时间执行，每年4月11号，8点42分执行
#'schedules': crontab(minute=42, hour=8, day_of_month=11, month_of_year=4),
#传递参数
'args': [1, 2],
}
}
第5章 Django中使用celery
在项目的目录下创建celeryconfig.py
import djcelery
djcelery.setup_loader()
CELERY_IMPORTS=(
'app01.tasks',
)
#有些情况可以防止死锁
CELERYD_FORCE_EXECV=True
# 设置并发worker数量
CELERYD_CONCURRENCY=4
#允许重试
CELERY_ACKS_LATE=True
# 每个worker最多执行100个任务被销毁，可以防止内存泄漏
CELERYD_MAX_TASKS_PER_CHILD=100
# 超时时间
CELERYD_TASK_TIME_LIMIT=12*30
在app目录下创建tasks.py
from celeryimport task
@task
def add(x, y):
returnx+y
试图函数views中使用
from django.shortcutsimport render,HttpResponse
from app01.tasks import add
from datetime import datetime, timedelta
def test(request):
# result=add.delay(2,3)
ctime = datetime.now()
# 默认用utc时间
utc_ctime = datetime.utcfromtimestamp(ctime.timestamp())
time_delay= timedelta(seconds=5)
task_time = utc_ctime + time_delay
result = add.apply_async(args=[4, 3], eta=task_time)
print(result.id)
returnHttpResponse('ok')
settings中注册
INSTALLED_APPS= [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'djcelery',
'app01',
]
from djceleryimport celeryconfig
# 默认使用rabbitmq，redis的话需要指定下
BROKER_BACKEND='redis'
BROKER_URL='redis://127.0.0.1:6379/1'
CELERY_RESULT_BACKEND='redis://127.0.0.1:6379/2'