1. What is Celery
Celery is a simple, flexible and reliable, distributed systems handle large messages
Focus on real-time processing of asynchronous task queue
Also supports task scheduling
2.Celery architecture
Celery architecture consists of three parts, the middleware message (message broker), the task execution unit (worker) and task execution result storage (task result store) composition.
2.1 messaging middleware
Celery does not provide messaging services, but can be easily and messaging middleware integration provided by third parties. Including, RabbitMQ, Redis, etc.
2.2 task execution unit
Worker Tasks performed Celery unit is provided, worker operating in a distributed concurrent system node.
2.3 Tasks results are stored
Task result store to store task execution Worker result, Celery support results in different ways to store tasks, including AMQP, redis, etc.
Support for version 2.4
Celery version 4.0 runs on
Python ❨2.7, 3.4, 3.5❩
PyPy ❨5.4, 5.5❩
This is the last version to support Python 2.7, and from the next version (Celery 5.x) Python 3.5 or newer is required.
If you’re running an older version of Python, you need to be running an older version of Celery:
Python 2.6: Celery series 3.1 or earlier.
Python 2.5: Celery series 3.0 or earlier.
Python 2.4 was Celery series 2.2 or earlier.
Celery is a project with minimal funding, so we don’t support Microsoft Windows. Please don’t open any issues related to that platform.3. Use the scene
Asynchronous task: the task to submit time-consuming operation to asynchronous execution Celery, such as sending SMS / e-mail, push messaging, audio and video processing, etc.
Regular tasks: the timing of implementation of something, such as statistics day
4.Celery Installation and Configuration
pip install celery
Messaging middleware: RabbitMQ / Redis
app = Celery ( 'task name', backend = 'xxx', broker = 'xxx')
5.Celery perform asynchronous tasks
5.1 Basic use
Create a project celerytest
Creating py file: celery_app_task.py
import celery
import time
# broker='redis://127.0.0.1:6379/2' 不加密码
backend='redis://:[email protected]:6379/1'
broker='redis://:[email protected]:6379/2'
cel=celery.Celery('test',backend=backend,broker=broker)
@cel.task
def add(x,y):
return x+yCreating py file: add_task.py, add a task
from celery_app_task import add
result = add.delay(4,5)
print(result.id)Creating py file: run.py, perform tasks, or use the command: celery worker -A celery_app_task -l info
注：windows下：celery worker -A celery_app_task -l info -P eventlet
from celery_app_task import cel
if __name__ == '__main__':
cel.worker_main()
# cel.worker_main(argv=['--loglevel=info')Creating py file: result.py, view task execution results
from celery.result import AsyncResult
from celery_app_task import cel
async = AsyncResult(id="e919d97d-2938-4d0f-9265-fd8237dc2aa3", app=cel)
if async.successful():
result = async.get()
print(result)
# result.forget() # 将结果删除
elif async.failed():
print('执行失败')
elif async.status == 'PENDING':
print('任务等待中被执行')
elif async.status == 'RETRY':
print('任务异常后正在重试')
elif async.status == 'STARTED':
print('任务已经开始被执行')Execution add_task.py, add tasks, and get the task ID
Performing run.py, or execute the command: celery worker -A celery_app_task -l info
Execution result.py, check job status and obtain results
More than 5.2 task structure
pro_cel
├── celery_task# celery相关文件夹
│ ├── celery.py # celery连接和配置相关文件,必须叫这个名字
│ └── tasks1.py # 所有任务函数
│ └── tasks2.py # 所有任务函数
├── check_result.py # 检查结果
└── send_task.py # 触发任务celery.py
from celery import Celery
cel = Celery('celery_demo',
broker='redis://127.0.0.1:6379/1',
backend='redis://127.0.0.1:6379/2',
# 包含以下两个任务文件，去相应的py文件中找任务，对多个任务做分类
include=['celery_task.tasks1',
'celery_task.tasks2'
])
# 时区
cel.conf.timezone = 'Asia/Shanghai'
# 是否使用UTC
cel.conf.enable_utc = Falsetasks1.py
import time
from celery_task.celery import cel
@cel.task
def test_celery(res):
time.sleep(5)
return "test_celery任务结果:%s"%restasks2.py
import time
from celery_task.celery import cel
@cel.task
def test_celery2(res):
time.sleep(5)
return "test_celery2任务结果:%s"%rescheck_result.py
from celery.result import AsyncResult
from celery_task.celery import cel
async = AsyncResult(id="08eb2778-24e1-44e4-a54b-56990b3519ef", app=cel)
if async.successful():
result = async.get()
print(result)
# result.forget() # 将结果删除,执行完成，结果不会自动删除
# async.revoke(terminate=True) # 无论现在是什么时候，都要终止
# async.revoke(terminate=False) # 如果任务还没有开始执行呢，那么就可以终止。
elif async.failed():
print('执行失败')
elif async.status == 'PENDING':
print('任务等待中被执行')
elif async.status == 'RETRY':
print('任务异常后正在重试')
elif async.status == 'STARTED':
print('任务已经开始被执行')send_task.py
from celery_task.tasks1 import test_celery
from celery_task.tasks2 import test_celery2
# 立即告知celery去执行test_celery任务，并传入一个参数
result = test_celery.delay('第一个的执行')
print(result.id)
result = test_celery2.delay('第二个的执行')
print(result.id)Add tasks (execution send_task.py), open work: celery worker -A celery_task -l info -P eventlet, check the task execution results (execution check_result.py)
6.Celery perform regular tasks
6.1 set the time to perform a task celery
add_task.py
from celery_app_task import add
from datetime import datetime
# 方式一
# v1 = datetime(2019, 2, 13, 18, 19, 56)
# print(v1)
# v2 = datetime.utcfromtimestamp(v1.timestamp())
# print(v2)
# result = add.apply_async(args=[1, 3], eta=v2)
# print(result.id)
# 方式二
ctime = datetime.now()
# 默认用utc时间
utc_ctime = datetime.utcfromtimestamp(ctime.timestamp())
from datetime import timedelta
time_delay = timedelta(seconds=10) # 10秒
task_time = utc_ctime + time_delay # utc时间 + 延时时间10秒
# 使用apply_async并设定时间
result = add.apply_async(args=[4, 3], eta=task_time)
print(result.id)(2) similar to the timing of tasks contab
Multitasking structure celery.py amended as follows
from datetime import timedelta
from celery import Celery
from celery.schedules import crontab
cel = Celery('tasks', broker='redis://127.0.0.1:6379/1', backend='redis://127.0.0.1:6379/2', include=[
'celery_task.tasks1',
'celery_task.tasks2',
])
cel.conf.timezone = 'Asia/Shanghai'
cel.conf.enable_utc = False
cel.conf.beat_schedule = {
# 名字随意命名
'add-every-10-seconds': {
# 执行tasks1下的test_celery函数
'task': 'celery_task.tasks1.test_celery',
# 每隔2秒执行一次
# 'schedule': 1.0,
# 'schedule': crontab(minute="*/1"),
'schedule': timedelta(seconds=2),
# 传递参数
'args': ('test',)
},
# 'add-every-12-seconds': {
# 'task': 'celery_task.tasks1.test_celery',
# 每年4月11号，8点42分执行
# 'schedule': crontab(minute=42, hour=8, day_of_month=11, month_of_year=4),
# 'schedule': crontab(minute=42, hour=8, day_of_month=11, month_of_year=4),
# 'args': (16, 16)
# },
}Start a beat: celery beat -A celery_task -l info
Start work execution: celery worker -A celery_task -l info -P eventlet
7.Django use Celery
Installation package
celery==3.1.25
django-celery==3.1.207.1 celeryconfig.py created under the project directory
import djcelery
djcelery.setup_loader()
CELERY_IMPORTS=(
'app01.tasks',
)
#有些情况可以防止死锁
CELERYD_FORCE_EXECV=True
# 设置并发worker数量
CELERYD_CONCURRENCY=4
#允许重试
CELERY_ACKS_LATE=True
# 每个worker最多执行100个任务被销毁，可以防止内存泄漏
CELERYD_MAX_TASKS_PER_CHILD=100
# 超时时间
CELERYD_TASK_TIME_LIMIT=12*30In app01 directory created tasks.py
from celery import task
@task
def add(a,b):
with open('a.text', 'a', encoding='utf-8') as f:
f.write('a')
print(a+b)View function views.py
from django.shortcuts import render,HttpResponse
from app01.tasks import add
from datetime import datetime
def test(request):
# result=add.delay(2,3)
ctime = datetime.now()
# 默认用utc时间
utc_ctime = datetime.utcfromtimestamp(ctime.timestamp())
from datetime import timedelta
time_delay = timedelta(seconds=5)
task_time = utc_ctime + time_delay
result = add.apply_async(args=[4, 3], eta=task_time)
print(result.id)
return HttpResponse('ok')settings.py
INSTALLED_APPS = [
...
'djcelery',
'app01'
]
...
from djagocele import celeryconfig
BROKER_BACKEND='redis'
BOOKER_URL='redis://127.0.0.1:6379/1'
CELERY_RESULT_BACKEND='redis://127.0.0.1:6379/2'