celery task - 2

# celery task

Foreword

Discuss a timed task, in general, it requires the following functions:

1. Packaging an object, independent execution;
   
2. Some object has an interface, to facilitate understanding of its state;
3. Regularly call;
4. Behavior control, including retry, success / failure callbacks and so on;

The following describes the celery realize these functions.

1.task basic

The base class is the task celery tasks.Task ()

1.1 bound tasks

The first parameter represents the default binding is self

logger = get_task_logger(__name__)

@task(bind=True)
def add(self, x, y):
    logger.info(self.request.id)

1.2 Task class inheritance

Note that the position statement, statement when the method is modified into the Task class.

@app.task(base=MyTask)
def add(x, y):
    #raise KeyError
    return x + y

1.3 names

Each task has a distinct instance of the name, such as the following example:

@app.task(name='tasks.mul')

def mul(x, y):

Generally unnecessary to use this feature, especially when the task is consigned to a separate module, the default name is the module name + method name (celery_tasks.mul).
Try not to make the task module named tasks.py, named celery_1.py better.

1.4 Other properties

Task.max_retries maximum number of retries
Task.default_retry_delay default retry wait time.
Task.ignore_result abandon result, it means you can not view the results by AsyncResult.

2.task custom task behavior

Documentation: http: //docs.celeryproject.org/en/latest/userguide/tasks.html#custom-task-classes

There are four, including the failure / success / Retry / completed
on_failure on_success on_retry after_return

# celery_tasks.py
class MyTask(Task):
    def on_success(self, retval, task_id, args, kwargs):
        print 'task done: {0}'.format(retval)
        return super(MyTask, self).on_success(retval, task_id, args, kwargs)
    
    def on_failure(self, exc, task_id, args, kwargs, einfo):
        print 'task fail, reason: {0}'.format(exc)
        return super(MyTask, self).on_failure(exc, task_id, args, kwargs, einfo)

@app.task(base=MyTask)
def add(x, y):
    return x + y

2.1 retry

app.Task.retry () method is implemented retry.

# an example of using retry:
@app.task(bind=True)
def send_twitter_status(self, oauth, tweet):
    try:
        twitter = Twitter(oauth)
        twitter.update_status(tweet)
    except (Twitter.FailWhaleError, Twitter.LoginError) as exc:
        raise self.retry(exc=exc)

Retry interval time can be specified, the default is 180 seconds to 1800 seconds following designated cases.

@app.task(bind=True, default_retry_delay=30 * 60) # retry in 30 minutes.

def add(self, x, y):

It is designated on the maximum number of retries parameter task instance.

3. The timing of periodic task execution

Documentation: http: //docs.celeryproject.org/en/latest/userguide/periodic-tasks.html

celery beat is a scheduler。

The default parameters from beat_schedule.

3.1 Time Zone

The scheduler default UTC time zone, of course, need to be modified:
TimeZone = 'Europe / London'
or app.conf.timezone = 'Europe / London'

3.2 entries

There are two ways to add a scheduled task, decorators and configuration files.
Common profile way.

3.2.1 read timing task from a configuration file

celery_config.py # Profiles

!/usr/bin/env python
coding:utf-8
"""

celery configure
"""
author = 'sss'
from future import absolute_import

from celery.schedules import crontab

from datetime import timedelta
Redis storage and use of the results of the task queue
broker_url = 'redis://:[email protected]:6379/0'

result_backend = 'redis://:[email protected]:6379/1'
task_serializer = 'json'

result_serializer = 'json'

accept_content = ['json']
Time zone
timezone = 'Asia/Shanghai'
celery default on their log
False representation is not closed
worker_hijack_root_logger = False
Store the results expiration time expired automatically deleted
Seconds
result_expires = 60 * 60 * 24
Import the file containing the task
imports = [

'celery_tasks',]
Timing Task Configuration
= {beat_schedule 

'test1': { 

# need to perform the specific function 

# This function must be used @ app.task decorative 

'Task': 'celery_tasks.test1_run', 

# regular time 

# performed once per minute, not as a decimal number 

'schedule' : crontab (minute = ' / 1'), 

# or so written, performed once per hour 

# "Schedule": crontab (= 0 minute, hour = " / 1") 

# function parameters required for execution of the 

'args': ( ) 

}, 

'test2': { 

'Task': 'celery_tasks.test2_run', 

# set the timing of time 10 seconds 

'Schedule': timedelta (seconds the = 10), 

'args': () 

}}

celery_workder.py # master file

from future import absolute_import

Implicit refusal to introduce, if celery.py and celery module name, to avoid conflict, need to add this statement
The code, the name is not the same, the best have not the same
celery test -- worker
from celery import Celery
def create_worker():

# app = Celery('tasks', broker=di)

'''app = Celery('tasks',

#backend=di_backend,

broker=di_broker,

include=['celery_tasks'])

'''

app = Celery()

#app.conf.update(result_expires=3600,)

app.config_from_object('celery_config')

return app
app = create_worker()

celery_tasks.py # task method files

from celery_worker import app

from celery.task import Task

import time
tasks.py
class MyTask(Task):

def on_success(self, retval, task_id, args, kwargs):

print('task done: {0}'.format(retval))

return super(MyTask, self).on_success(retval, task_id, args, kwargs)
def on_failure(self, exc, task_id, args, kwargs, einfo):
    print('task fail, reason: {0}'.format(exc))
    return super(MyTask, self).on_failure(exc, task_id, args, kwargs, einfo)
@app.task(base=MyTask)

def add(x, y):

#raise KeyError

print('wwww')

return x + y
@app.task

def mul(x, y):

return x * y
@app.task

def xsum(numbers):

return sum(numbers)
@app.task

def test(arg):

print(arg)
def test11():

time.sleep(1)

print('test11')
....slightly

Run - publishing tasks
celery -A celery_worker beat

Run - mission
celery -A celery_worker -l info -P eventlet

3.2.2 decorator add tasks - dynamically add

Case presented in the last chapter is written in the configuration file parameters beat_schedule;
sometimes this is not easy, need to be more flexible to add a scheduled task;

setup_periodic_tasks DEF (SENDER, kwargs **): 
    # 5s per call test ( 'Hello') 
    sender.add_periodic_task (5.0, test.s ( 'Hello'), name = 'Every the Add. 5') 

    # 20s each call test ( ' World ') 
    sender.add_periodic_task (10.0, test.s (' World '), Expires =. 7) 

    # a week 7:30 am performing Test (' Happy Mondays! ') 
    sender.add_periodic_task ( 
        the crontab (=. 7 hour, minute = 30, day_of_week = 1), # modify flexibility 
        test.s ( 'Happy Mondays!'), 
    ) 


setup_periodic_tasks (App) 
Print ( 'timer task list', app.conf.beat_schedule)

Run celery -A celery_test beat -l debug

You can compare the output timing of the task list.
Do not add a task:

(vir_venv) E: \ python \ code_2> celery -A celery_test beat -l debug 

timer task list} { 

Celery Beat v4.3.0 (rhubarb) Starting IS.

After adding tasks:

(vir_venv) E:\python\code_2>celery -A celery_test beat -l debug

定时任务列表 # 已格式化

{

"add every 5": {

"schedule": 5.0,

"task": "celery_tasks.test",

"args": [

"hello"

],

"kwargs": {},

"options": {}

},

"celery_tasks.test('world')": {

"schedule": 10.0,

"task": "celery_tasks.test",

"args": [

"world"

],

"kwargs": {},

"options": {

"expires": 7

}

},

"celery_tasks.test('Happy Mondays!')": {

"schedule": "<crontab: 30 7 1 * * (m/h/d/dM/MY)>",

"task": "celery_tasks.test",

"args": [

"HappyMondays!"

],

"kwargs": {},

"options": {}

}

}

相关启动命令：
celery -A celery_worker worker -l info -P eventlet --logfile=c.log
celery -A celery_test beat -l debug

4. celery related commands

Publishing tasks
celery -A celery_task beat

Mission
celery -A celery_task worker -l info -P eventlet

Background celery worker process starts
celery multi start work_1 -A appcelery

Stop worker process, if not stop, plus -A
Celery Multi STOP WORKNAME

Restart worker processes
celery multi restart WORKNAME

View the number of processes
celery status -A celery_task

The specified time format

Complex timing functions can use the crontab feature that comes with linux crontab format supported is the same, very easy to customize the task execution time.

from celery.schedules import crontab
 
app.conf.beat_schedule = {
    # Executes every Monday morning at 7:30 a.m.
    'add-every-monday-morning': {
        'task': 'tasks.add',
        'schedule': crontab(hour=7, minute=30, day_of_week=1),
        'args': (16, 16),
    },
}

The above case is to perform tasks in each tasks.add 7:30 Monday.