(Rpm) using Django django-celery and celery

I. Introduction

Django is a more popular Web framework under the python language, more and more companies and developers using Django implement their own Web server. In the Web Server development process, sometimes we just want to implement simple logic interactive Web server and the client, but also to achieve some regular tasks. Examples include the following:　　

1. Regularly delete or record cache Redis database
   in order to pursue higher performance database access, I Redis as a cache MySql database. Often the data accessed in Redis then Mysql in the timing storage. And the expiration of Redis data deleted. So this time, we need the timing to complete this task.

2. Generate reports 　　
   analogy, you have a Web server electricity supplier, every day users will shop at the top. In order to easily count the amount of consumption per user per month, you designed a monthly statistical reports in the database. Then use regular tasks, statistics on the 1st of each month, searchable database, to calculate the amount of consumption of each user last month, the store each monthly statistical report. , Which is the No. 1 mentioned earlier month then this mission report is time to complete.

3. Sending a message
   Another example: When your birthday coming website user, you wish on his birthday, to the user's mailbox to send Happy Birthday wishes. So this is a scheduled task to achieve.

The above example, the timing tasks are needed. In Python, we use Celery module to complete this task. Celery on the network on a lot of Bowen, Bowen most logical bit confusing, so there was this blog. I hope readers have a clear understanding of reading, and good combat it, there is no longer any confusion on celery regular tasks, this is my original intention of this blog.

The blog post did not introduce the principle Celery, such as Broker, Worker and so on. Before combat, these concepts must be understood clearly. As the Internet has been a lot of such content, I posted some reference documentation at the end of the article to allow readers to learn.

---

二.Celery,Django和Djcelery

It is always clear:
Celery is a third-party Python library that can be used in any of the Python project, because we can always put Celery as a separate module to manipulate other modules. Therefore, we can use the Django project Celery, but it is worth noting that the use of Celery in Django two ways:

1. Use only Celery.
2. Use Celery + djcelery.

Method 1 : Django equivalent of adding the task script Celery, in order to manipulate Django, Django environment and therefore require additional configuration in Celery in order to operate Django's database.

Method 2 : Due to the use djcelery, can easily operate directly in the task Django database, and the final task to view and modify the related tasks in Django backend database.
Selecting two methods:
from the above description see Method 1 Method 2 less than djcelery introducing a module, needs its own disadvantage is arranged in conjunction with Django environment. The method, more convenient peace of mind, but also to manage their own tasks in the background Django. So if you use Celery with Django, I strongly recommend Method 2.

The blog post describes the use of 2, but they both are essentially the same. I have practiced in two ways, I had originally planned to write a companion piece on Bowen (about 1), and want a good name, called "Django Celery is used to achieve timing tasks (not djcelery)". However, due later also wrote that no one, too lazy to update, together with other factors and then leave alienated Django back-end, so the companion plan has been shelved. This article was originally published in CSDN above, subsequent received some good feedback, this blog will update once again. (However, the companion program was or was I gave up).

Chat time has ended, the following is the text.

---

Three. Django directory structure

First, remember: pip install django-celery

The following example shows the directory structure of a Django project:

• app
  　 - admin.py
  　 - views.py
  　 - urls.py
  　 - models.py
  　 - tasks.py
• pro
  　 - settings.py
  　 - urls.py
  　 - urls.py
  　 - models.py
• manage.py

Note that the above directory tasks.pyfile is my new, placed in the directory app under the whole Celery task, I just built this one file.

IV. Configuring setting.py

In order to set Celery, we need to configure setting.py file, as follows:

1. Add djcelery

INSTALLED_APPS = (
    'django.contrib.admin',
    'django.contrib.auth',
    'djcelery', #此处是新加入的djcelery
    'app', )

Above  INSTALLED_APPS, I omitted a module unrelated, adding attention djceleryto. 　

After adding djcelery need to implement, migrate migration, generate data tables.
　

2. Set parameters celery

I'm at the end of setting.py file, adding the following parameter configuration of celery, first posted the code to explain.

import djcelery
djcelery.setup_loader()
BROKER_URL = 'redis://127.0.0.1:6379/6'
CELERY_IMPORTS = ('app.tasks', )
CELERY_TIMEZONE = TIME_ZONE
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'

# 下面是定时任务的设置，我一共配置了三个定时任务. from celery.schedules import crontab CELERYBEAT_SCHEDULE = { #定时任务一：　每24小时周期执行任务(del_redis_data) u'删除过期的redis数据': { "task": "app.tasks.del_redis_data", "schedule": crontab(hour='*/24'), "args": (), }, #定时任务二:　每天的凌晨12:30分，执行任务(back_up1) u'生成日报表': { 'task': 'app.tasks.back_up1', 'schedule': crontab(minute=30, hour=0), "args": () }, #定时任务三:每个月的１号的6:00启动，执行任务(back_up2) u'生成统计报表': { 'task': 'app.tasks.back_up2', 'schedule': crontab(hour=6, minute=0, day_of_month='1'), "args": () }, }

The code Interpretation:

When djcelery.setup_loader () run-time, Celery will go to view all files tasks.py app directory INSTALLD_APPS included, find the mark for the task approach, they will be registered as celery task.

BROKER_URL = 'redis://127.0.0.1:6379/6'

broker is an agent, which is responsible for distributing tasks to the worker to perform. I am using Redis as a broker, of course, you can also use other broker, such as the official is more recommended to use RabbitMQ.

Some mentioned the need to configure the blog Keywords: CELERY_RESULT_BACKENDfor example:

CELERY_RESULT_BACKEND = 'amqp://guest@localhost//'　#可以不用写

I do not have to configure this keyword. Because if not configured, then Django will use the default database (you also specify the database orm), as its result as its backend. So you can not write, Django database using default settings is very good.

CELERY_IMPORTS = ('app.tasks', )
CELERY_TIMEZONE = TIME_ZONE
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'

The above is the introduction of a certain task file, the second sentence is set zone, the third sentence indication of the default database django-celery scheduling model, task execution period are specified by default orm presence database.

Celery more in-depth configuration: (http://www.cnblogs.com/ajianbeyourself/p/4950758.html)

from celery.schedules import crontab
CELERYBEAT_SCHEDULE = {
    #定时任务一：　每24小时周期执行任务(del_redis_data)
    u'删除过期的redis数据': {
        "task": "app.tasks.del_redis_data", "schedule": crontab(hour='*/24'), "args": (), },

The above configuration is to set the timing of the time, on crontabthe specific usage, celery official document to explain in great detail (table):

http://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html

I chose three tasks, I deliberately picked, very representative. The first is a periodic task, it will be to perform a respective task every fixed period of time, such as every 1 minute every 1 hour, etc.; the second and third tasks are timed, the timing at each time point , such as 6:00 per day, or 1 month timing.
Periodic tasks and timing tasks small difference, which is the power of the crontab, it supports both.

5.Tasks task

Each task is essentially a function in tasks.py, the write function you want to perform. My tasks.py as follows: each task I wrote long and boring, so the specific details will be omitted.

# coding=utf-8
from celery import task

@task()
def del_redis_data(): # 此处是一个删除redis数据的操作。具体略过 @task() def back_up1(): # 此处是一个备份到日报表的操作。具体略过 @task() def back_up2(): # 此处是一个生成统计报表的操作。具体略过

If the reader needs to implement a timed task yourself, then the above-mentioned task is bound to have to define a function, I can only provide a reference. My top three tasks, corresponding to the setting.pyfiles CELERYBEAT_SCHEDULEof the three timer configuration.

Keep in mind that the task is just a function when the function call, depending on your setting.pyconfiguration.

6. Start a scheduled task

Django log in to the background, you can see the back-end database revealed that task parameters, renderings temporarily slightly.
Then start terminal, switching to the root directory Django projects, run:

python manage.py runserver    # 启动web服务
python manage.py celery worker -l info   # 启动celery woker

This command is used to start a thread to perform tasks on the worker, worker essence, is a drudge workers.

python manage.py celery beat -l info  # 启动beat, 执行定时任务.（原博客在此处有错误，已经修改。提示：最新的django-celery匹配celery<4.0,>=3.1.15版本。）

Above this task is to enable beater, it's like a leader, responsible for the tasks distributed to workers.

To direct, this blog is basically over, due to the two updates a long interval of time, the initial idea can remember. Of course, you certainly have an important question, and that is if the task because the system is rebooted or other causes crashes how to restart it? For linux systems, supervisor can host these two commands, if the task is not up and running, it can restart start their own task, so that we can guarantee the regular tasks on the server side for some reason does not crash out.

7. Recommended Articles

1. Celery more in-depth configuration: (http://www.cnblogs.com/ajianbeyourself/p/4950758.html)