Celery 项目架构和使用 systemctl 正式版部署

基本实例

项目layout

test/__init__.py
    /celery_app.py
    /tasks.py
    /t1.py

celery_app.py（保存celery应用）

from celery import Celery

app = Celery(
    'celery_app',
    broker='redis://localhost:6379/12',
    backend='redis://localhost:6379/13',
    include=['tasks']
)

if __name__ == '__main__':
    app.start()

tasks.py（编写需要的task）

from celery_app import app


@app.task
def add(x, y):
    return x + y


@app.task
def mul(x, y):
    return x * y

t1.py（执行task）

import tasks

tasks.add.delay(1, 5)

启动，关闭，重启

执行命令

# 启动
$ celery multi start anyname -A yourapp -l info --pidfile=/var/celery/run/%n.pid --logfile=/var/celery/log/%n%I.log

# 启动2个worker
$ celery multi start 2 -A yourapp -l info

# 停止
$ celery multi stop anyname -A yourapp -l info

# 确保所有任务完成后停止
$ celery multi stopwait anyname -A yourapp -l info

针对上面的示例，可以使用如下命令：

# 启动2个worker
$ celery multi start 2 -A celery_app -l info --pidfile=/var/celery/run/%n.pid --logfile=/var/celery/log/%n%I.log

# 关闭
$ celery multi stop 2 -A celery_app -l info --pidfile=/var/celery/run/%n.pid --logfile=/var/celery/log/%n%I.log

# 重启
$ celery multi restart 2 -A celery_app -l info --pidfile=/var/celery/run/%n.pid --logfile=/var/celery/log/%n%I.log

使用systemctl部署

创建日志和pid目录

$ sudo mkdir -p /var/celery/log
$ sudo mkdir -p /var/celery/run
$ sudo chown -R $USER:$USER /var/celery/log
$ sudo chown -R $USER:$USER /var/celery/run

创建celery.conf

$ vim celery.conf
""
# See
# http://docs.celeryproject.org/en/latest/tutorials/daemonizing.html#available-options

CELERY_APP="ccnu_resource"
CELERYD_NODES=2
CELERYD_OPTS=""
CELERY_BIN="/vagrant/gongku/env/bin/celery"
CELERYD_PID_FILE="/var/celery/run/%n.pid"
CELERYD_LOG_FILE="/var/celery/log/%n.log"
CELERYD_LOG_LEVEL="INFO"
""

创建system.services

$ sudo vim /etc/systemd/system/celery.service
""
[Unit]
Description=Celery Service
After=network.target

[Service]
Type=forking
User=vagrant
Group=vagrant
EnvironmentFile=/vagrant/gongku/celery.conf
WorkingDirectory=/vagrant/gongku

ExecStart=/bin/sh -c '${CELERY_BIN} multi start ${CELERYD_NODES} \
  -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
  --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'

ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait ${CELERYD_NODES} \
  --pidfile=${CELERYD_PID_FILE}'

ExecReload=/bin/sh -c '${CELERY_BIN} multi restart ${CELERYD_NODES} \
  -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
  --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'

[Install]
WantedBy=multi-user.target
""

启动systemctl

$ sudo systemctl daemon-reload
$ sudo systemctl enable celery.service
$ sudo systemctl start celery.service 
$ sudo systemctl stop celery.service 
$ sudo systemctl restart celery.service