Linux system management large jobs: using Celery to realize distributed computing

Zero, preliminary knowledge

1 Celery Basics

  Celery is an asynchronous task queue implemented in Python/a job queue based on distributed messaging.
  Celery's architecture consists of three parts:
    message middleware (broker),
    task execution unit (worker),
    task execution result storage (result)

  Using celery includes three aspects:
    1. Define task functions.
    2. Run the Celery service.
    3. Invocation of the client application.

2 Environment of this paper

  There are three main aspects
    Two Ubuntu 18.04
    Python3
    pip3

3 Steps implemented in this article

  1. Configure Node1 (send tasks and middleware)
    1. Install and configure to start Redis (as middleware and store results)
    2. Install Celery
  2. Configure Node2 (used to receive and complete tasks)
    1. Install Redis (no configuration required)
    2. Install Celery
  3. Write Python programs
    1. Write programs on Worker
    2. Write programs that issue tasks to middleware
  4. Run

---

---

Below is the text

---

---

1. Configure node1 (send tasks and middleware)

1. Install and configure to start Redis

1.1 Install Redis and python library

stfnode1@ubuntu:~$ sudo apt-get install redis-server
stfnode1@ubuntu:~$ pip3 install redis

1.2 Configure Redis

Pay attention to the notes

#通过这个命令获取node1的IP地址，并且记住
stfnode1@ubuntu:~$ ifconfig 
#我的IP是192.168.30.139
stfnode1@ubuntu:~$ sudo vim /etc/redis/redis.conf
#找到bind 127.0.0.1 ::1在69行左右
#将127.0.0.1 改为自己的IP地址
# ::1 删掉
#其余地方不用改动,直接保存退出

1.3 Start Redis with configuration file

sudo redis-server /etc/redis/redis.conf

 

2. Install Celery

sudo apt install python-celery-common

At this point, Celery is installed
 

---

2. Configure Node2 (used to receive and complete tasks)

1. Install Redis and python library (no configuration required)

stfnode2@ubuntu:~$ sudo apt-get install redis-server
stfnode1@ubuntu:~$ pip3 install redis

2. Install Celery

stfnode2@ubuntu:~$ sudo apt install python-celery-common
stfnode2@ubuntu:~$ pip3 install celery[redis]

 

---

At this point, the environment should be configured Next, start writing Python programs↓

---

3. Write Python programs

1. Write the program on node1

①Create a new tasks.py, the content is as follows:

from celery import Celery
# 注意: 这里*.*.*.*替换成配置文件中修改的地址即可
app = Celery('tasks',broker='redis://*.*.*.*:6379',backend='redis://*.*.*.*:6379')

@app.task
def add(x, y):
    return x + y

②Create a new Runtasks.py, the content is as follows:

from tasks import add
from celery.result import AsyncResult
import redis
import time

a = int(input("input first number:"))
b = int(input("input second number:"))

id_my=add.delay(a,b)

time.sleep(1)

res=AsyncResult(str(id_my)) # 参数为task id
print(str(id_my))
print(res.result)

Note: tasks.py and Runtasks.py here should be in the same directory (that is, folder)
Note: remember to replace *.* .*.* in tasks.py with the address modified in the configuration file
Note: It is recommended not to exit Current directory

 

2. Write the program on node2

As a worker, node2 always monitors the middleware publishing tasks on node1, and returns the data to node1's redis after completing the tasks.
Create a new tasks.py with the following content:

from celery import Celery
# broker设置中间件，backend设置后端存储
app = Celery('tasks',broker='redis://*.*.*.*:6379',backend='redis://*.*.*.*:6379')
@app.task
def add(x,y):
  return x+y

Note: It is recommended not to exit the current directory!
Note: Remember to replace *.* .*.* in tasks.py with the address modified in the configuration file

Four, run

1. Run the task on celery on node2

  Run the following command in the directory where tasks.py is located. At this moment, celery on node2 will always monitor the tasks assigned by the middleware on node1.

celery -A tasks worker --loglevel=info

 The result is as follows

2. node1 release task

  Run the following command in the directory where tasks.py and Runtask.py are located, run Runtask.py

python3 Runtasks.py

3. View the results

  At this moment, you can see the result on node2, as shown in the figure below

---

reference article

1. Under Ubuntu16.04, erlang installation and rabbitmq installation steps
2. Python Parallel Distributed Framework Celery - Catch the thief first
3. Celery Preliminary - Official Documentation
4. Using RabbitMQ-Celery 3.1 Official Documentation
5. Detailed explanation of Python parallel distributed framework Celery (this is the next step)
6. Turn on the redis-server prompt # Creating Server TCP listening socket *:6379: bind: Address already in use–solution
7. python+celery报错：consumer: Cannot connect to redis://127.0.0.1:6379/10: Error 111 connecting
8. Distributed on multiple cloud servers based on Celery
9. Celery | Flask Error: Expected byteslike object, but found AsyncResult
10. Celery gets task execution results
11. How to view the results stored in redis by Celery?
12. python – redis connection and use
13. Several ways to implement delay operation in Python
14. Several ways of concatenating strings in Python
15. python data storage excel
16. modulenotfounderror: no module named 'xlwt' solution python padans output excel error

---

Ability is limited, please point out any mistakes