Python study notes, special records, share with you, I hope it will be helpful to everyone.
Distributed process
In Thread and Process, Process should be preferred, because Process is more stable, and Process can be distributed to multiple machines, while Thread can only be distributed to multiple CPUs of the same machine.
Python's multiprocessing module not only supports multiple processes, but the managers sub-module also supports the distribution of multiple processes to multiple machines. A service process can act as a scheduler, distribute tasks to multiple other processes, and rely on network communication. Because the managers module is well encapsulated, it is easy to write distributed multi-process programs without knowing the details of network communication.
For example: if we already have a multi-process program that communicates via Queue running on the same machine, now, because the process of processing tasks is heavy, we hope to distribute the process of sending tasks and the process of processing tasks to two machines . How to use distributed process to achieve?
The original Queue can continue to be used, but by exposing the Queue through the network through the managers module, the processes of other machines can access the Queue.
Let's look at the service process first. The service process is responsible for starting the Queue, registering the Queue on the network, and then writing tasks to the Queue:
# task_master.py
import random, time, queue
from multiprocessing.managers import BaseManager
# 发送任务的队列:
task_queue = queue.Queue()
# 接收结果的队列:
result_queue = queue.Queue()
# 从BaseManager继承的QueueManager:
class QueueManager(BaseManager):
pass
# 把两个Queue都注册到网络上, callable参数关联了Queue对象:
QueueManager.register('get_task_queue', callable=lambda: task_queue)
QueueManager.register('get_result_queue', callable=lambda: result_queue)
# 绑定端口5000, 设置验证码'abc':
manager = QueueManager(address=('', 5000), authkey=b'abc')
# 启动Queue:
manager.start()
# 获得通过网络访问的Queue对象:
task = manager.get_task_queue()
result = manager.get_result_queue()
# 放几个任务进去:
for i in range(10):
n = random.randint(0, 10000)
print('Put task %d...' % n)
task.put(n)
# 从result队列读取结果:
print('Try get results...')
for i in range(10):
r = result.get(timeout=10)
print('Result: %s' % r)
# 关闭:
manager.shutdown()
print('master exit.')
Please note that when we write a multi-process program on a machine, the created Queue can be used directly. However, in a distributed multi-process environment, adding tasks to the Queue cannot directly operate on the original task_queue. It bypasses the encapsulation of QueueManager and must be added through the Queue interface obtained by manager.get_task_queue().
Then, start the task process on another machine (starting on this machine is also possible):
# task_worker.py
import time, sys, queue
from multiprocessing.managers import BaseManager
# 创建类似的QueueManager:
class QueueManager(BaseManager):
pass
# 由于这个QueueManager只从网络上获取Queue，所以注册时只提供名字:
QueueManager.register('get_task_queue')
QueueManager.register('get_result_queue')
# 连接到服务器，也就是运行task_master.py的机器:
server_addr = '127.0.0.1'
print('Connect to server %s...' % server_addr)
# 端口和验证码注意保持与task_master.py设置的完全一致:
m = QueueManager(address=(server_addr, 5000), authkey=b'abc')
# 从网络连接:
m.connect()
# 获取Queue的对象:
task = m.get_task_queue()
result = m.get_result_queue()
# 从task队列取任务,并把结果写入result队列:
for i in range(10):
try:
n = task.get(timeout=1)
print('run task %d * %d...' % (n, n))
r = '%d * %d = %d' % (n, n, n*n)
time.sleep(1)
result.put(r)
except Queue.Empty:
print('task queue is empty.')
# 处理结束:
print('worker exit.')
The task process needs to be connected to the service process through the network, so the IP of the service process must be specified.
Now, you can try the work effect of the distributed process. First start the task_master.py service process:
$ python3 task_master.py
Put task 3411...
Put task 1605...
Put task 1398...
Put task 4729...
Put task 5300...
Put task 7471...
Put task 68...
Put task 4219...
Put task 339...
Put task 7866...
Try get results...
After the task_master.py process finishes sending the task, it starts to wait for the result of the result queue. Now start the task_worker.py process:
$ python3 task_worker.py
Connect to server 127.0.0.1...
run task 3411 * 3411...
run task 1605 * 1605...
run task 1398 * 1398...
run task 4729 * 4729...
run task 5300 * 5300...
run task 7471 * 7471...
run task 68 * 68...
run task 4219 * 4219...
run task 339 * 339...
run task 7866 * 7866...
worker exit.
The task_worker.py process ends, and the results will continue to be printed in the task_master.py process:
Result: 3411 * 3411 = 11634921
Result: 1605 * 1605 = 2576025
Result: 1398 * 1398 = 1954404
Result: 4729 * 4729 = 22363441
Result: 5300 * 5300 = 28090000
Result: 7471 * 7471 = 55815841
Result: 68 * 68 = 4624
Result: 4219 * 4219 = 17799961
Result: 339 * 339 = 114921
Result: 7866 * 7866 = 61873956
What is the use of this simple Master/Worker model? In fact, this is a simple but true distributed computing. With a little modification of the code and starting multiple workers, tasks can be distributed to several or even dozens of machines. For example, the code for calculating n*n can be replaced by sending emails. , The asynchronous sending of the mail queue is realized.
Where are the Queue objects stored? Note that there is no code to create a Queue in task_worker.py, so the Queue object is stored in the task_master.py process:
The reason why Queue can be accessed through the network is achieved through QueueManager. Since the QueueManager manages more than one Queue, a name must be given to the network call interface of each Queue, such as get_task_queue.
What is the use of authkey? This is to ensure that the two machines communicate normally and are not maliciously interfered by other machines. If the authkey of task_worker.py is inconsistent with the authkey of task_master.py, it will definitely fail to connect.
Welcome to pay attention to the public account "Web Development" , you can receive python test demo and learning resources, everyone learn python together, and collect the world's methods, which is convenient for you and me to develop .
I hope I can help you. If you have any questions, you can join the QQ technical exchange group: 668562416
If there is something wrong or insufficient, I also hope that readers can provide more comments or suggestions.
If you need to reprint, please contact me. You can reprint with authorization, thank you
Welcome to pay attention to the public account "Web Development"