Python中分布式进程原理

分布式进程是指将Process进程分布到多台机器中,充分利用多台机器的性能完成复杂的任务.

multiprocessing中的子模块managers支持吧多进程分布到多台机器上.

 实现思想:写一个服务端进程作为调度者,将任务分布到其他多个进程中,依靠网络通信进行管理.!

服务端核心代码:

import random,time,queue
from multiprocessing.managers import BaseManager
task_queue = queue.Queue()
result_queue = queue.Queue()

class Queuemanager(BaseManager):
    pass

Queuemanager.register('get_task_queue',callable=lambda :task_queue)
Queuemanager.register('get_result_queue',callable=lambda :result_queue)
a = 'qiye'.encode('utf-8')
manager = Queuemanager(address=('127.0.0.1',8001),authkey=a)

manager.start()

task = manager.get_task_queue()
result = manager.get_result_queue()

for url in ["ImageUrl_"+str(i) for i in range(10)]:
    print('put task %s...'% url)
    task.put(url)

print('try get result...')

for i in range(10):
    print('result is %s'%result.get(timeout=10))

任务端核心代码

import time
from multiprocessing.managers import BaseManager
class QueueManager(BaseManager):
    pass

QueueManager.register('get_task_queue')
QueueManager.register('get_result_queue')
server_addr = '127.0.0.1'
print('connect to server %s...'%server_addr)
a = 'qiye'.encode('utf-8')
m = QueueManager(address=(server_addr,8001), authkey=a)
m.connect()


task = m.get_task_queue()
result = m.get_result_queue()
print(task)
while(not task.empty()):
    image_url = task.get(True,timeout=5)
    print('run task download %s.....'%image_url)
    time.sleep(1)
    result.put('%s----->sucess'%image_url)
print('worker exit')