Architecture tuning (a): What is distributed computing (distributed multi-process + + + multi-threaded multi-coroutines)

Copyright: Life program https://blog.csdn.net/cnpinpai/article/details/89294610

Tuning a framework: What is distributed computing (distributed multi-process + + + multi-threaded multi-coroutines)? The following soft alliance network Xiaobian first to cite a simple example:

The number 1-10000 each squaring

Server server:

Two queue stores tasks, the result

Define two functions

To achieve distributed entailed multiprocessing.managers.BaseManager

multiprocessing.freeze_support () support open distributed in the main function

Registration of two functions to client calls

Create a manager, set the ip address and open ports, link password.

Plus two queues tasks reception result. With function just registered

1-10000 pressed into the queue,

The result onto queue

The last complete shut down the server

Client client:

Also we need to inherit multiprocessing.managers.BaseManager

A definition of a data processing coroutine, and push the result into the result queue

Define a thread processing data 10, 10 open coroutine

Define a process, the process drive 10 threads

The main function: two functions with client registration

Client Manager to create the same set ip addresses and open ports, link password.

Linked Server

Calling the same client registration function, two queues

Sets of four cycles: 10 processes, threads 100, 1000 coroutine

Cycle process function

On the code:

Server server:

#coding:utf-8
import multiprocessing  #分布式进程
import multiprocessing.managers #分布式进程管理器
import random,time  #随机数，时间
import Queue #队列

task_queue=Queue.Queue() #任务
result_queue=Queue.Queue() #结果

def  return_task(): #返回任务队列
    return task_queue
def return_result(): #返回结果队列
    return   result_queue

class  QueueManger(multiprocessing.managers.BaseManager):#继承，进程管理共享数据
    pass

if __name__=="__main__":
    multiprocessing.freeze_support()#开启分布式支持
    QueueManger.register("get_task",callable=return_task)#注册函数给客户端调用
    QueueManger.register("get_result", callable=return_result)
    manger=QueueManger(address=("192.168.112.11",8848),authkey="123456") #创建一个管理器，设置地址与密码
    manger.start() #开启
    task,result=manger.get_task(),manger.get_result() #任务，结果
    for  i  in range(10000):
        print "task add data",i
        task.put(i)
    print "waitting for------"
    for  i  in range(10000):
        res=result.get(timeout=100)
        print "get data",res

    manger.shutdown()#关闭服务器

Client client:

#coding:utf-8
import multiprocessing  #分布式进程
import multiprocessing.managers  # 分布式进程管理器
import random,time  #随机数，时间
import Queue #队列
import threading
import gevent
import gevent.monkey


class  QueueManger(multiprocessing.managers.BaseManager):# 继承，进程管理共享数据
    pass
def  gevetygo(num ,result): #协程处理一个数据
    print num*num
    result.put(num*num)

def  threadgo(datalist,result): # 线程处理10个数据，开启10个协程
    tasklist=[]
    for  data  in datalist:
        tasklist.append(gevent.spawn(gevetygo, data,result))
    gevent.joinall(tasklist)

def  processgo(ddatalist,result): # [[1,2,3],[4,5,6]] 进程驱动了10个线程
    threadlist=[]
    for  datalist in ddatalist:
        mythread=threading.Thread(target=threadgo,args=(datalist,result))
        mythread.start()
        threadlist.append(mythread)
    for mythread in threadlist:
        mythread.join()

if __name__=="__main__":
    QueueManger.register("get_task")  # 注册函数调用服务器
    QueueManger.register("get_result")
    manger=QueueManger(address=("192.168.112.11",8848),authkey="123456")
    manger.connect()  # 链接服务器
    task= manger.get_task()
    result =manger.get_result()  # 任务，结果

    # 1000
    # 10个进程
    # 100个线程
    # 1000个协程

    for  i  in range(10):
        cubelist = []  # [[[1],[2]]]
        for j in range(10):
            arealist = []
            for k in range(10):
                linelist = []
                for l in range(10):
                    data = task.get()
                    linelist.append(data)
                arealist.append(linelist)
            cubelist.append(arealist)

        processlist = []
        for myarealist in cubelist:
            process = multiprocessing.Process(target=processgo, args=(myarealist, result))
            process.start()
            processlist.append(process)
        for process in processlist:
            process.join()

Pit encountered : a month ago Distributed get time to write ip address how are not open, they had to change computer will support a = =.

If you just get on your computer, then write 127.0.0.1 can run, if you have encountered ip address is not how all open cases

Finishing is not easy, if that helps, I hope you can leave your wonderful remarks before you go. Call it as soon as possible to play your favorite frame.

Since finishing this article: www.ruanbe.com