Network programming of concurrent programming - producer-consumer model

Network programming of concurrent programming - producer-consumer model

First, the producer-consumer model introduction

Why use producer-consumer model?

Producers refers to the task of production data, consumer refers to the task of data processing. In concurrent programming, if the producer processed quickly, while consumer processing speed is very slow, then the producer must wait for consumers processed, in order to continue production data. By the same token, if the consumer's capacity is larger than the producer, the consumer what I have to wait for the producer. To solve this problem so the introduction of producer and consumer patterns.

What is the producer and consumer model?

Producer-consumer model is the strong coupling problem is solved by a producer and consumer of container. Producers and consumers do not directly communicate with each other, and to communicate by blocking queue, so producers after completion of processing of production data without waiting for the consumer, direct throw blocking queue, consumers do not find a producer to data, but taken directly from blocking the queue, the queue is equivalent to a blocking buffer, balance the producers and consumers of processing power.

The blocking queue is used to decouple the producers and consumers.

Second, the producer-consumer model implementation

To implement a producer-consumer model is based on a study section of the queue:

from multiprocessing import Process,Queue
import time,random,os
def consumer(q,name):
    while True:
        res=q.get()
        time.sleep(random.randint(1,3))
        print('\033[43m%s 吃 %s\033[0m' %(name,res))
def producer(q,name,food):
    for i in range(3):
        time.sleep(random.randint(1,3))
        res='%s%s' %(food,i)
        q.put(res)
        print('\033[45m%s 生产了 %s\033[0m' %(name,res))
if __name__ == '__main__':
    q=Queue()
    #生产者们:即厨师们
    p1=Process(target=producer,args=(q,'egon','包子'))
    #消费者们:即吃货们
    c1=Process(target=consumer,args=(q,'alex'))
    #开始
    p1.start()
    c1.start()
    print('主')

Results of the:

主
egon 生产了 包子0
egon 生产了 包子1
alex 吃 包子0
alex 吃 包子1
egon 生产了 包子2
alex 吃 包子2

At this time, the main problem is that the process will never end, because: p producers in the production after it ended, but consumers c After taking empty q, has remained in a cycle of death and engaged in q.get ( ) this step.

The solution but it allows producers after production is completed, to the end of the queue and then send a signal, so that consumers received again after the end of the signal can break out of the cycle of death.

from multiprocessing import Process,Queue
import time,random,os
def consumer(q,name):
    while True:
        res=q.get()
        if res is None:break
        time.sleep(random.randint(1,3))
        print('\033[43m%s 吃 %s\033[0m' %(name,res))
def producer(q,name,food):
    for i in range(3):
        time.sleep(random.randint(1,3))
        res='%s%s' %(food,i)
        q.put(res)
        print('\033[45m%s 生产了 %s\033[0m' %(name,res))
if __name__ == '__main__':
    q=Queue()
    #生产者们:即厨师们
    p1=Process(target=producer,args=(q,'egon','包子'))
    #消费者们:即吃货们
    c1=Process(target=consumer,args=(q,'alex'))
    #开始
    p1.start()
    c1.start()
    p1.join()
    q.put(None)
    print('主')

However, the above solution, when there are multiple producers and multiple consumers, we then choose a way to solve very low, there are a few end consumers need to send several signals: quite low. E.g:

from multiprocessing import Process,Queue
import time,random,os
def consumer(q,name):
    while True:
        res=q.get()
        if res is None:break
        time.sleep(random.randint(1,3))
        print('\033[43m%s 吃 %s\033[0m' %(name,res))
def producer(q,name,food):
    for i in range(3):
        time.sleep(random.randint(1,3))
        res='%s%s' %(food,i)
        q.put(res)
        print('\033[45m%s 生产了 %s\033[0m' %(name,res))
if __name__ == '__main__':
    q=Queue()
    #生产者们:即厨师们
    p1=Process(target=producer,args=(q,'egon1','包子'))
    p2=Process(target=producer,args=(q,'egon2','骨头'))
    p3=Process(target=producer,args=(q,'egon3','泔水'))
    #消费者们:即吃货们
    c1=Process(target=consumer,args=(q,'alex1'))
    c2=Process(target=consumer,args=(q,'alex2'))
    #开始
    p1.start()
    p2.start()
    p3.start()
    c1.start()
    c2.start()
    p1.join()
    p2.join()
    p3.join()
    q.put(None)
    q.put(None)
    q.put(None)
    print('主')

A digital