The producer consumer model of python

The producer consumer model of python

The producer consumer model acts on:
1. When crawling
2. Distributed operation:
the essence of celery : it is to balance the efficiency of production data and consumption data and maximize efficiency

Why use the producer consumer model?
In concurrent programming, if the producer's processing speed is fast, and the consumer's processing speed is relatively slow, then the producer must wait for the consumer to finish processing before it can continue to produce data. In the same way, if the consumer's processing power is greater than that of the producer, then the consumer must wait for the producer. In order to solve this waiting problem, the producer and consumer model was introduced. Allow them to continuously produce and consume.

When to use this model?
There are two obvious types of tasks in the program, one type of task is responsible for production, and the other type of task is responsible for processing production data (such as crawlers)

What are the benefits of using this model?
1. Realize the decoupling and integration of producers and consumers

2. It balances productivity and consumption power, that is, producers have been producing continuously, and consumers can continue to consume, because the two are no longer directly communicating, but communicating with the queue.

from multiprocessing import Queue
from multiprocessing import Process
import time
import random

def consumer(q,name):  # 消费者:通常取到某些数据之后还要进行某些操作
    while True:
        food = q.get()
        if food:
            print("%s吃了%s" % (name, food))
        else:
           break

def producer(q,name,food):  # 生产者：通常在数据之前需要通过某些代码来获取数据
    for i in range(10):
        foodi = "%s%s" % (food,i)
        print("%s生产了%s" % (name,foodi))
        # time.sleep(random.randint(1,3))
        q.put(foodi)
    #q.put(None)


if __name__ == "__main__":
    q = Queue()
    p1 = Process(target=producer,args=(q,'大壮','泪水'))
    c1 = Process(target=consumer,args=(q,'alex'))
    c1.start()
    p1.start()
    p1.join()

大壮生产了泪水0
大壮生产了泪水1
大壮生产了泪水2
大壮生产了泪水3
alex吃了泪水0
alex吃了泪水1
alex吃了泪水2
大壮生产了泪水4
大壮生产了泪水5
alex吃了泪水3
alex吃了泪水4
大壮生产了泪水6
alex吃了泪水5
大壮生产了泪水7
大壮生产了泪水8
大壮生产了泪水9
alex吃了泪水6
alex吃了泪水7
alex吃了泪水8
alex吃了泪水9

Now it is true that producers continue to produce and consumers continue to consume. But there is a problem at this time that the main process is not over. The reason is: the producer p1 ends after the production, but the consumer c1, after the q.get() is empty, has been waiting in place. Solving this problem is nothing more than letting the producer send an end signal to the queue after the production is completed, so that the consumer can jump out of the infinite loop after receiving the end signal. Modify as follows:

from multiprocessing import Queue
from multiprocessing import Process
import time
import random

def consumer(q,name):  # 消费者:通常取到某些数据之后还要进行某些操作
    while True:
        food = q.get()
        if food:
            print("%s吃了%s" % (name, food))
        else:
           break

def producer(q,name,food):  # 生产者：通常在数据之前需要通过某些代码来获取数据
    for i in range(10):
        foodi = "%s%s" % (food,i)
        print("%s生产了%s" % (name,foodi))
        # time.sleep(random.randint(1,3))
        q.put(foodi)
    q.put(None)  #修改了这里


if __name__ == "__main__":
    q = Queue()
    p1 = Process(target=producer,args=(q,'大壮','泪水'))
    c1 = Process(target=consumer,args=(q,'alex'))
    c1.start()
    p1.start()
    p1.join()