Producer and consumer queue model

Producer and consumer queue model

A queue Queue module

• For IPC: communication between processes, as well as having the same function Manager module
  • Communication between processes can be used in a queue and pipe Pipe
  • Queue is based socket, pickle and locks to achieve, because the lock is achieved based on, the data transmission queue must be safe, but the transmission efficiency loss will be.
  • Based pipeline socket and pickle achieve, but not locked, so the high pipeline transmission efficiency, data is not secure.
• Multi-process queue is different from the ordinary queue
• 导入：from multiprocessing import Queue
• Examples of objects: q = Queue ()
• To queue incoming data: q.put (types of data)

• Extract data from the queue: q.get (stored data)

  Of particular note are: data in the queue can only be taken once from inside, can not take a second, if still need to take the data, the data must be stored again, similar Poen usage and the usage of this subprocess module, if only to put a data, but taking twice, first to take the program in a data card in there, it has not terminated.

• Data queue follow the FIFO principle, while the stack is to follow the principle of last-out.

from multiprocessing import Queue,Process
import os


def son(q):
    q.put('你好')
    # 往队列中添加消息：put，第一次存入数据
    print(q.get())
    # 从队列中获取消息：get()

def pro(q):
    print(q.get())
    # 接收第二个数据

if __name__ == '__main__':

    q = Queue()
    p = Process(target=son, args=(q,))
    p.start()
    p1 = Process(target=pro, args=(q,))
    p1.start()
    print(q.get())
    # 接收第一个数据
    q.put('hello')
    q.put('第二个你好')
    # 第二次存入数据

'''
结果是：
你好
hello
第二个你好

'''

Second, the producer-consumer model

• Basic concepts:
  • Producer and consumer model the complete original tightly coupled data processing is split into a plurality of loosely coupled data processing section, to facilitate planning modifications to different parts of the producer and consumer data processing efficiency according to the production and consumption of number, to achieve maximum efficiency and balance.
  • Manufacturer: acquiring first data codes, and the data processing process, processing is complete the data stored into the queue
  • Consumer: producers extracted data after processing from the queue, but also to perform certain operations
  • Queue: Queue Queue is a data exchange between consumers and producers need to use a multi-process communication.
• Application scenarios:
  • In reptiles time for acquisition and processing of web pages you need to use the model
  • Will be used in a distributed operating in celery, celery is actually two large producers and news by model
• The essence of the model is:
  let the efficiency of production data and consumption data can achieve balance and maximize efficiency.

from multiprocessing import Queue, Process
import time
import random

def consumer(q):
    for i in range(12):
        print(q.get(i))



def producer(q):
    for i in range(12):
        q.put(i)
        time.sleep(random.randint(1,3))



if __name__ == '__main__':
    q = Queue()
    c1 = Process(target=consumer, args=(q,))
    p1 = Process(target=producer, args=(q,))

    p1.start()
    c1.start()

• Problems and solutions arise:
  • When an equal number of producers and consumers, the producers are likely to speed the production of consumer spending will not keep pace, each producing a commodity, consumers will quickly eaten.

  '''版本一：生产者和消费者都是一个，或者数量一样，每生产一个，消费者快速的吃掉，这个会出现生产慢，消费快，
  供不应求的情况
  '''
  from multiprocessing import Process, Queue
  import time
  import random
  
  
  def producer(q, name):
      for i in range(1,11):
          food = 'food%s' % i
          q.put(food)
          print('%s生产了商品%s' % (name, food))
          time.sleep(1)
  
  
  
  def consumer(q, name):
      for i in range(1, 11):
          food = q.get()
          print('%s消费了商品%s' % (name, food))
          time.sleep(1)
  
  
  if __name__ == '__main__':
      q = Queue()
      c1 = Process(target=consumer, args=(q,'小明'))
      p1 = Process(target=producer, args=(q,'刘洋'))
      p1.start()
      c1.start()
  

  • In order to solve the case of the top short supply directly to open a producer process, production volume will be doubled this time, but consumers can only consume half of the oversupply situation, if the increase in consumer demand and oversupply will continue case cycle.

  '''版本二：为解决版本一的问题，版本二中添加一个生产者，扩大供应'''
  from multiprocessing import Process, Queue
  import time
  import random
  
  
  def producer1(q, name):
      for i in range(1,11):
          food = 'food%s' % i
          q.put(food)
          print('%s生产了商品%s' % (name, food))
          time.sleep(1)
  
  def produce2(q, name):
      for i in range(1,11):
          food = 'food%s' % i
          q.put(food)
          print('%s生产了商品%s' % (name, food))
          time.sleep(1)
  
  
  
  def consumer(q, name):
      for i in range(1, 11):
          food = q.get()
          print('%s消费了商品%s' % (name, food))
          time.sleep(1)
  
  
  
  if __name__ == '__main__':
      q = Queue()
      c1 = Process(target=consumer, args=(q,'小明'))
      p1 = Process(target=producer1, args=(q,'刘洋'))
      p2 = Process(target=produce2, args=(q, '玲玲'))
      p1.start()
      p2.start()
      c1.start()
  

  • In response to these two versions of the question, we need to model the consumer is set to an infinite loop, stop the spending, the producers set the number of production models, and then complete the transfer of production to a None queue when consumers get to None when out of the loop, note that you must set the asynchronous synchronization, that producers must produce finished this way, there is not a query queue length.

  '''版本三：解决上述两个版本的问题：不设定消费者消费的数量，只设定生产者生产的数量'''
  
  from multiprocessing import Process, Queue
  import time
  import random
  
  
  def producer1(q, name):
      for i in range(1,11):
          food = 'food%s' % i
          q.put(food)
          print('%s生产了商品%s' % (name, food))
          time.sleep(1)
  
  def produce2(q, name):
      for i in range(1,11):
          food = 'food%s' % i
          q.put(food)
          print('%s生产了商品%s' % (name, food))
          time.sleep(1)
  
  
  
  def consumer(q, name):
      while True:
          food = q.get()
          # 注意：q.get()一个数据只能取一次，q.get()不能直接用于判断条件，需要赋值给变量
          if food:
              # 存在消费
              print('%s消费了商品%s' % (name, food))
              time.sleep(1)
          else:
              # 如果不存在直接跳出循环
              break
  
  if __name__ == '__main__':
      q = Queue()
      c1 = Process(target=consumer, args=(q,'小明'))
      p1 = Process(target=producer1, args=(q,'刘洋'))
      p2 = Process(target=produce2, args=(q, '玲玲'))
      p1.start()
      p2.start()
      p1.join()
      p2.join()
      # 添加join()让两个生产者生产完成
      q.put(None)
      # 再向队列最后再添加一个None，因为队列遵循先进先出的原则，所以最后一个获取的是None
      c1.start()
  

  Extraneous: the daemon, the main process, sub-process, the daemon must precede the main process ends, the main process must finally end.