table of Contents
1. Queue
Isolated from each other processes, to achieve inter-process communication (IPC), multiprocessing module supports two forms: the queue and the pipeline, these two methods are used messaging.
1.1 queue (Queue)
And locking the pipe to the underlying implementation form
1.2 Queue method
Queue([maxsize])
Create a shared process queue
maxsize: the maximum allowable number of queue entries, if omitted, no size limit.
from multiprocessing import Queue q = Queue(5) #队列最大项数5q.put(obj, block=True, [timeout])
from multiprocessing import Queue q = Queue(1) # 将obj放入队列，如果队列已满，将阻塞至有可用空间为止 q.put('a') # block控制阻塞行为，block默认为True，如果设置为False且Queue已满，则引发Queue.Full异常 q.put('b', block=False) q.put_nowait('e') # 等同于q.put('b', block=False) # timeout控制阻塞时间，block为True,在计时结束后，如果Queue还是满的，则引发Queue.Full异常 q.put('c', block=True, timeout=2) q.put('d', timeout=2)q.get(block=True, [timeout])
from multiprocessing import Queue q = Queue() # 从队列中弹出一个元素，如果队列已空，将阻塞至可以获取到元素为止 q.get() # block控制阻塞行为，block默认为True，如果设置为False且Queue已空，则引发Queue.Empty异常 q.get(block=False) q.get_nowait() # 等同于q.get(block=False) # timeout控制阻塞时间，block为True,在计时结束后，如果Queue还是空的，则引发Queue.Empty异常 q.get(block=True, timeout=2) q.get(timeout=2)
2. The producer-consumer model
2.1 What is a producer-consumer model
Producer-consumer model is the strong coupling problem is solved by a producer and consumer of container. Producers and consumers do not directly communicate with each other, and to communicate by blocking queue, so producers after completion of processing of production data without waiting for the consumer, direct throw blocking queue, consumers do not find a producer to data, but taken directly from blocking the queue, the queue is equivalent to a blocking buffer, balance the producers and consumers of processing power.
2.2 Why use producer-consumer model
In the world of the thread, the thread is the producer of production data, the consumer is the thread consumption data. In multi-threaded development which, if treated quickly producer, and consumer processing speed is very slow, then the producer must wait for consumers processed, in order to continue production data. By the same token, if the consumer's capacity is larger than the producer, the consumer would have to wait for the producer. To solve this problem so the introduction of producer and consumer patterns.
2.3 Producer Consumer role model
Producer and consumer use patterns in concurrent programming can solve the vast majority of concurrency problems. The overall pattern to increase the speed of data processing programs by working ability to balance production and consumption thread thread.
# 生产者消费者模型总结
# 程序中有两类角色
一类负责生产数据（生产者）
一类负责处理数据（消费者）
# 引入生产者消费者模型为了解决的问题是：
平衡生产者与消费者之间的工作能力，从而提高程序整体处理数据的速度
# 如何实现：
生产者<-->队列<——>消费者
# 生产者消费者模型实现类程序的解耦和2.4 queue implementation using producer-consumer model
import time, random
from multiprocessing import Process, Queue
def producer(q):
"""生产者"""
for i in range(10):
time.sleep(random.randint(1, 3))
q.put(i)
print(f'生产{i}')
def consumer(q):
"""消费者"""
while True:
time.sleep(random.randint(0, 5))
res = q.get()
print(f'消费{res}')
if __name__ == '__main__':
q = Queue()
p = Process(target=producer, args=(q,))
p.start()
c = Process(target=consumer, args=(q,))
c.start()At this time, the main problem is that the process will never end, because: p producers in the production after it ended, but consumers c After taking empty q, has remained in a cycle of death and engaged in q.get ( ) this step.
The solution but it allows producers after production is completed, to the end of the queue and then send a signal, so consumers end after receiving the signal can break out of the cycle of death.
Solution one: None producers add tags to the queue at the end, consumers get to None will break out of the while loop
import time, random
from multiprocessing import Process, Queue
def producer(q, name):
"""生产者"""
for i in range(10):
time.sleep(random.randint(1, 3))
res = f'{name}生产{i}'
q.put(res)
print(res)
q.put(None)
def consumer(q, name):
"""消费者"""
while True:
time.sleep(random.randint(0, 5))
if not res:
break
res = q.get()
print(f'{name}消费{res}')
if __name__ == '__main__':
q = Queue()
p1 = Process(target=producer, args=(q, 'p1'))
p2 = Process(target=producer, args=(q, 'p2'))
p1.start()
p2.start()
c = Process(target=consumer, args=(q, 'c'))
c.start()Question: multiple consumers at a time is not the end of the queue multiple doping None, if you have a consumer access to None, the consumer is over, but there are elements in the queue, do not meet the requirements.
Solution two: the main process to add tags None queue after the producer
import time, random
from multiprocessing import Process, Queue
def producer(q, name):
"""生产者"""
for i in range(10):
time.sleep(random.randint(1, 3))
res = f'{name}生产{i}'
q.put(res)
print(res)
def consumer(q, name):
"""消费者"""
while True:
time.sleep(random.randint(0, 5))
res = q.get()
if not res:
break
print(f'{name}消费{res}')
if __name__ == '__main__':
q = Queue()
p1 = Process(target=producer, args=(q, 'p1'))
p2 = Process(target=producer, args=(q, 'p2'))
p1.start()
p2.start()
c = Process(target=consumer, args=(q, 'c'))
c.start()
p1.join()
p2.join()
q.put(None)Question: How many producers have to join how many times, how many how many consumers have to add None
3. Use JoinableQueue achieve producer-consumer model
3.1 JoinabaleQueue
JoinabaleQueue([maxsize])
maxsize maximum number of entries allowed in the queue, if omitted, no size limit.
3.2 JoinabaleQueue method
q.task_done()
After each get, signals, get mission has ended, if the number is greater than the number of calls this method to delete items from the queue, will throw ValueError exception
q.join()
Blocks, waiting for the completion of all tasks in the queue
3.3 JoinabaleQueue usage
from multiprocessing import JoinableQueue
q = JoinableQueue()
q.put('a') # 往队列里放入了一个任务
q.put('b')
q.get()
q.task_done() # 完成了一个任务
q.get()
# q.join # 如果放在这里，任务数并没有降为0，将会一直阻塞
q.task_done()
q.join() # 阻塞，等待所有任务完成3.4 JoinableQueue achieve producer-consumer model
import time, random
from multiprocessing import Process, JoinableQueue
def producer(q, name):
"""生产者"""
for i in range(10):
time.sleep(random.randint(0, 2))
res = f'{name}生产{i}'
q.put(res)
print(res)
def consumer(q, name):
"""消费者"""
while True:
time.sleep(random.randint(0, 2))
res = q.get()
q.task_done() # 一个任务已完成，任务数减一
print(f'{name}消费{res}')
if __name__ == '__main__':
q = JoinableQueue()
p1 = Process(target=producer, args=(q, 'p1'))
p2 = Process(target=producer, args=(q, 'p2'))
p1.start()
p2.start()
c = Process(target=consumer, args=(q, 'c'))
c.daemon = True # 将消费者设置为守护进程
c.start()
p1.join() # 确保所有的生产者任务都执行，即任务都加入了队列
p2.join()
q,join() # 如果队列里所有任务都被执行，则阻塞解除，主进程结束，守护进程结束