table of Contents
A room, interprocess communication
IPC. Is among multiple processes communicate with each other. Interprocess communication is not just a matter of a single language. Instead, each language has of. There are a lot of inter-process communication to achieve. C language are: pipe: pipe (anonymous pipes), named pipes, signals, shared memory, message queues, semaphores, etc.
Python is here achieved through inter-process communication queues
Second, the queue
2.1 Concepts --- multiprocessing.Queue
Create a shared process queue, Queue is a multi-process safe queue Queue can be used to pass data between multiple processes.
Queue([maxsize])Create a shared process queue.
Parameters: maxsize is the maximum number of entries allowed in the queue. If omitted, no size limit.
Using pipes and locking the underlying queue implementation.
2.2 Method Introduction
Queue([maxsize]): Create a shared process queue. maxsize is the maximum number of entries allowed in the queue. If omitted, no size limit. Using pipes and locking the underlying queue implementation. In addition, support is also required to run the process to transfer data queue to the underlying pipe.
Examples of q Queue following methods:
q.get( [ block [ ,timeout ] ] ): Return of a project q. If q is empty, this method will block until the queue becomes available items. block for controlling the blocking behavior, the default is True. If set to False, throws exception Queue.Empty (defined in the Queue module). timeout timeout is optional, for use in blocking mode. If there are no changes to the project within the time interval making available, it will lead to Queue.Empty exception.
q.get_nowait(): Same q.get(False)method.
q.put(item [, block [,timeout ] ] ): The item in the queue. If the queue is full, this method will block until space becomes available. block blocking behavior control, the default is True. If set to False, throws exception Queue.Empty (defined in the library module Queue). timeout specifies the length of the wait for an available space of time in blocking mode. After a timeout caused Queue.Full exception.
q.qsize(): Returns the queue correct number of current projects. The results of this function are not reliable, because between the returns, and use the results later in the program, the queue may add or delete items. On some systems, this method may lead to NotImplementedError exception.
q.empty(): Q If this method is called when empty, returns True. If other processes or process is to add items to the queue, the results are not reliable. That is to say, between the return and use the results, the queue may have been added to new projects.
q.full(): If q is full, return to True because of the process, the results may be unreliable (reference. q.empty()Method).
2.3 Other methods (understanding)
q.close(): Turn off the queue, more data is added to prevent the queue. When you call this method, a background process that data will continue to be written into the queue but has not yet been written, but will close immediately upon completion of this method. If q is garbage collected, it will automatically call this method. Close queue does not generate any type of data or an abnormal end signal of the user in the queue. For example, if a user is blocked in get()operation, close the queue does not result in the producer get()method returns an error.
q.cancel_join_thread(): Automatically connect when the background process will not withdraw from the process. This prevents the join_thread()method blocks.
q.join_thread(): Connecting background process queue. This method is used to invoke q.close()the method, all waiting queue items are consumed. By default, this method is called by all processes is not the original creator of q. Call the q.cancel_join_thread()method may prohibit such behavior.
Three, Queue Queue - examples demonstrate
But look at the queue usage 3.1
'''
ipc 进程间通讯。使用队列实现
管道:pipe 基于共享的内存空间
队列:pipe+锁 Queue
put：放（可以设置阻塞非阻塞，和等待时间）
get：取（可以设置阻塞非阻塞，和等待时间）
'''
### 语法
q = Queue(3) # 创建队列, 可以设置最大值
## put 放
q.put('你好') # 往队列中放入值，可以设置阻塞和等待时间，默认：满了再放就会阻塞等待
q.put([1,2,4])
q.put(2)
### 不设置等待时间，队列满了继续放
# q.put(5) # 阻塞，队列满了，等待队列空了就放进去
## get 拿
print(q.get()) # 获取队列中的内容，可以设置阻塞和等待时间，默认：拿不到内容就会阻塞等待
print(q.get())
print(q.get())
### 不设置等待时间
# print(q.get()) # 阻塞，一直等待获取队列内容
### 设置等待时间
# print(q.get(timeout=2)) # 等两秒，等不到就报错
# q.put(88,timeout=2) # 不会报错，因为队列中没内容
### 全都设置非阻塞模式
q = Queue(3) # 创建队列, 可以设置最大值
q.put('你好', block=False) # 设置非阻塞，如果满了再放就会报错
q.put([1,2,4], block=False)
q.put(2, block=False)
# q.put(4, block=False) # 报错，队列满了
# q.put_nowait('666') # 等同 block = False，报错，队列满了3.2 sub-process sends data to the parent process
import time
from multiprocessing import Process, Queue
def f(q):
q.put([time.asctime(), 'from Eva', 'hello']) #调用主函数中p进程传递过来的进程参数 put函数为向队列中添加一条数据。
if __name__ == '__main__':
q = Queue() #创建一个Queue对象
p = Process(target=f, args=(q,)) #创建一个进程
p.start()
print(q.get())
p.join()The above is a simple application of the queue, the queue q using the object invokes the get function to get into the first data queue. Next, look at a slightly more complex example: batch production data acquisition result into the queue again.
Fourth, the producer-consumer model
Producer and consumer use patterns in concurrent programming can solve the vast majority of concurrency problems. The overall pattern to increase the speed of data processing programs by balancing production and consumption processes and the ability to work the process.
Why do you want to use 4.1 Producers and consumer patterns
In the process of world producers is the process of production data, the consumer is the process of consumption data. In the multi-process development which, if treated quickly producer, and consumer processing speed is very slow, then the producer must wait for consumers processed, in order to continue production data. By the same token, if the consumer's capacity is larger than the producer, the consumer would have to wait for the producer. To solve this problem so the introduction of producer and consumer patterns.
4.2 What is a producer-consumer model
Producer-consumer model is the strong coupling problem is solved by a producer and consumer of container. Producers and consumers do not directly communicate with each other, and to communicate by blocking queue, so producers after completion of processing of production data without waiting for the consumer, direct throw blocking queue, consumers do not find a producer to data, but taken directly from blocking the queue, the queue is equivalent to a blocking buffer, balance the producers and consumers of processing power.
4.3 queue-based producer-consumer model to achieve
ef producer(q,name,food):
'''生产者'''
for i in range(10):
print(f'{name}生产了{food}{i}')
res = f'{food}{i}'
q.put(res)
def consumer(q,name):
'''消费者'''
while 1:
res = q.get()
if res == None:
break
print(f'{name}买了{res}')
if __name__ == '__main__':
q = Queue() # 使用队列
p1 = Process(target=producer,args=(q,"xc","意大利面"))
c1 = Process(target=consumer,args=(q,"haha"))
p2 = Process(target=producer, args=(q, "xc", "牛肉"))
c2 = Process(target=consumer, args=(q, "xixi"))
p3 = Process(target=producer, args=(q, "xc", "可乐"))
p1.start()
c1.start()
p2.start()
c2.start()
p3.start()
p1.join() # 等待生产者结束
p2.join() # 等待生产者结束
p3.join() # 等待生产者结束
q.put(None) # 有几个消费者就要发几个None，让 子进程收到None就结束
q.put(None) # 有几个消费者就要发几个None，让 子进程收到None就结束End signal None, not necessarily made by the producer, the main process where you can send the same, but the process need to wait until after the end of the main producers should send the signal. There are several consumer should be made a few None, so that the child received None is over
Five, JoinableQueue queue
JoinableQueue queue on a modified version of Queue queue, adding a mechanism similar to the semaphore.
5.1 JoinableQueue use
grammar:
- Examples of objects:
q = JoinableQueue() - Add the contents of the queue, the signal corresponding to the amount of +1:
q.put(xxx) - Taken out from the queue:
q.get() - End of the mission, a signal corresponding to an amount of 1 operation:
q.task_done() - When the semaphore is not 0, it will block waiting, after the counter is 0 through.
Example:
# JoinableQueue的使用
q = JoinableQueue()
q.put('1') # +1
q.put('2') # +1
print(q.get())
q.task_done() # -1
print(q.get())
q.task_done() # -1
q.join() #计数器不为0会阻塞等待 计数器为0后通过5.2 realized by the producer consumer model queue JoinableQueue
def producer(q, name, food):
'''生产者'''
for i in range(10):
print(f'{name}生产了{food}{i}')
res = f'{food}{i}'
q.put(res) # +1
def consumer(q, name):
'''消费者'''
while 1:
res = q.get()
if res == None:
break
print(f'{name}买了{res}')
q.task_done()
if __name__ == '__main__':
q = JoinableQueue() # 使用加了join的队列
p1 = Process(target=producer, args=(q, "xc", "意大利面"))
c1 = Process(target=consumer, args=(q, "haha"))
p2 = Process(target=producer, args=(q, "xc", "牛肉"))
c2 = Process(target=consumer, args=(q, "xixi"))
p3 = Process(target=producer, args=(q, "xc", "可乐"))
# 把消费者变成守护进程，主进程结束，子进程就结束
c1.daemon = True # 进程结束，子进程就结束
c2.daemon = True # 进程结束，子进程就结束
p1.start()
c1.start()
p2.start()
c2.start()
p3.start()
p1.join() # 等待生产者结束
p2.join() # 等待生产者结束
p3.join() # 等待生产者结束The above code, is through the child process becomes a daemon , consumers in the child's messages task_done each time it receives , and then wait for the end of the primary process, the child process is ended .