python uses Queue for interprocess communication

Inter-process communication-Communication is sometimes required between Queue Processes
, and the operating system provides many mechanisms to achieve inter-process communication.

Use of Queue You can use the Queue of the multiprocessing module to realize data transfer between multiple processes. Queue itself is a message queue program. First, a small example is used to demonstrate the working principle of Queue:

#coding=utf-8
from multiprocessing import Queue
q=Queue(3) #初始化一个Queue对象，最多可接收三条put消息
q.put("消息1") 
q.put("消息2")
print(q.full())  #False
q.put("消息3")
print(q.full()) #True

#因为消息列队已满下面的try都会抛出异常，第一个try会等待2秒后再抛出异常，第二个Try会立刻抛出异常
try:
    q.put("消息4",True,2)
except:
    print("消息列队已满，现有消息数量:%s"%q.qsize())

try:
    q.put_nowait("消息4")
except:
    print("消息列队已满，现有消息数量:%s"%q.qsize())

#推荐的方式，先判断消息列队是否已满，再写入
if not q.full():
    q.put_nowait("消息4")

#读取消息时，先判断消息列队是否为空，再读取
if not q.empty():
    for i in range(q.qsize()):
        print(q.get_nowait())

operation result:

False
True
消息列队已满，现有消息数量:3
消息列队已满，现有消息数量:3
消息1
消息2
消息3

Note When
initializing a Queue() object (for example: q=Queue()), if the maximum number of messages that can be received is not specified in parentheses, or the number is negative, it means that there is no upper limit on the number of messages that can be received (until the end of memory );

Queue.qsize(): Returns the number of messages contained in the current queue;

Queue.empty(): If the queue is empty, return True, otherwise False;

Queue.full(): If the queue is full, return True, otherwise False;

Queue.get([block[, timeout]]): Get a message in the queue, and then remove it from the queue, the default value of block is True;

1) If the default value of block is used and the timeout (in seconds) is not set, if the message queue is empty, the program will be blocked (stop in the reading state) until the message is read from the message queue, if the timeout is set , it will wait for timeout seconds, and if no message has been read, a "Queue.Empty" exception will be thrown;

2) If the block value is False, if the message queue is empty, the "Queue.Empty" exception will be thrown immediately;

Queue.get_nowait()：相当Queue.get(False)；

Queue.put(item,[block[, timeout]]): Write the item message to the queue, the default value of block is True;

1) If the block uses the default value and the timeout (in seconds) is not set, if the message queue has no space to write, the program will be blocked (stop in the writing state) until space is vacated from the message queue, If timeout is set, it will wait for timeout seconds, and if there is no space, a "Queue.Full" exception will be thrown;

2) If the block value is False, if there is no space to write in the message queue, the "Queue.Full" exception will be thrown immediately;

Queue.put_nowait(item): Equivalent to Queue.put(item, False);
2. Queue instance
Let's take Queue as an example, create two child processes in the parent process, one writes data to the Queue, and the other reads data from the Queue:

from multiprocessing import Process, Queue
import os, time, random

# 写数据进程执行的代码:
def write(q):
    for value in ['A', 'B', 'C']:
        print 'Put %s to queue...' % value
        q.put(value)
        time.sleep(random.random())

# 读数据进程执行的代码:
def read(q):
    while True:
        if not q.empty():
            value = q.get(True)
            print 'Get %s from queue.' % value
            time.sleep(random.random())
        else:
            break

if __name__=='__main__':
    # 父进程创建Queue，并传给各个子进程：
    q = Queue()
    pw = Process(target=write, args=(q,))
    pr = Process(target=read, args=(q,))
    # 启动子进程pw，写入:
    pw.start()    
    # 等待pw结束:
    pw.join()
    # 启动子进程pr，读取:
    pr.start()
    pr.join()
    # pr进程里是死循环，无法等待其结束，只能强行终止:
    print ''
    print '所有数据都写入并且读完'

operation result:

Put A to queue...
Put B to queue...
Put C to queue...
Get A to queue...
Get B to queue...
Get C to queue...
所有数据都写入并且读完

Queue in the process pool
If you want to use Pool to create a process, you need to use Queue() in multiprocessing.Manager() instead of multiprocessing.Queue(), otherwise you will get the following error message:

RuntimeError: Queue objects should only be shared between processes through inheritance.

The following example demonstrates how processes in a process pool communicate:

#coding=utf-8

#修改import中的Queue为Manager
from multiprocessing import Manager,Pool
import os,time,random

def reader(q):
    print("reader启动(%s),父进程为(%s)"%(os.getpid(),os.getppid()))
    for i in range(q.qsize()):
        print("reader从Queue获取到消息：%s"%q.get(True))

def writer(q):
    print("writer启动(%s),父进程为(%s)"%(os.getpid(),os.getppid()))
    for i in "dongGe":
        q.put(i)

if __name__=="__main__":
    print("(%s) start"%os.getpid())
    q=Manager().Queue() #使用Manager中的Queue来初始化
    po=Pool()
    #使用阻塞模式创建进程，这样就不需要在reader中使用死循环了，可以让writer完全执行完成后，再用reader去读取
    po.apply(writer,(q,))
    po.apply(reader,(q,))
    po.close()
    po.join()
    print("(%s) End"%os.getpid())

operation result:

(21156) start
writer启动(21162),父进程为(21156)
reader启动(21162),父进程为(21156)
reader从Queue获取到消息：d
reader从Queue获取到消息：o
reader从Queue获取到消息：n
reader从Queue获取到消息：g
reader从Queue获取到消息：G
reader从Queue获取到消息：e
(21156) End

python uses Queue for interprocess communication

Guess you like