Data communication and data sharing between processes

Processes are isolated from each other. To implement inter-process communication (IPC), the multiprocessing module supports two forms: queues and pipes, both of which use message passing

process communication

queue

import multiprocessing

def run(q):
    q.put('hello')
    q.put('mainprocess')
    print('subprocess q ID:',id(q))

if __name__ == '__main__':
    q=multiprocessing.Queue()#Queue for inter-process communication
    t=multiprocessing.Process(target=run,args=(q,))
    t.start()
    t.join()

    print('main process q ID:',id(q))
    print(q.get())
    print(q.get())
'''
subprocess q ID: 2359100652232
main process q ID: 2025858869736
hello
mainprocess
'''

pipeline

import multiprocessing
def foo(conn):
    data=conn.recv()
    print(data)
    conn.send('subprocess conn id is %s'%(id(conn)))
    print('this is subprocess')
if __name__ == '__main__':
    parent_conn,child_conn=multiprocessing.Pipe()#Use the pipeline PIPE to get the conn similar to tcp, so as to carry out two-way communication (because it does not involve network transmission, so there is no need to transcode)
    t=multiprocessing.Process(target=foo,args=(child_conn,))
    t.start()
    parent_conn.send('hello subprocess,main proess conn id is %s'%(id(parent_conn)))
    print(parent_conn.recv())

'''
hello subprocess,main proess conn id is 2188825603096
this is subprocess
subprocess conn id is 2818361495736
'''

shared data

Data between processes is independent and can be communicated by means of queues or pipes, both of which are based on message passing

Although the data between processes is independent, data sharing can be achieved through the Manager. In fact, the functions of the Manager are far more than that.

The types supported by Manager are list, dict, Namespace, Lock, RLock, Semaphore, BoundedSemaphore, Condition, Event, Queue, Value and Array

import multiprocessing
def fun(d,l,i):
    d[i]='2'
    l.append(i)
    print('son process id',id(d),id(l))
if __name__ == '__main__':
    with multiprocessing.Manager() as manager:
        d=manager.dict()
        l=manager.list()
        print('main process d  is %s and l is %s'%(d,l))
        print('main process d ID is %s and l ID is %s'%(id(d),id(l)))
        p_list=[]
        for i in range(6):
            t=multiprocessing.Process(target=fun,args=(d,l,i))
            p_list.append(t)
            t.start()
        for t in p_list:
            t.join()
        print(d)
        print(l)
'''
main process d  is {} and l is []
main process d ID is 1304911555608 and l ID is 1304915564584
son process id 1782039016000 1782042035144
son process id 1888584888896 1888587908152
son process id 2214483358104 2214486381456
son process id 1443060675080 1443063694392
son process id 2225121096200 2225123677408
son process id 2640201478664 2640204502016
{0: '2', 1: '2', 2: '2', 3: '2', 4: '2', 5: '2'}
[0, 1, 2, 3, 4, 5]
'''