python multiprocessing

multiprocessing in python

1. Create a process with fork (used in Linux)

1  import os
 2  # fork function, only run on Unix/Linux/Mac, not windows 
3 pid = os.fork()
 4  if pid == 0:
 5  　　print ( 'hello-- 1 ' )
 6  else :
 7  　　print ( 'hello-- 2 ' )

When the program goes to os.fork(), it will create a child process and copy all the information in the parent process to the child process. Both the parent process and the child process will get a return value from os.fork(). The return value in the parent process is 0, and the return value in the parent process is the id number of the child process

Note: when fork is called, it will return two values, one is 0 representing the child process, and the other is the child process id number representing the parent process

The execution order of the parent process and the child process is irregular and depends entirely on the scheduling algorithm of the system

Get the process id number:

　　　　Get the process id number of the current process: getpid()

　　　　Get the process id number of the parent process: getppid()

The impact of multiple processes on global variables:

　　　　　In multi-process, each process has a copy of all data (including global variables), which is not affected

　　　　　　Mainly because the essence of multi-process is that each process occupies each core of the cpu in turn, and there is only one process on each core of the cpu at each moment.

2. Use multiprocessing to create multiple processes

1  from multiprocessing import *
 2  import os
 3  
4  def run_process(num):
 5      print ( " child process %d--->%d its parent--->%d " % (num,os.getpid() ,os.getppid()))
 6  
7  def main():
 8      #Start to engage in 5 processes 
9      for num in range(5 ):
 10          # Commonly used parameters in Process are target , args , kwargs , name where name means To name an alias for this process, if the name parameter is not written, the default is name = Process-N N is an incrementing integer 
11          p = Process(target=run_process,args=(num,))
 12          p.start()
 13          # p.join() The function of join() is to wait for the child process to end before going down, which is usually used for synchronization between processes 
14          # join(timeout) join() Accepts a timeout parameter to determine how many seconds it should wait 
15  
16  if  __name__ == " __main__ " :
 17      main()

The results of its operation are as follows:

The number of the child process in the running result does not follow the order of 0, 1, 2, 3, 4 in the for loop because the operation of the child process is not in order, it depends on the scheduling algorithm of the system

3. Create a process through a subclass of Process

1  from multiprocessing import Process
 2  import os
 3  
4  #Create   a subclass of Process 
5  class process_children(Process):
 6      def  __init__ (self, num):
 7          #This is mainly because the Process class also has an __init__ method 
8          #If Not adding or not writing this sentence is equivalent to rewriting the __init__ method, but this is not the result we want. 
9          #The function of this sentence is to pass the parameters received in the subclass to the Process parent 
class          Process. __init__ (self)
 11 self. num          = num
 12      
13      #This run method must be defined and written as run, because the code in this function determines how the created child process will execute 
14      def run(self):
 15          print ( " %s ---> The current process id is %s , the parent process is %s " % (self.num, os.getpid(), os.getppid()))
 16  
17  def main():
 18      while True:
 19          for num in range(5 ):
 20              #if we Use a subclass of Process to create a process and pass the parameters that need to be passed directly to the class. 
21              p = process_children(num)
 22              p.start()
 23              # p.join() 
24  
25  if  __name__ == "__main__":
26     main()

4. Process pool (code tested under Ubuntu)

1  from multiprocessing import Pool
 2  import os
 3  
4  def func(num):
 5      print ( " %d---> The current process is: %d, and its parent process is: %d " % (num, os.getpid( ), os.getppid()))
 6  
7 #Open  a process pool, the maximum number of processes in the pool is 3 8 #The maximum number of processes means that the maximum number of processes in this process pool at the same time is 3 9 #If When a process in the process pool ends, a new process will be created for the waiting process 10 po=Pool(3 )
 11 12 def main():
 13 for i in range(0,10 ):
 14
 
 
 
               # apply_async is non-blocking mode apply is blocking mode    
15          po.apply_async(func, args= (i,))
 16      print ( " start " )
 17      # close must be placed before join 
18      po.close()
 19      # join The role is to let the main process wait for the child process to finish executing 
20      po.join()
 21      print ( " finish " )
 22  
23  if  __name__ == " __main__ " :
 24      main()

operation result:

Important: The final close() must be written before join()

5. Communication between processes ------ Queue Queue

　　Queue: first in first out (FIFO) --------- first in first out

　　Stack: first in last out(FILO)---------- first in last out

　　　　　　The use of the queue: q = Queue(num) num indicates the maximum length of the queue

　　　　　　　　　Write: q.put(value) , q.put_nowait() Read: q.get() , q.get_nowait()

　　　　　　　　　Judgment: q.full() , q.empty() , q.qsize()

　　　　　　　　　　q.get([block[timeout]])　　　　　　

　　1> You can use Queue to transfer data between two processes, one process Queue.put(), the other process Queue.get()

　　2> The inter-process communication in the process pool needs to be imported into the Manager in multiprocessing: from multiprocessing impor Manager, Pool