day 36 (multi-process)

multi-Progress

A, multiprocessing module

multithreading in python can not take advantage of multi-core advantage, if you want to get the most from multicore CPU resources (os.cpu_count () view), in python in most cases require the use of multiple processes. Python provides multiprocessing.
multiprocessing module to enable the child process, and the implementation of our custom tasks (such as function) in the child process, similar to the module threading module and multi-threaded programming interface.
many features multiprocessing module: support for the child process, communicate and share data and synchronize different forms, provides Process, Queue, Pipe, Lock and other components.

Need to re-emphasize is this: Unlike threads, processes, no shared state, the process of modifying the data, only changes in the process.

1, Introduction Process class

1.1 to create a process of class:

Process ([group [, target [ , name [, args [, kwargs]]]]]), the object of this class is instantiated obtained indicate a sub-task process (not yet started)
stressed:

1. You need to use the keyword way to specify parameters
2. args specified target position for the transfer function parameters, is in the form of a tuple, there must be a comma

1.2 parameters introduced:

• group parameter is not used, the values ​​are always None
• target indicates that the call target, that is the task to be executed child process
• args parameter indicates the position of the tuple call object, args = (1,2, 'egon',)
• represents kwargs call object dictionary, kwargs = { 'name': 'egon', 'age': 18}
• name is the name of the child process

1.3 Introduction property:

1 p.daemon: The default value is False, if set to True, the representative of p is a daemon running in the background, when the termination of the parent process p, p also will be terminated, and after setting True, you can not create your own p the new process must be p.start () before setting

Name of the process: 2 p.name

pid process: 3 p.pid

4 p.exitcode: process at runtime to None, if -N, representing the end of the signal N (to understand)

5 p.authkey: identity verification key process, by default os.urandom 32 randomly generated character string (). To succeed (to understand) when the key purpose is to provide security for the network connection between the underlying processes involved in communication, this type of connection only with the same authentication key

Note: You must put # if name == 'main' in the windows Process (): down

Since Windows has no fork, the multiprocessing module starts a new Python process and imports the calling module.
If Process() gets called upon import, then this sets off an infinite succession of new processes (or until your machine runs out of resources).
This is the reason for hiding calls to Process() inside

IF name == " main "
Operating since statements of Statement Inside the this-IF GET Called upon by Will not Import.
Because Windows does not fork, the multiprocessing module to start a new process and import the Python calling module.
If you call Process in import (), then it will start a new process unlimited inherited (or until the machine is running out of resources).
It is hidden inside the original call to Process (), use if name == " main ", this statement if the statement will not be called upon import.

2, two ways to create a child process

The first

from multiprocessing import Process
import time

def task(n):
    print('子进程开始')
    time.sleep(n)
    print('子进程结束')

if __name__ == '__main__':
    p = Process(target=task,args=(1,))
    p.start()  # p.start 只是向操作系统发了一个信号，请求开启子进程，操作系统具体什么时候开，开多长时间你控制不了。
    time.sleep(5) #
    print('我是主进程')

The second

from multiprocessing import Process
import time


class MyProcesszzz(Process):
    def __init__(self,x):# 如果不传参没必要重写init
        super().__init__()
        self.n = x

    def run(self):
        print('我是子进程 start')
        time.sleep(self.n)
        print('我是子进程 end')


if __name__ == '__main__':
    p1 = MyProcesszzz(2)
    p1.start()

    print('我是主进程')


'''
思考：主进程是什么结束的呢？
ps: 主进程会在子进程都结束的时候再结束 
why？详情见僵尸进程
'''

3, orphaned zombie process

'''引号的内容不适合初次阅读本博客思考
# 僵尸进程
1 何为没死干净的进程--》2 父亲都会有什么需求--》3 僵尸进程的概念--》4 什么时候回收pid合理--》5 之所以父等待子进程结束的原因。
# 孤儿进程
1 父进程死了谁来回收自己程? --》2 孤儿进程的概念 --》是父进程来回收好？还是init回收好？--》孤儿进程总归是无害的。
'''

#二：僵尸进程（父进程一直不死不停造子进程并且不回收僵尸进程有害）
    一个进程使用fork创建子进程，如果子进程退出，而父进程并没有调用wait或waitpid获取子进程的状态信息，那么子进程的进程描述符仍然保存在系统中。这种进程称之为僵尸进程。
因此，UNⅨ提供了一种机制可以保证父进程可以在任意时刻获取子进程结束时的状态信息：
1、在每个进程退出的时候，内核释放该进程所有的资源，包括打开的文件，占用的内存等。但是仍然为其保留一定的信息（包括进程号the process ID，退出状态the termination status of the process，运行时间the amount of CPU time taken by the process等）
2、直到父进程通过wait / waitpid来取时才释放. 但这样就导致了问题，如果进程不调用wait / waitpid的话，那么保留的那段信息就不会释放，其进程号就会一直被占用，但是系统所能使用的进程号是有限的，如果大量的产生僵死进程，将因为没有可用的进程号而导致系统不能产生新的进程. 此即为僵尸进程的危害，应当避免。

　　任何一个子进程(init除外)在exit()之后，并非马上就消失掉，而是留下一个称为僵尸进程(Zombie)的数据结构，等待父进程处理。这是每个子进程在结束时都要经过的阶段。如果子进程在exit()之后，父进程没有来得及处理，这时用ps命令就能看到子进程的状态是“Z”。如果父进程能及时 处理，可能用ps命令就来不及看到子进程的僵尸状态，但这并不等于子进程不经过僵尸状态。  如果父进程在子进程结束之前退出，则子进程将由init接管。init将会以父进程的身份对僵尸状态的子进程进行处理。
# 二：孤儿进程（无害）
　　孤儿进程：一个父进程退出，而它的一个或多个子进程还在运行，那么那些子进程将成为孤儿进程。孤儿进程将被init进程(进程号为1)所收养，并由init进程对它们完成状态收集工作。

　　孤儿进程是没有父进程的进程，孤儿进程这个重任就落到了init进程身上，init进程就好像是一个民政局，专门负责处理孤儿进程的善后工作。每当出现一个孤儿进程的时候，内核就把孤 儿进程的父进程设置为init，而init进程会循环地wait()它的已经退出的子进程。这样，当一个孤儿进程凄凉地结束了其生命周期的时候，init进程就会代表党和政府出面处理它的一切善后工作。因此孤儿进程并不会有什么危害。

Possibilities arise:

I.

1 parent child normal recovery process (regardless of the end of the parent process automatic and manual recovery processes are harmless zombie recovery)

Scenario 2

2 parent die first, there has been orphaned (probably the first parent dies, the child is not dead, orphaned child processes programmed by the init process management recovery, harmless.)

Three cases

3 dead parent has ever turned a child process, and not manually recovered. (harmful)

The parent process has been dead ever open a child process, the situation has not initiated recovery zombie, parent die, init will not help the parent process recycling, generated a lot of zombies unmanned recycling, taking up a lot of pid, it can cause other programs no pid is available.

How to solve the case of a three?

Directly kill the parent process, all the sub-processes are changed to become orphaned child processes are unified recovered from init.

4, inter-process memory space proof isolation

# 一个进程修改代码，肯定生效
x = 0
def task():
    global x
    x = 50
    print(x)

task()
print(x)

from multiprocessing import Process
# 一个进程修改代码，肯定生效
x = 0
def task():
    global x # 子进程修改的是自己的名称空间里的x，与主进程无关。
    x = 50
    print(f'子进程的x：{x}') 
if __name__ == '__main__':    
    p = Process(target=task)
    p.start()
    print(f'主进程的x:{x}')

5, Process of join usage

5.1 join specific usage

from multiprocessing import Process
import time
def task():
    print('子进程 开始')
    time.sleep(5)
    print('子进程 结束')
if __name__ == '__main__':
    start_time = time.time() # 记录开始时间
    p = Process(target=task)
    p.start()
    p.join() # 如遇join会阻塞该子进程，直到该子进程结束。并且join调用了wait方法释放僵尸进程。
    end_time = time.time() # 记录结束时间
    print(end_time-start_time) # 计算时间差
    print('主线程')
    
    '''输出：
    
    子进程 开始
    子进程 结束
    5.051951885223389
    
    主线程
    '''

5.2 Reflection

If there are multiple join is whether the serial?

from multiprocessing import Process
import time
def task(n):
    print('子进程 开始')
    time.sleep(n)
    print('子进程 结束')
if __name__ == '__main__':
    start_time = time.time() # 记录开始时间
    p1 = Process(target=task,args=(2,))
    p2 = Process(target=task,args=(4,))
    p3 = Process(target=task,args=(6,))

    p1.start()
    p2.start()
    p3.start()
    # 换顺序也是一样的也是按照时长最长的那个计算。
    p1.join() # 等待2s
    p2.join() # 等待2s
    p3.join() # 等待2s
    end_time = time.time() # 记录结束时间
    print(end_time-start_time) # 计算时间差
    print('主线程')



    '''输出：
    
    子进程 开始
    子进程 开始
    子进程 开始
    子进程 结束
    子进程 结束
    子进程 结束
    7.566513776779175
    主线程
    
    '''

5.3 join serial case

from multiprocessing import Process
import time
def task(n):
    print('子进程 开始')
    time.sleep(n)
    print('子进程 结束')
if __name__ == '__main__':
    start_time = time.time() # 记录开始时间
    p1 = Process(target=task,args=(2,))
    p2 = Process(target=task,args=(4,))
    p3 = Process(target=task,args=(6,))

    p1.start()
    p1.join() 
    p2.start()
    p2.join() 
    p3.start()
    p3.join() 
    end_time = time.time() # 记录结束时间
    print(end_time-start_time) # 计算时间差
    print('主线程')

    '''输出：
    
    子进程 开始
    子进程 结束
    子进程 开始
    子进程 结束
    子进程 开始
    子进程 结束
    15.407774925231934
    主线程
    
    '''
    # ps：反而不如不开进程，正常调用三次来的快。

5.4 Refining Code

from multiprocessing import Process
import time
def task(n):
    print('子进程 开始')
    time.sleep(n)
    print('子进程 结束')
if __name__ == '__main__':
    start_time = time.time() # 记录开始时间
    task_list = []
    for i in range(1,4):
        p = Process(target=task,args=(i,))
        p.start()
        task_list.append(p)
    print(task_list) # [<Process(Process-1, started)>, <Process(Process-2, started)>, <Process(Process-3, started)>]
    for i in task_list:
        i.join()
    end_time = time.time() # 记录结束时间
    print(end_time-start_time) # 计算时间差 4.764175891876221
    print('主线程')

6. Other uses of Process (understand)

6.1 pid Usage

'''
    在当前进程查看当前进程pid
        os.getpid()
        current_process().pid
    在当前进程查看子进程pid
        子进程对象.pid()
    在当前进程查看父进程pid
        os.getppid()   
'''

from multiprocessing import Process,current_process
import time,os
def task(x):
    print('进程开始')
    print('子进程对象的pid：',os.getpid()) # 在子进程中查看自己的pid
    print('子进程对象的pid：',current_process().pid) # 在子进程中查看自己的pid
    print('子进程对象的父进程对象的pid：',os.getppid()) # 查看父进程对象的pid
    time.sleep(100) # 注意要保证子进程不死，方便在cmd下测试。
    print('进程结束')

if __name__ == '__main__':
    p = Process(target=task,args=('子进程',))
    p.start()
    print('在父进程中查看子进程的pid',p.pid) # 在父进程中查看子进程的pid
    print('主进程自己查看自己的pid：',os.getpid()) # 父进程查看自己的pid
    print('主进程自己查看自己的pid：',current_process().pid) # 父进程查看自己的pid

6.2 name usage

from multiprocessing import Process,current_process

def task():
    print(current_process().name) # 在子进程查看自己的name

if __name__ == '__main__':
    p = Process(target=task) 
    p2 = Process(target=task)
    p3 = Process(target=task,name='rocky') # 已定义name属性为rocky
    p.start()
    p2.start()
    p3.start()
    print(p.name) #Process-1
    print(p2.name) #Process-2
    print(p3.name) #rocky

6.3 is_alive usage

from multiprocessing import Process,current_process
import time
def task():
    print('子进程开始')
    time.sleep(1)
    print('子进程结束')

if __name__ == '__main__':
    p = Process(target=task)
    p.start()
    # p.terminate() #发送一个指令给操作系统但是不会立即结束子进程
    # time.sleep(1)
    print(p.is_alive()) # True 判断子进程代码是否结束
    time.sleep(3)
    print(p.is_alive()) # False

7, daemon

Create a master daemon process

　　First: the daemon will terminate after the end of the main process code execution

　　Second: within daemon can no longer turn a child process, otherwise it throws an exception: AssertionError: daemonic processes are not allowed to have children

7.1 cases a

The main process code is finished running daemon ended immediately

from multiprocessing import Process
import time
def foo():
    print('守护进程开始')
    time.sleep(5)
    print('守护进程开始')


if __name__ == '__main__':
    p1 = Process(target=foo)
    p1.daemon = True # 一定要凡在start之前,表示设置为一个守护进程。
    p1.start()
    print('主进程')
    
    '''输出：
    主进程
    '''
    # 守护进程一旦发现主进程代码运行完，立刻结束，并不会管自己的进程是否运行完

7.2 cases two

After the main process code running in the child process and other operational phase, the thread will not participate Guardian

from multiprocessing import Process

import time
def foo():
    print('守护进程开始')
    time.sleep(5)
    print('守护进程结束')
def task():
    print('子进程开始')
    time.sleep(3)
    print('子进程开始')

if __name__ == '__main__':
    p1 = Process(target=foo)
    p2 = Process(target=task)
    p1.daemon = True # 一定要凡在start 之前
    p1.start()
    p2.start()
    # p2 = Process(target=task)
    print('主进程')

    '''输出：
    主进程
    子进程开始
    子进程开始
    '''
    #分析 守护进程在运行完主进程最后一行代码就结束，但是主进程并没有结束，主进程在等待子进程运行结束.

7.3 What is suitable for use daemon?

When the end of the main process code, then execute the child process meaningless situation can daemon.