Use fork() to create child process under linux
The Linux operating system provides a fork() function to create a child process. This function is very special. It is called once and returned twice, because the operating system copies the current process (parent process) (child process), and then Return within the parent and child processes, respectively. The child process always returns 0, while the parent process returns the child's PID (greater than 0). We can judge whether it is currently executing in the parent process or the child process by judging whether the return value is 0.
The fork() function is also provided in Python under the os module.
example:
import os
import time
ret = os.fork() #创建了一个子进程
if ret == 0: #子进程
while True:
print("-------1--------")
time.sleep(1)
else: #父进程
while True:
print("-------2--------")
time.sleep(1)The output is:
-------2--------
-------1--------
-------1--------
-------2--------
-------1--------
-------2--------
-------1--------
-------2--------
-------1--------
-------2--------
-------1--------
-------2--------
-------1--------
-------2--------Infinite loop. . . . . . (Note: The execution order of the child process and the parent process is uncertain, that is, the output order of 1 and 2 is not necessarily determined by the scheduling algorithm of the operating system)
Note: If the parent process ends, the child process also ends.
For example, the program:
import os
import time
ret = os.fork()
if ret == 0:
while True:
print("---1-----")
time.sleep(1)
else:
print("---2-----")The output is:
---2-----
---1-----Use Process to create child processes
You can use fork to create child processes under linux, but the fork function is not supported under windows, but you can use Process to create child processes
Note: Unlike fork, the main process will wait for the Process child process to complete and end.
The syntax structure of Process is as follows:
Process([group [, target [, name [, args [, kwargs]]]]])
target: Indicates the object called by this process instance;
args: A tuple of positional parameters representing the calling object;
kwargs: a dictionary of keyword arguments representing the calling object;
name: the alias of the current process instance;
group: not used in most cases;
Common methods of the Process class:
is_alive(): Determine whether the process instance is still executing;
join([timeout]): whether to wait for the execution of the process instance to end, or how many seconds to wait;
start(): start a process instance (create a child process);
run(): If the target parameter is not given, when the start() method is called on this object, the run() method in the object will be executed;
terminate(): terminates immediately regardless of whether the task is completed;
Common properties of the Process class:
name: alias of the current process instance, the default is Process-N, N is an integer starting from 1;
pid: the PID value of the current process instance;
from multiprocessing import Process
import time
import random
def test():
for i in range(1,5):
print("---%d---"%i)
time.sleep(1)
p = Process(target=test)
p.start() #让这个进程开始执行test函数里面的代码
p.join() #等进程p结束之后，才会继续向下走
print("---main----")The output is:
---1---
---2---
---3---
---4---
---main----Process pool creates child process
When the number of subprocesses to be created is not large, you can directly use the Process in multiprocessing to dynamically generate multiple processes, but if there are hundreds or even thousands of targets, the workload of manually creating processes is huge, and you can use to the Pool method provided by the multiprocessing module.
When initializing the Pool, you can specify a maximum number of processes. When a new request is submitted to the Pool, if the pool is not full, a new process will be created to execute the request; but if the number of processes in the pool is already When the specified maximum value is reached, the request will wait until a process in the pool ends, and a new process will be created for execution.
Analysis of common functions of multiprocessing.Pool:
apply_async(func[, args[, kwds]]) : use non-blocking method to call func (parallel execution, blocking mode must wait for the previous process to exit before executing the next process), args is the parameter list passed to func, kwds is passed a list of keyword arguments to func;
apply(func[, args[, kwds]]): call func in blocking mode
close(): Close the Pool so that it no longer accepts new tasks;
terminate(): terminates immediately regardless of whether the task is completed;
join(): The main process blocks, waiting for the exit of the child process, which must be used after close or terminate;
Procedure (non-blocking way):
from multiprocessing import Pool
import os
import time
def worker(num):
for i in range(2):
print("===pid=%d===num=%d"%(os.getpid(), num))
time.sleep(1)
pool = Pool(3) #定义一个进程池，最大进程数3
for i in range(5):
print("---%d---"%i)
pool.apply_async(worker, [i,]) #使用非阻塞方式调用func（并行执行，堵塞方式必须
#等待上一个进程退出才能执行下一个进程）
print("---start----")
pool.close() #关闭进程池，关闭后pool不能再添加新的请求
pool.join() #等待pool中所有子进程执行完成，必须放在close语句之后
print("---end----")output:
---0---
---1---
---2---
---3---
---4---
---start----
===pid=24958===num=0
===pid=24959===num=1
===pid=24960===num=2
===pid=24958===num=0
===pid=24960===num=2
===pid=24959===num=1
===pid=24960===num=3
===pid=24958===num=4
===pid=24960===num=3
===pid=24958===num=4
---end----Procedure (blocking mode):
from multiprocessing import Pool
import os
import time
def worker(num):
for i in range(2):
print("===pid=%d===num=%d"%(os.getpid(), num))
time.sleep(1)
pool = Pool(3) #定义一个进程池，最大进程数3
for i in range(5):
print("---%d---"%i)
pool.apply(worker, [i,]) #使用非阻塞方式调用func（并行执行，堵塞方式必须
#等待上一个进程退出才能执行下一个进程）
print("---start----")
pool.close() #关闭进程池，关闭后pool不能再添加新的请求
pool.join() #等待pool中所有子进程执行完成，必须放在close语句之后
print("---end----")output:
---0---
===pid=24999===num=0
===pid=24999===num=0
---1---
===pid=25000===num=1
===pid=25000===num=1
---2---
===pid=25001===num=2
===pid=25001===num=2
---3---
===pid=24999===num=3
===pid=24999===num=3
---4---
===pid=25000===num=4
===pid=25000===num=4
---start----
---end----Note: The process pool, like fork(), does not wait for the subroutine to end
example:
from multiprocessing import Pool
import os
import time
def worker():
for i in range(2):
print("===pid=%d==="%os.getpid())
time.sleep(1)
pool = Pool(3) #定义一个进程池，最大进程数3
for i in range(5):
print("---%d---"%i)
pool.apply_async(worker) #使用非阻塞方式调用func（并行执行，堵塞方式必须
#等待上一个进程退出才能执行下一个进程）output:
---0---
---1---
---2---
---3---
---4---