[Python] Multi-process summary

every blog every motto: Light tomorrow with today.

0. Preface

This section conducts experiments and summarizes the multi-process in python.
Suggestion: first look at the concept of multithreading, this section will not repeat the concept, click me

1. Text

1.1 Decorator

Decorator, used to view the running time of the function, click me

import threading
import time
import functools
from multiprocessing import Process

c = 0


def count_time(func):
    """装饰器：计算程序运行时间"""

    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        t1 = time.time()
        res = func(*args, **kwargs)
        t2 = time.time()

        # 计算时间
        time_consumed = t2 - t1
        print('{}函数一共花费了{}秒'.format(func.__name__, time_consumed))
        return res

    return wrapper

1.2 Multi-process testing part

1.2.1 The tested function is two functions

The tested function is as follows:

@count_time
def mop_floor(arg):
    """拖地操作，"""
    global c
    print('开始拖地……')
    for i in range(3):
        time.sleep(1)
        c += 5
        c -= 5
        print('{} is running……c={} !!'.format(arg, c))
    print('---------地拖完了！-----------')


@count_time
def heat_water(arg):
    """烧水操作"""
    global c
    print('我要烧水了……')
    for i in range(4):
        time.sleep(1)
        print('{} is running……c={} !!'.format(arg, c))
    print('------------水烧开了！-------------')

1>. Single process

Single process function:

@count_time
def single_processing():
    """单进程"""
    # 对两个函数进行测试
    mop_floor('1')
    heat_water('2')
    print('-' * 100)

Main function:

def main():
    # 单进程
    single_processing()

if __name__ == '__main__':
    main()

Single process running results:

2>. Multi-process

Multi-process function

@count_time
def my_processing():
    p1 = Process(target=mop_floor, args=('1',))
    p2 = Process(target=heat_water, args=('2',))
    p1.start()
    p2.start()
    p1.join()
    p2.join()

    # -----------------------------------------
    # process_array = {}
    # n = 1
    # for tid in range(n):
    #     t = Process(target=single_func_tested, args=('1',))
    #     t.start()
    #     process_array[tid] = t
    # for i in range(n):
    #     process_array[i].join()

    print('-' * 100)

Main function

def main():

    # 多进程
    my_processing()


if __name__ == '__main__':
    main()

As a result, it can be seen that the total time is reduced

1.2.2 The function under test is a computational function

Function under test

@count_time
def single_func_tested(arg):
    """对一个函数进行测试"""
    sum = 0
    for i in range(10000000):  # 100000
        sum += i

    print('{} over'.format(arg))

1>. Single process

Single process function

@count_time
def single_processing():
    """单进程"""

    # 对一个函数进行测试
    single_func_tested('1')
    print('-' * 100)

Main function:

def main():
    # 单进程
    single_processing()

    # 多进程
    # my_processing()


if __name__ == '__main__':
    main()

Single-process calculation results:

2>. Multi-process

Multi-process function

@count_time
def my_processing():

    process_array = {
    
    }
    n = 1
    for tid in range(n):
        t = Process(target=single_func_tested, args=('1',))
        t.start()
        process_array[tid] = t
    for i in range(n):
        process_array[i].join()

    print('-' * 100)

Main function:

def main():


    # 多进程
    my_processing()


if __name__ == '__main__':
    main()

When n=1,

when n=2,

when n=3,

when n=4,

when n=5,

summary:

1. When n<=4, the total running time of the program does not change. This is because the experimental machine has 4 cores, so when there are less than 4 processes, the cores are enough. When n> 4, the running time of the program increases. The number of open processes exceeds the number of cores.
2. The conditions for opening the process are multiple CPUs, or one CPU has multiple cores
3. Multi-process for a single program will not shorten the running time of the program, just run a single program a few more times

1.2.3 The function being tested is a waiting function

Function under test

@count_time
def mop_floor(arg):
    """拖地操作，"""
    global c
    print('开始拖地……')
    for i in range(3):
        time.sleep(1)
        c += 5
        c -= 5
        print('{} is running……c={} !!'.format(arg, c))
    print('---------地拖完了！-----------')

1>. Single process

Single process function

@count_time
def single_processing():
    """单进程"""
    # 对两个函数进行测试
    mop_floor('1')
    # heat_water('2')

    # 对一个函数进行测试
    # single_func_tested('1')
    print('-' * 100)

Main function

def main():
    # 单进程
    single_processing()



if __name__ == '__main__':
    main()

result:

2>. Multi-process

Multi-process function:

@count_time
def my_processing():

    n = 1
    for tid in range(n):
        t = Process(target=mop_floor, args=('1',))
        t.start()
        process_array[tid] = t
    for i in range(n):
        process_array[i].join()

    print('-' * 100)

Main function:

def main():


    # 多进程
    my_processing()


if __name__ == '__main__':
    main()

When n=1,

when n=2,

when n=3,

when n=4,

when n=5,

when n=8,

summary:

1. When the process theory exceeds the core theory, the total running time increases slightly, and the increase is not as obvious as the computational function .
2. Multi-process for a single program will not shorten the running time of the program, just run a single program a few more times

the above.

---

to sum up:

1. The conditions for opening multiple processes are multiple CPUs or 1 multi-core CPU
2. Open multiple processes for a single program, no effect

1.3 Source code

https://gist.github.com/onceone/f5489dcc1499c81ee0161f0ea3bb442b

references

[1] https://blog.csdn.net/weixin_39190382/article/details/107107980
[2] http://www.uml.org.cn/python/201901221.asp
[3] https://www.jianshu.com/p/644dbb6d4cc8
[4] https://blog.csdn.net/m0_37324740/article/details/85765167
[5] https://www.cnblogs.com/-qing-/p/11291581.html
[6] https://blog.csdn.net/lzy98/article/details/88819425
[7] https://www.cnblogs.com/yssjun/p/11302500.html
[8] https://www.jb51.net/article/167165.htm
[9] https://blog.csdn.net/weixin_44850984/article/details/89165731
[10] https://www.cnblogs.com/justbreaking/p/7218909.html?utm_source=itdadao&utm_medium=referral