Efficiency comparison experiment of single thread, multi thread and multi process in Python

Python is a language that runs in an interpreter. Looking up information, I know that there is a global lock (GIL) in Python. When using multi-process (Thread), it cannot take advantage of multi-core. The use of multi-process (Multiprocess), you can take advantage of multi-core to really improve efficiency.

Comparative Experiment

According to the data, if the multi-threaded process is CPU-intensive , then multi-threading will not have much efficiency improvement. On the contrary, the efficiency may decrease due to frequent thread switching. Multi-process is recommended; if it is IO-intensive , Multi-threaded processes can use the idle time of IO blocking and waiting to execute other threads to improve efficiency. So we compare the efficiency of different scenarios based on experiments

operating system	CPU	RAM	hard disk
Windows 10	Dual-core	8GB	Mechanical hard disk

(1) Import the required modules

1 2 3 4

import requests import time from threading import Thread from multiprocessing import Process

(2) Define CPU-intensive calculation functions

1 2 3 4 5 6 7

def count (x, y) : # Make the program complete 500,000 calculations c = 0 while c < 500000: c += 1 x += x y += y

（3）定义IO密集的文件读写函数

1 2 3 4 5 6 7 8 9 10

def write(): f = open( "test.txt", "w") for x in range( 5000000): f.write( "testwrite\n") f.close() def read(): f = open( "test.txt", "r") lines = f.readlines() f.close()

(4) 定义网络请求函数

1 2 3 4 5 6 7 8 9 10

_head = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36'} url = "http://www.tieba.com" def http_request(): try: webPage = requests.get(url, headers=_head) html = webPage.text return { "context": html} except Exception as e: return { "error": e}

(5)测试线性执行IO密集操作、CPU密集操作所需时间、网络请求密集型操作所需时间

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

# CPU密集操作 t = time.time() for x in range( 10): count( 1, 1) print( "Line cpu", time.time() - t) # IO密集操作 t = time.time() for x in range( 10): write() read() print( "Line IO", time.time() - t) # 网络请求密集型操作 t = time.time() for x in range( 10): http_request() print( "Line Http Request", time.time() - t)

输出

• CPU密集：95.6059999466、91.57099986076355 92.52800011634827、 99.96799993515015
• IO密集：24.25、21.76699995994568、21.769999980926514、22.060999870300293
• 网络请求密集型: 4.519999980926514、8.563999891281128、4.371000051498413、4.522000074386597、14.671000003814697

(6)测试多线程并发执行CPU密集操作所需时间

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

counts = [] t = time.time() for x in range( 10): thread = Thread(target=count, args=( 1, 1)) counts.append(thread) thread.start() e = counts.__len__() while True: for th in counts: if not th.is_alive(): e -= 1 if e <= 0: break print(time.time() - t)

Output: 99.9240000248 、101.26400017738342、102.32200002670288

(7)测试多线程并发执行IO密集操作所需时间

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

def io(): write() read() t = time.time() ios = [] t = time.time() for x in range( 10): thread = Thread(target=count, args=( 1, 1)) ios.append(thread) thread.start() e = ios.__len__() while True: for th in ios: if not th.is_alive(): e -= 1 if e <= 0: break print(time.time() - t)

Output: 25.69700002670288、24.02400016784668

(8)测试多线程并发执行网络密集操作所需时间

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

t = time.time() ios = [] t = time.time() for x in range( 10): thread = Thread(target=http_request) ios.append(thread) thread.start() e = ios.__len__() while True: for th in ios: if not th.is_alive(): e -= 1 if e <= 0: break print( "Thread Http Request", time.time() - t)

Output: 0.7419998645782471、0.3839998245239258、0.3900001049041748

(9)测试多进程并发执行CPU密集操作所需时间

1 2 3 4 5 6 7 8 9 10 11 12 13 14

counts = [] t = time.time() for x in range( 10): process = Process(target=count, args=( 1, 1)) counts.append(process) process.start() e = counts.__len__() while True: for th in counts: if not th.is_alive(): e -= 1 if e <= 0: break print( "Multiprocess cpu", time.time() - t)

Output: 54.342000007629395、53.437999963760376

(10)测试多进程并发执行IO密集型操作

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

t = time.time() ios = [] t = time.time() for x in range( 10): process = Process(target=io) ios.append(process) process.start() e = ios.__len__() while True: for th in ios: if not th.is_alive(): e -= 1 if e <= 0: break print( "Multiprocess IO", time.time() - t)

Output: 12.509000062942505、13.059000015258789

(11) Test multi-process concurrent execution of Http request-intensive operations

t = time.time()
httprs = []
t = time.time()
for x in range(10):
    process = Process(target=http_request)
    ios.append(process)
    process.start()

e = httprs.__len__()
while True:
    for th in httprs:
        if not th.is_alive():
            e -= 1
    if e <= 0:
        break
print("Multiprocess Http Request", time.time() - t)

Output: 0.5329999923706055、0.4760000705718994

---

Experimental results

	CPU-intensive operations	IO-intensive operations	Network request-intensive operations
Linear operation	94.91824996469	22.46199995279	7.3296000004
Multithreaded operation	101.1700000762	24.8605000973	0.5053332647
Multi-process operation	53.8899999857	12.7840000391	0.5045000315

Through the above results, we can see:
>

• Multi-threading does not seem to have a great advantage under IO-intensive operations (maybe even more heavier tasks for IO operations can show advantages). Under CPU-intensive operations, it is obviously worse than single-threaded linear execution. But for network requests such as busy and other blocked thread operations, the advantage of multithreading is very significant
• Multi-processes can show performance advantages whether they are CPU-intensive or IO-intensive, and network request-intensive (operations that often occur thread blocking). However, in similar network request-intensive operations, it is not much different from multi-threading, but it takes up resources such as CPU, so in this case, we can choose multi-threaded execution
  

Original address
http://blog.atomicer.cn/2016/09/30/Python%E4%B8%AD%E5%A4%9A%E7%BA%BF%E7%A8%8B%E5%92%8C% E5%A4%9A%E8%BF%9B%E7%A8%8B%E7%9A%84%E5%AF%B9%E6%AF%94/