concurrent(Introduction)
The use of processes or threads.
concurrent (understand)
[Differences between thread processes]
-
Threads are shared memory spaces.
-
Processes are memory independent.
Threads in the same process can communicate directly with each other.
If two processes want to communicate, they must go through an intermediate agent.
Using multi-threading can make full use of the CPU to improve program execution efficiency.
When each process starts, it will first create a thread, the main thread, and then the main thread will create other child threads.
A process can contain multiple threads.
【Synchronous Asynchronous】
| Synchronize | After submitting a task, you must wait for the task to be executed (get the return value) before the next line of code can be executed. Equivalent to serial execution of tasks. |
| asynchronous | After handing in a task, you can directly execute the next line of code without waiting for execution to complete. |
concurrent (parameter list)
| concurrent.futures |
The module provides a highly encapsulated asynchronous calling interface. |
| ThreadPoolExecutor |
Thread pool, providing asynchronous calls |
| ProcessPoolExecutor |
Process pool, providing asynchronous calls |
| subprocess |
Subprocess operations |
| result(timeout=None) |
get results |
| add_done_callback(fn) |
Callback |
| submit(fn, *args, **kwargs) |
Submit tasks asynchronously |
| wait=True |
Wait for all tasks in the pool to be executed and resources to be reclaimed before continuing. |
| wait=False |
Return immediately and will not wait for the tasks in the pool to be completed. But regardless of the value of the wait parameter, the entire program will wait until all tasks are completed. |
| shutdown(wait=True) |
Equivalent to the pool.close()+pool.join() operation of the process pool |
| map(func, *iterables, timeout=None, chunksize=1) |
Replace the for loop submit operation. |
concurrent (reference code)
Simple multi-threaded operation
from concurrent.futures import ThreadPoolExecutor
import time
def task(i):
print(f"在执行任务{i}")
print("假设这是一个URL请求的地址....")
time.sleep(5)
pool = ThreadPoolExecutor(2) # 4个线程
for i in range(20):
pool.submit(task, i)Simplified version of the previous example (map)
# 例子-map用法
from concurrent.futures import ThreadPoolExecutor
from threading import Thread, currentThread
import time
def task(i):
print(f"{currentThread().name} 在执行任务{i}")
time.sleep(1)
if __name__ == "__main__":
pool = ThreadPoolExecutor(4)
pool.map(task, range(0, 20)) # map取代了for+submit
multiprocessing (introduction)
Advanced usage of processes.
multiprocessing (understand)
The multiprocessing-module is used to start child processes.
Function: Support child processes, communicate and share data, perform synchronization of different executions.
Provides -> Process, Queue, Pipe, Lock and other components.
multiprocessing (parameter list)
Process class |
|
| The object instantiated by this class can be used to start a child process. [Syntax] Process([group [, target [, name [, args [, kwargs]]]]]) The group parameter (ignored for now) is not used and the value is always None. target (function) represents the calling object, that is, the task to be performed by the child process. args (function parameters) represents the positional parameter tuple of the calling object, args=(1,2,'mike'). kwargs (ignored for now) represents the dictionary of the calling object, kwargs={'name':'mike','age':'18'}. |
Commonly used functions |
|
| start() |
(Start the process) Call run() in the subprocess |
| run() |
(Method run when the process starts) It is it that calls the function specified by target. |
| terminate() |
(Force terminate the process), no cleaning operation will be performed. If a child process is created, The child process becomes a zombie process. You need to be particularly careful when using this method. If a lock is still saved, it will not be released, resulting in a deadlock. |
| is_alive() |
The process is still running and returns True. |
| join([timeout]) |
The main process is waiting for termination (the main process is in a waiting state) |
| daemon |
The default value is False. If set to True, it means that the child process is a daemon process running in the background. When the parent process terminates, the child process also terminates and is set to True. |
| name |
process name |
| pid |
process pid |
multiprocessing (reference code)
single process
import multiprocessing
import time
def func(msg):
for i in range(20):
print(msg, i)
time.sleep(1)
if __name__ == "__main__":
p = multiprocessing.Process(target=func, args=("input content....",))
p.start()
p.join()
print("Sub-process done.")
multi-Progress
import multiprocessing
import time
def fc1(msg):
for i in range(3):
print("{}:黄岛主 正在进行任务{}....".format(msg, i + 1))
time.sleep(1)
print(">>>>>> {}大佬 任务完成".format(msg))
def fc2(msg):
for i in range(2):
print("{}:欧阳锋 正在进行任务{}....".format(msg, i + 1))
time.sleep(1)
print(">>>>>> {}大佬 任务完成".format(msg))
def fc3(msg):
for i in range(5):
print("{}:段大师 正在进行任务{}....".format(msg, i + 1))
time.sleep(1)
print(">>>>>> {}大佬 任务完成".format(msg))
def fc4(msg):
for i in range(2):
print("{}:洪七公 正在进行任务{}....".format(msg, i + 1))
time.sleep(1)
print(">>>>>> {}大佬 任务完成".format(msg))
if __name__ == "__main__":
pool = multiprocessing.Pool(processes=4) # CPU核数
result = []
person = ["东邪", "西毒", "南帝", "北丐"]
result.append(pool.apply_async(fc1, (person[0],)))
result.append(pool.apply_async(fc2, (person[1],)))
result.append(pool.apply_async(fc3, (person[2],)))
result.append(pool.apply_async(fc4, (person[3],)))
pool.close()
pool.join()
print("全部OK....")
multithread queue
import multiprocessing as mp
def job(x):
res = 0
for i in range(4):
res += i
x.put(res)
if __name__ == "__main__":
# 定义一个多线程队列，用来存储结果
q = mp.Queue()
# 定义两个线程函数，用来处理同一个任务
# Process / Thread
p1 = mp.Process(target=job, args=(q,))
p2 = mp.Process(target=job, args=(q,))
# 分别启动、连接两个线程
p1.start()
p2.start()
p1.join()
p2.join()
fc1 = q.get()
fc2 = q.get()
print(fc1 + fc2)
process lock
# 防止不同进程之间抢占共享资源
import multiprocessing as mp
import time
def job(v, num, l):
l.acquire() # 锁住
for _ in range(5):
time.sleep(1)
v.value += num # 获取共享内存
print("value=", v.value)
l.release() # 释放
def multicore():
lock = mp.Lock() # 定义一个进程锁
value = mp.Value("i", 20) # 定义共享内存
p1 = mp.Process(target=job, args=(value, 1, lock)) # 需要将lock传入
p2 = mp.Process(target=job, args=(value, 3, lock))
p1.start()
p2.start()
p1.join()
p2.join()
if __name__ == "__main__":
multicore()