Recursive locks, semaphores, GIL lock, socket-based communication and process multi-threaded thread pool pool
Recursive lock
Deadlock: it refers to the phenomenon of two or more processes and threads due to snatch computer resources resulting from a mutually waiting
from threading import Thread
from threading import Lock
import time
lock_A = Lock()
lock_B = Lock()
class MyThread(Thread):
def run(self):
self.f1()
self.f2()
def f1(self):
lock_A.acquire()
print(f"{self.name}拿到了A锁")
lock_B.acquire()
print(f"{self.name}拿到了B锁")
lock_B.release()
lock_A.release()
def f2(self):
lock_B.acquire()
print(f"{self.name}拿到了B锁")
time.sleep(0.1)
lock_A.acquire()
print(f"{self.name}拿到了A锁")
lock_A.release()
lock_B.release()
if __name__ == '__main__':
for i in range(3):
t = MyThread()
t.start()
# 结果：
Thread-1拿到了A锁
Thread-1拿到了B锁
Thread-1拿到了B锁
Thread-2拿到了A锁Recursive lock:
Recursive lock has a function of counting, the original number is 0, the lock time, count + 1, the lock is released once, counted -1,
As long as the above recursive digital lock is not zero, other threads can not grab the lock.
from threading import Thread
from threading import RLock
import time
lock_A = lock_B = RLock()
class MyThread(Thread):
def run(self):
self.f1()
self.f2()
def f1(self):
lock_A.acquire()
print(f"{self.name}拿到了A锁")
lock_B.acquire()
print(f"{self.name}拿到了B锁")
lock_B.release()
lock_A.release()
def f2(self):
lock_B.acquire()
print(f"{self.name}拿到了B锁")
time.sleep(0.1)
lock_A.acquire()
print(f"{self.name}拿到了A锁")
lock_A.release()
lock_B.release()
if __name__ == '__main__':
for i in range(3):
t = MyThread()
t.start()signal
Is also a lock, control the number of concurrent
from threading import Thread,Semaphore,current_thread
import time
import random
sem = Semaphore(5)
def task():
sem.acquire()
print(f"{current_thread().name}厕所ing")
time.sleep(random.randint(1,3))
print(f"{current_thread().name}厕所ed")
sem.release()
if __name__ == '__main__':
for i in range(20):
t = Thread(target=task,)
t.start()GIL lock
GIL lock definition:
The global interpreter lock is a mutex, it will become concurrent serial, only one thread at a time using an interpreter resources, sacrificing efficiency, ensure data security interpreter.
py file execution process in memory:
- Py file when executed, it will open a process in memory
- Process not only includes documents as well as python interpreter py, py file in the code thread will be handed over to the interpreter,
- Interpreter python code into C language code byte can be identified, and then to a virtual machine interpreter bytecode into a binary code to the CPU performs the final
When the first thread to get GIL lock thread 1 2 3 thread can only wait, when the execution thread 1 encounters an obstruction or a period of time the CPU executes, thread 1 will be suspended, while the GIL lock will be released at this time thread 2 or 3 thread will get locked into the interpreter, too, when faced with an obstruction or execution after a period of time the CPU executes suspended, while the GIL lock will be released at this time last thread will enter the interpreter.
As can be seen from the above, when faced with a single process containing a plurality of threads, due to the presence of the locks GIL, CPython not use multi-core parallel processing, but may be implemented concurrently on a single core.
But multi-threading between the different processes that can take advantage of multiple cores.
GIL lock the two effects:
1, to ensure the safety of the data inside the interpreter;
2, forced the lock, reducing development burdens
Question: multithreaded single process can not take advantage of multi-core
How to determine what circumstances the use of multi-threaded and multi-process concurrency
For calculation, cpu better, but for I / O, it is more useless cpu
Of course, running a program, the cpu with increased efficiency will certainly improve (no matter how much the magnitude of increase, there will always be improved), it is because a program is not substantially pure or pure computing I / O , it should be calculated to see a program in the end opposite the intensive or I / O intensive, as follows:
#分析：
我们有四个任务需要处理，处理方式肯定是要达到并发的效果，解决方案可以是：
方案一：开启四个进程
方案二：一个进程下，开启四个线程
#单核情况下，分析结果:
如果四个任务是计算密集型，没有多核来并行计算，方案一徒增了创建进程的开销，方案二胜
如果四个任务是I/O密集型，方案一创建进程的开销大，且进程的切换速度远不如线程，方案二胜
#多核情况下，分析结果：
如果四个任务是计算密集型，多核意味着并行计算，在python中一个进程中同一时刻只有一个线程执行,可以利用多核，方案一胜
如果四个任务是I/O密集型，再多的核也解决不了I/O问题，方案二胜
#结论：现在的计算机基本上都是多核，python对于计算密集型的任务开多线程的效率并不能带来多大性能上的提升，甚至不如串行(没有大量切换)，但是，对于IO密集型的任务效率还是有显著提升的。Summary: Under the premise of multi-core, if the IO-intensive tasks, the use of multi-threaded; if the task is computationally intensive, using more complicated process.
Verification of concurrent efficiency Cpython
- Compute-intensive: a single process multi-threaded concurrency of multiple processes in parallel vs
from threading import Thread
from multiprocessing import Process
import time
import random
if __name__ == '__main__':
# 多进程的并发,并行
start_time = time.time()
l1 = []
for i in range(4):
p = Process(target=task,)
l1.append(p)
p.start()
for p in l1:
p.join()
print(f'执行效率:{time.time()- start_time}') # 2.5342209339141846from threading import Thread
from multiprocessing import Process
import time
import random
if __name__ == '__main__':
# 多线程的并发
start_time = time.time()
l1 = []
for i in range(4):
p = Thread(target=task,)
l1.append(p)
p.start()
for p in l1:
p.join()
print(f'执行效率:{time.time()- start_time}') # 5.262923240661621Summary: Compute-intensive: high-efficiency multi-concurrent parallel processes.
- IO-intensive: Concurrent multi-threaded multiple processes in parallel in a single process vs
from threading import Thread
from multiprocessing import Process
import time
import random
def task():
count = 0
time.sleep(random.randint(1,3))
count += 1
if __name__ == '__main__':
# 多进程的并发,并行
start_time = time.time()
l1 = []
for i in range(50):
p = Process(target=task,)
l1.append(p)
p.start()
for p in l1:
p.join()
print(f'执行效率:{time.time()- start_time}') # 7.145753383636475from threading import Thread
from multiprocessing import Process
import time
import random
def task():
count = 0
time.sleep(random.randint(1,3))
count += 1
if __name__ == '__main__':
start_time = time.time()
l1 = []
for i in range(50):
p = Thread(target=task,)
l1.append(p)
p.start()
for p in l1:
p.join()
print(f'执行效率:{time.time()- start_time}') # 3.0278055667877197Summary: For IO intensive: high-efficiency multi-threaded concurrency of a single process.
Based on multi-threaded socket communication
Client:
import socket
client = socket.socket()
client.connect(('127.0.0.1',8848))
while 1:
try:
to_server_data = input('>>>').strip()
client.send(to_server_data.encode('utf-8'))
from_server_data = client.recv(1024)
print(f'来自服务端的消息: {from_server_data.decode("utf-8")}')
except Exception:
break
client.close()Server:
import socket
from threading import Thread
def communicate(conn,addr):
while 1:
try:
from_client_data = conn.recv(1024)
print(f'来自客户端{addr[1]}的消息: {from_client_data.decode("utf-8")}')
to_client_data = input('>>>').strip()
conn.send(to_client_data.encode('utf-8'))
except Exception:
break
conn.close()
def _accept():
server = socket.socket()
server.bind(('127.0.0.1', 8848))
server.listen(5)
while 1:
conn, addr = server.accept()
t = Thread(target=communicate,args=(conn,addr))
t.start()
if __name__ == '__main__':
_accept()Process pool thread pool
Thread pool: a container, the container locked you turn the number of threads, such as 4, processing four concurrent tasks for the first time can only be sure of, as long as the task is completed, the thread will soon take over a task.
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor
import os
import time
import random
# print(os.cpu_count()) 获取cpu数量
def task(n):
print(f'{os.getpid()} 接客')
time.sleep(random.randint(1,3))
# 开启进程池 (并行(并行+并发))
if __name__ == '__main__':
p = ProcessPoolExecutor() # 进程池，默认不写，开启数量为cpu数量
for i in range(20):
p.submit(task,i)
# 开启线程池 (并发)
t = ThreadPoolExecutor() # 默认不写, cpu个数*5 线程数
# t = ThreadPoolExecutor(100) # 100个线程
for i in range(20):
t.submit(task,i)