Shared memory (IPC inter-process communication)
1. Pipeline When
one process writes and one process reads, the two processes do not involve data security issues. You can use pipes to achieve shared memory
2. Queue (pipe + lock)
Queue is used locally and cannot be based on network communication
# 队列代码实现
from multiprocessing import Queue
q = Queue(3) # 列表里面存的值跟取的值都是放入q里面的，Queue(3)表示队列里面最多只能放3个值，如果超过3个值，那么该队列则处于堵塞状态
队列里面放值
q.put([1, 2, 3])
q.put({
"a": 1})
q.put("xxxx")
q.put(100000)
q.put(1000, block=True, timeout=3) # block=True,可以通过timeout来指定超时时间，队列满了，超时会直接抛出异常
q.put(1000, block=False) # block=False，队列满了就直接抛出异常
队列里面取值
print(q.get())
print(q.get())
print(q.get())
print(q.get(block=False)) # block=False，取不到值就直接抛出异常
print(q.get(block=True, timeout=3)) # block=True,可以通过timeout来指定超时时间，取不到值的情况下，超时会直接抛出异常
Producer-consumer model
What is the producer-consumer model:
This model has two roles: one is a producer, the other is a consumer The
producer is responsible for generating data, and the consumer is responsible for taking the data for processing
Producers and consumers communicate through queues
Advantages: decoupling, balancing the productivity of the producer and the processing power of the consumer
生产者与消费者模型案例
方式1()
from multiprocessing import Process, Queue
import time
import random
def producer(q, name, food):
for i in range(3):
res = "%s %s" % (food, i)
time.sleep(random.randint(1, 3))
q.put(res)
print("%s 生成了 %s" % (name, res))
def consumer(q, name):
while True:
res = q.get()
if res is None:
break
time.sleep(random.randint(1, 3))
print("%s 吃了 %s" % (name, res))
if __name__ == '__main__':
q = Queue()
p1 = Process(target=producer, args=(q, "厨师1", "包子"))
p2 = Process(target=producer, args=(q, "厨师2", "烧卖"))
p3 = Process(target=producer, args=(q, "厨师3", "面条"))
c1 = Process(target=consumer, args=(q, "吃货1"))
c2 = Process(target=consumer, args=(q, "吃货2"))
p1.start()
p2.start()
p3.start()
c1.start()
c2.start()
p1.join()
p2.join()
p3.join()
q.put(None)
q.put(None)
print("主进程")
逻辑：先造产品，造好产品后，再运行主进程，主进程再增加两个None，按照队列依次取值的特性，最后两次取值只能取到None。
方式2
from multiprocessing import Process, JoinableQueue
import time
import random
def producer(q, name, food):
for i in range(3):
res = "%s %s" % (food, i)
time.sleep(random.randint(1, 3))
q.put(res)
print("%s 生成了 %s" % (name, res))
q.join() # 等队列的值都被取干净了，再结束
def consumer(q, name):
while True:
res = q.get()
time.sleep(random.randint(1, 3))
print("%s 吃了 %s" % (name, res))
q.task_done() # 每次执行一次取值都会发送一次信号
if __name__ == '__main__':
q = JoinableQueue()
p1 = Process(target=producer, args=(q, "厨师1", "包子"))
p2 = Process(target=producer, args=(q, "厨师2", "烧卖"))
p3 = Process(target=producer, args=(q, "厨师3", "面条"))
c1 = Process(target=consumer, args=(q, "吃货1"))
c2 = Process(target=consumer, args=(q, "吃货2"))
c1.daemon = True
c2.daemon = True
p1.start()
p2.start()
p3.start()
c1.start()
c2.start()
p1.join()
p2.join()
p3.join()
print("主进程")
Thread theory
Thread: the running process of a pipeline (the process of code running in the process is called a thread, which has nothing to do with resources)
A thread is an execution unit, and what cup executes is a thread. A
process is a resource unit
. Multiple threads in the same process share process resources; threads in different processes cannot share resources
The relationship and difference between threads and processes
1. Multiple threads in the same process share the memory resources of the process
2. The overhead of opening a child thread is much smaller than that of opening a child process
User space thread and kernel space thread
User space thread: User-level thread kernel switching is controlled by the user-mode program itself,
without kernel intervention, which reduces the consumption of getting in and out of the kernel mode, but it cannot make good use of multi-core cpu.
Kernel space thread: The switch is controlled by the kernel. When the thread switches, the user mode is
converted to the kernel mode. After switching, return to user mode from kernel mode.
Python belongs to the kernel space thread and is controlled by cup
Two ways to start threads
开启进程的方式1
进程的特性1：开启子线程的开销要远远小于开启子进程
from threading import Thread, current_thread # current_thread
def task():
print("%s is running" % current_thread().name) # 先打印这一行，pid是一样的
# current_thread().name 会获得当前线程的名字
if __name__ == '__main__':
t = Thread(target=task)
t.start()
print("主线程", current_thread().name) # 再打印这一行，pid是一样的
# current_thread().name 会获得当前线程的名字
进程的特性2：同一个进程下的多个线程共享该进程的内存资源
from threading import Thread
n = 100
def task():
global n
n = 0
if __name__ == '__main__':
t = Thread(target=task)
t.start()
t.join()
print("主线程", n)
开启进程的方式2
from threading import Thread
class Mythread(Thread):
def __init__(self, name):
super().__init__()
self.name = name
def run(self) -> None:
print("%s is running" % self.name)
if __name__ == '__main__':
t = Mythread("线程1")
t.start()
Thread related objects and methods
from threading import Thread, current_thread, active_count,enumerate
import time
def task():
print("%s is running" % current_thread().name)
time.sleep(5)
if __name__ == '__main__':
t = Thread(target=task)
t.start()
print(t.is_alive()) # is_alive()判断线程是否存活
print(active_count()) # 查看活着的线程数
print(enumerate()) # 显示了一个列表，里面是线程对象
Daemon thread (understand)
The daemon process guards the running time of the main process, and the daemon thread guards the life cycle of the main process.
Threads are the same as processes. The main thread also needs to wait for the child threads to end before the main thread ends. The
daemon process is the same as the daemon thread. Cannot start child process or child thread
from threading import Thread, current_thread, active_count
import os
import time
def task(n):
print("%s is run" % current_thread().name)
time.sleep(n)
print("%s is end" % current_thread().name)
if __name__ == '__main__':
t1 = Thread(target=task, args=(3,))
t2 = Thread(target=task, args=(5,))
t3 = Thread(target=task, args=(100,))
t3.daemon = True
t1.start()
t2.start()
t3.start()
print("主")
Mutex
Reduce the efficiency of data operation and protect data security.
from threading import Thread, Lock
import time
n = 100
mutex = Lock()
def task():
global n
with mutex:
teep = n
time.sleep(0.1)
n = teep - 1
if __name__ == '__main__':
thread_l = []
start_time = time.time()
for i in range(100):
t = Thread(target=task)
thread_l.append(t) # 把每一次运行的线程对象添加到列表里面（并发）
t.start()
for obj in thread_l:
obj.join() # 把线程对象依次加上join，这样一个线程抢到锁运行完后，下一次其他线程才能抢锁运行
print("主", n, time.time() - start_time)