table of Contents
python day 20: the thread pool and coroutines
2019/11/1
Data from Old Boys Education
2. Thread
Thread IO suitable dense flow operation, the thread execution unit is the smallest
between the thread data is shared, one shared memory
import threading: import module thread
T of the threading.Thread = (target = FUNC, args = ())
t.start ( )
t.join ([timeout]), join a wait here meant that the main thread waits for the child thread end, and so on up to two seconds
the main process whether to wait for the child threads
t.setDaemon (True / False)
thread lock: use RLOCK
mutex = threading.RLock () to create a lock
mutex.acquire () to obtain a lock
mutex.release () to release the lock
event event
event_obj = threading.Event () Creates an event object
event_obj.wait () wait state, flag is False all threads are blocked
event_obj.clear () the flag is set to False
event_obj.set () will flal to True
3. Process
Create a process
Import multiprocessing
IF name == " main ":
the p-multiprocessing.Process = (target = FUNC, args = (),)
p.deamon = True / False (the main process whether to wait for the child process end)
p.start ()
the p- .join ([timeout]) main process waits for the child process end up waiting for the number of seconds
of data is not shared between processes, each using a separate memory
data sharing between processes or implementation Manage array
array
array must be a good start on the definition of an array the length of the
elements of the array must be unified data type of
the Manage ()
Manage the Manage = ()
manage.dict () no length limit, convenient than the array, the data communication between processes using manage.dict is the best choice.
Process pool Pool
the pool = multiprocessing.Pool (5), supports up to 5 to create a process of process pool.
pool.apply (func, (1,) ) to apply a process to perform a function. Each task is queued for execution. Each process has join execution.
pool.apply_async (func = Foo, args = (1,), callback = Bar), to apply a process to execute the method Foo, Foo return value as the method argument function assigned to the Bar. Each task is executed concurrently. More is to use apply_async. Each process does not join execution. Deamon process = True.
pool.close () process pool closed, no new request is received. Or pool.terminate ()
pool.join (), the pool emperor process process process is finished and then shut down.
4. coroutine: gevent module, also known as micro-threaded
Greenlet module must be installed gevent module
gevent representative of performance, when the IO request is issued, especially when the IO request the network, using coroutine
gevent.sleep (0)
gevent.joinall ()
from gevent import monkey; monkey.patch_all()
import requests
import gevent
def f(url):
print("url: %s"%url)
ret = requests.get(url)
data = ret.text
print(url,len(data))
gevent.joinall([
gevent.spawn(f,"https://juejin.im/post/5aa7314e6fb9a028d936d2a4"),
gevent.spawn(f,"https://www.cnblogs.com/lanxing0422/p/pythonday19.html"),
gevent.spawn(f,"http://www.baidu.com"),
])
# # 用于IO密集流
# def foo():
# print("foo")
# # 执行gevent的sleep方法
# gevent.sleep(0)
# print("foo again")
#
# def bar():
# print("bar")
# gevent.sleep(0)
# print("bar again")
#
# gevent.joinall([
# gevent.spawn(foo),
# gevent.spawn(bar),
# ])5. Extended
Producer Consumer Model
queue: Queue
Characteristics queue: FIFO
Q = Queue.Queue (max)
q.put ()
q.get ()
q.get_nowait
q.put_nowait ()
rabbit_m_queue open source, particularly regressed
6. Custom thread pool
Internal python does not provide, you need to customize.
Actual use, was based more on the thread pool to use, instead of creating a separate thread.
Very important, the future will always be used.
import threading
import queue
import time
import contextlib
Stopevent = object()
class ThreadPool(object):
'''
进程池
'''
def __init__(self, max_num):
self.max_num = max_num
self.q = queue.Queue()
self.terminate_flag = False
# 真实创建的线程列表
self.generate_list = []
# 空闲中的线程列表
self.free_list = []
def generate_thread(self):
'''
创建一个线程执行任务
:return:
'''
t = threading.Thread(target=self.call)
t.start()
def run(self, func, args, callback=None):
'''
线程池执行一个任务
:param func: 任务函数名
:param args: 任务函数所需参数元组
:param callback: 任务执行失败或成功后执行的回调函数
:return: 如果线程池已经终止，则返回True否则None
'''
if len(self.free_list) == 0 and len(self.generate_list) < self.max_num:
self.generate_thread()
tuple_obj = (func, args, callback)
self.q.put(tuple_obj)
def call(self):
'''
循环获取任务并执行
:return:
'''
# 获取当前线程并添加到列表中
current_thread = threading.currentThread()
self.generate_list.append(current_thread)
# 从队列中取任务
event = self.q.get()
# 当该任务的类型不是Stopevent时，死循环
while event != Stopevent:
func, args, callback = event
status = True
try:
ret = func(*args)
except Exception as e:
status = False
ret = e
# 当回调函数不为空时，就执行回调函数
if callback:
try:
callback(status, ret)
except Exception as e:
pass
# self.free_list.append(current_thread)
# event = self.q.get()
# self.free_list.remove(current_thread)
with self.work_state(self.free_list,current_thread):
if not self.terminate_flag:
event = self.q.get()
else:
event = Stopevent
else:
self.generate_list.remove(current_thread)
def close(self):
num = len(self.generate_list)
while num:
self.q.put(Stopevent)
num -= 1
def terminate(self):
self.terminate_flag = True
# 终止线程，不清空队列
# max = len(self.generate_list)
# while max:
# self.q.put(Stopevent)
# max -= 1
# self.q.empty()
# 终止线程且清空队列
while self.generate_list:
self.q.put(Stopevent)
self.q.empty()
@contextlib.contextmanager
def work_state(self,state_list,work_thread):
'''用于记录线程池中正在等待的线程数'''
state_list.append(work_thread)
try:
yield
finally:
state_list.remove(work_thread)
def foo(i):
time.sleep(0.1)
print("当前i是>>>",i)
return i + 100
def call_back(status, ret):
print("ret:>>>", ret)
pool = ThreadPool(10)
for i in range(50):
'''
将任务放进队列中：
创建线程：
只有空闲线程列表为空且创建的线程数量小于最大线程数时才会创建线程。
从队列中取到任务，线程执行任务
'''
pool.run(func=foo, args=(i,), callback=call_back)
pool.close()
print("gene_list",len(pool.generate_list))7. The multi-process server TCP
serSocket.setsockopt(SOL_SOCKET,SO_REUSEADDR,1)
Re-set socket options, reusable binding information
when there is a socket1 have the same local address and port TIME_WAIT state, while socket2 you start the program you want to occupy the address and port, you have to use the program SO_REUSEADDR options.
from socket import *
from multiprocessing import *
# 处理客户端的请求并为其服务
def dealwithClient(newSocket):
while True:
recvData = newSocket.recv(1024)
if len(recvData)==0:
break
else:
print(recvData)
newSocket.close()
def main():
serSock = socket(AF_INET,SOCK_STREAM)
# 设置套接字选项，使其可以重复使用绑定的信息
serSock.setsockopt(SOL_SOCKET,SO_REUSEADDR,1)
bindAddr= ('',9091)
serSock.bind(bindAddr)
serSock.listen(5)
try:
while True:
newSocket,clientAddr = serSock.accept()
p1 = Process(target= dealwithClient,args=(newSocket,))
p1.start()
# 因为已经向⼦进程中copy了⼀份（引⽤） ，
# 并且⽗进程中这个套接字也没有用处了
# 所以关闭
newSocket.close()
finally:
serSock.close()
if __name__=='__main__':
main()8. The multi-threaded TCP server
from socket import *
from threading import *
# 处理客户端的请求并为其服务
def dealwithClient(newSocket):
while True:
recvData = newSocket.recv(1024)
if len(recvData)==0:
break
else:
print(recvData)
newSocket.close()
def main():
serSock = socket(AF_INET,SOCK_STREAM)
# 设置套接字选项，使其可以重复使用绑定的信息
serSock.setsockopt(SOL_SOCKET,SO_REUSEADDR,1)
bindAddr= ('',9091)
serSock.bind(bindAddr)
serSock.listen(5)
try:
while True:
newSocket,clientAddr = serSock.accept()
p1 = Thread(target= dealwithClient,args=(newSocket,))
p1.start()
# 因为线程中共享这个套接字， 如果关闭了会导致这个套接字不可⽤，
# 但是此时在线程中这个套接字可能还在收数据， 因此不能关闭
# newSocket.close()
finally:
serSock.close()
if __name__=='__main__':
main()9. coroutine greenlet and gevent
⽐ threads to more smaller size YES unit (micro-threaded)
⼀ threads as ⼀ container can be placed ⾥ ⾯ plurality coroutine
Only switch function call multitasking can reduce the CPU switch
coroutine Your Own initiative to the CPU
Using generators, only to switch the function call to complete the task switching
import time
def A():
while True:
print(“----A---”)
yield
time.sleep(0.5)
def B(c):
while True:
print(“----B---”)
c.next()
time.sleep(0.5)
if __name__==‘__main__’:
a = A() # 如果一个函数中有yield，返回值就是一个生成器
B(a)# python中的greenlet模块对协程进行了封装（底层相当于yield）
# 安装模块： pip3 install greenlet
from greenlet import greenlet
import time
def t1():
while True:
print("........A........")
gr2.switch()
time.sleep(1)
def t2():
while True:
print("........b........")
gr1.switch()#调到上次执行的地方继续执行
time.sleep(1)
gr1 = greenlet(t1)#创建一个greenlet对象
gr2 = greenlet(t2)
gr1.switch()#此时会执行1函数There ⼀ a python ⽐ greenlet more can be zoomed and automatic switching module task gevent.
principle is that when a ⼀ greenlet encountered IO (input output means is the START input output) operation, such as access ⽐ Open networks, it automatically switch to other greenlet, IO wait until the operation is complete, then switch back to continue to perform at the appropriate time.
IO-intensive and cpu-intensive: some of the processes most of the time on a computer, called compute-intensive (cPU-intensive type), this time with multi-process.
there are a number of processes in the input and output spent most of the time, called I / O intensive. Most search engines such as waiting time (time-consuming operation), which belongs to the corresponding I / O intensive. This time with multithreading.
import gevent
def A():
while True:
print(".........A.........")
gevent.sleep(1)#用来模拟一个耗时操作
#gevent中：当一个协程遇到耗时操作会自动交出控制权给其他协程
def B():
while True:
print(".........B.........")
gevent.sleep(1)#每当遇到耗时操作，会自用转到其他协程
g1 = gevent.spawn(A) # 创建一个gevent对象（创建了一个协程），此时就已经开始执行A
g2 = gevent.spawn(B)
g1.join() #等待协程执行结束
g2.join() #会等待协程运行结束后再退出