Python3 concurrent programming 4

Event Event

• Used to control the execution of threads
• e.isSet()See e current signal state objects, the default is False
• e.wait() Signal status is False, then the current thread is blocked
• e.set() The signal state e is set to True, the blocked thread becomes unblocked

from threading import Thread
from threading import Event
import time

e = Event()


def light():
    print('*********红灯!**********')
    print(f'对象当前的信号状态为{e.isSet()}')
    time.sleep(5)
    print('*********绿灯!**********')
    e.set()  # 将e的信号标准设为True
    print(f'对象当前的信号状态为{e.isSet()}')


def driver(name):
    print(f'{name}正在等红灯!')
    e.wait()  # 如果e信号标志为False, 则当前线程阻塞
    print(f'{name}弹射起步!')


if __name__ == '__main__':

    t1 = Thread(target=light)
    t1.start()

    for i in range(10):
        t2 = Thread(target=driver, args=(f'老司机{i+1}号',))
        t2.start()

        
'''
**********红灯!**********
对象当前的信号状态为False
老司机1号正在等红灯!
老司机2号正在等红灯!
老司机3号正在等红灯!
老司机4号正在等红灯!
老司机5号正在等红灯!
老司机6号正在等红灯!
老司机7号正在等红灯!
老司机8号正在等红灯!
老司机9号正在等红灯!
老司机10号正在等红灯!
**********绿灯!**********
对象当前的信号状态为True
老司机1号弹射起步!
老司机5号弹射起步!
老司机7号弹射起步!
老司机8号弹射起步!
老司机9号弹射起步!
老司机6号弹射起步!
老司机4号弹射起步!
老司机2号弹射起步!
老司机10号弹射起步!
老司机3号弹射起步!
'''

Thread pool and process pool

basic concepts

• Used to control the number of the current program allows you to create a process / thread
• Process to prevent the program creates too much / thread, more than affordable range of hardware

Instructions

• pool = ProcessPoolExecutor(5) The current maximum of five tasks simultaneously open process, the default number of threads is the number of CPU
• pool = ThreadPoolExecutor(5) The current maximum of five tasks simultaneously open thread, the default number of threads is the number of CPU * 5
• pool.submit(函数地址, 参数) Submit jobs
• pool.submit(函数地址, 参数).add_done_callback(回调函数地址) Submit the task, and the task of the return value passed to the callback function
• pool.shutdown() Let the thread pool tasks are executed after their execution of code down

from concurrent.futures import ThreadPoolExecutor
import time

pool = ThreadPoolExecutor(5)


# 送快递
def deliver(goods):
    print(f'{goods}开始发货!')
    time.sleep(1)
    print(f'{goods}已经签收!')
    return True


# 拿快递(回调函数)
def get_goods(res):
    get = res.result()
    if get:
        print('开始拆快递!')
    else:
        print('投诉!')


for i in range(5):
    pool.submit(deliver, f'格子衬衫牛仔裤{i+1}').add_done_callback(get_goods)

# 让线程池中的线程运行完毕再执行下面的代码
pool.shutdown()
print('钱包空空!')


'''
格子衬衫牛仔裤1开始发货!
格子衬衫牛仔裤2开始发货!
格子衬衫牛仔裤3开始发货!
格子衬衫牛仔裤4开始发货!
格子衬衫牛仔裤5开始发货!
格子衬衫牛仔裤1已经签收!
开始拆快递!
格子衬衫牛仔裤2已经签收!
开始拆快递!
格子衬衫牛仔裤5已经签收!
开始拆快递!
格子衬衫牛仔裤4已经签收!
开始拆快递!
格子衬衫牛仔裤3已经签收!
开始拆快递!
钱包空空!
'''

When we want to open 10 threads, both for i in range(10):results are as follows

'''
格子衬衫牛仔裤1开始发货!
格子衬衫牛仔裤2开始发货!
格子衬衫牛仔裤3开始发货!
格子衬衫牛仔裤4开始发货!
格子衬衫牛仔裤5开始发货!
格子衬衫牛仔裤2已经签收!
开始拆快递!
格子衬衫牛仔裤6开始发货!
格子衬衫牛仔裤1已经签收!
开始拆快递!
格子衬衫牛仔裤7开始发货!
格子衬衫牛仔裤4已经签收!
格子衬衫牛仔裤3已经签收!
开始拆快递!
格子衬衫牛仔裤8开始发货!
格子衬衫牛仔裤5已经签收!
开始拆快递!
格子衬衫牛仔裤9开始发货!
开始拆快递!
格子衬衫牛仔裤10开始发货!
格子衬衫牛仔裤6已经签收!
开始拆快递!
格子衬衫牛仔裤10已经签收!
开始拆快递!
格子衬衫牛仔裤7已经签收!
开始拆快递!
格子衬衫牛仔裤8已经签收!
开始拆快递!
格子衬衫牛仔裤9已经签收!
开始拆快递!
钱包空空!
'''

And the amount of the difference signal

• Semaphore: our own worker thread is created, we need to manually limiting
• Thread pool: thread pool worker thread is created, the thread pool automatically limiting

Coroutine (the coroutine)

basic concepts

• In single-threaded implementation of concurrency (save switching +)
• A thread is a system level, by the operating system scheduler. Coroutine is a program level, programmers need their own schedule
• Advantages: no context switching overhead, saving time and space
• Disadvantages: can not use the advantages of multi-core, a blocking operation will block the entire program

Method to realize

• yield achieved

import time


# 生成器
def consumer():
    r = ''
    while True:
        n = yield r
        if not n:
            return
        print(f'[CONSUMER] consuming {n}...')
        time.sleep(1)
        r = '200 OK'


def producer(c):
    c.__next__()  # 初始化生成器
    n = 0
    while n < 5:
        n = n + 1
        print(f'[PRODUCER] producing {n}...')
        r = c.send(n) # 切换到consumer执行
        print(f'[PRODUCER] consumer return: {r}')
    c.close()


if __name__ == '__main__':
    # c是生成器对象
    c = consumer()
    producer(c)
    
'''
[PRODUCER] producing 1...
[CONSUMER] consuming 1...
[PRODUCER] consumer return: 200 OK
[PRODUCER] producing 2...
[CONSUMER] consuming 2...
[PRODUCER] consumer return: 200 OK
[PRODUCER] producing 3...
[CONSUMER] consuming 3...
[PRODUCER] consumer return: 200 OK
[PRODUCER] producing 4...
[CONSUMER] consuming 4...
[PRODUCER] consumer return: 200 OK
[PRODUCER] producing 5...
[CONSUMER] consuming 5...
[PRODUCER] consumer return: 200 OK
'''

• gevent module implements

from gevent import monkey;
monkey.patch_all()  # 猴子补丁, 修改Python一些标准库
from gevent import spawn, joinall
import time


def func1():
    print('1')
    time.sleep(1)


def func2():
    print('2')
    time.sleep(2)


def func3():
    print('3')
    time.sleep(3)


start_time = time.time()

s1 = spawn(func1)
s2 = spawn(func2)
s3 = spawn(func3)

joinall([s1, s2, s3])

end_time = time.time()

print(end_time - start_time)


'''
1
2
3
3.007172107696533
'''

High-performance video crawling pear