python coroutine & asynchronous & asyncio

coroutine

Coroutines do not really exist in the computer, they are artificially created by programmers.

Coroutine, also known as micro-thread, is a context switching technology in user mode. In short, it actually implements code blocks to switch executions between each other through a thread, for example:

def fun1():
    print(1)
    ...
    print(2)

def fun2():
    print(3)
    ...
    print(4)


fun1()
fun2()

Coroutines enable the program to run cross-wise between the fun1() and fun2() functions.

Detailed description of coroutine

Coroutines are a programming pattern that supports non-blocking concurrency, which can be paused and resumed during execution, allowing the program to perform other tasks while waiting for certain operations to complete. Through coroutines, efficient concurrent and asynchronous programming can be achieved, computing resources can be effectively utilized, and the performance and responsiveness of the program can be improved.

The main features of coroutines are as follows:

Non-blocking concurrency: Coroutines allow the program to actively suspend and resume during execution without blocking the execution of other tasks. This means that you can switch to performing other tasks while waiting for certain operations to complete, allowing for concurrent processing.

Explicit suspend and resume: Coroutines implement suspend and resume operations through specific syntax and keywords. In Python, you can use the await keyword to pause the execution of a coroutine and wait for an asynchronous operation to complete before resuming execution.

Collaborative scheduling: Coroutines need to implement a mechanism for switching tasks through collaborative scheduling. The scheduler can switch execution between coroutines according to certain strategies, usually when the coroutine actively suspends.

Shared state: Coroutines can share state because they execute in the same thread. This means that communication and synchronization between coroutines can be achieved through shared variables.

Coroutines can be used to handle time-consuming tasks such as network requests, file reading and writing, and database operations. By using coroutines, the time spent waiting for operations can be fully utilized to perform other tasks, thereby improving the efficiency of the program.

In Python, the asyncio module provides support for coroutines, uses the async/await keywords to define coroutine functions, and provides a set of event loops and related tools to schedule and manage the execution of coroutines.

It should be noted that the coroutine is not a thread. It is executed in a single thread and uses the event loop to switch the execution of the coroutine. This makes coroutines highly efficient and scalable, but also requires careful handling of shared state and blocking operations to avoid blocking the execution of coroutines or creating race conditions.

Four ways to implement coroutines in python

• greenlet, early module.
• yield keyword.
• asyncio decorator (py3.4).
• async, await keywords (py3.5) [recommended].

1. greenlet
   greenlet is a third-party module. You need to install the module before using it.

pip3 install greenlet

Sample code:

from greenlet import greerflet

def func1():
	print (1)	＃第2步：输出 1
	
	gr2.switch ()	＃ 第3步：切换到 func2函数
	
	print (2)	＃ 第6步：输出 2

	gr2.switch ()	＃ 第7步：切换到 func2 西数，从上一次执行的位置继续向后执行
	
def func2():
	print (3)	＃第4步：输出 3
	
	gr1.switch ()	＃第5步：切换到 func1 函数，从上一次执行的位置继续向后执行
	
	print (4)	＃ 第8步：输出 4
	
	
gr1 = greenlet(func1)
gr2 = greenlet(func2)
gr1.switch(） ＃ 第1步：去执行 func1 函数

2. yield keyword

def func1():
	yield 1
	yield from func2()
	yield 2
	
def func2():
	yield 3
	yield 4

f1 = func1()
for item in f1():
	print(item)

# 打印结果 1 3 4 2

3. asynico
   in python3.4 and later versions.

import asyncio

@asyncio.coroutine 
def func1():
	print (1)
	yield from asyncio.sleep(2) # 遇到IO耗时操作，自动化切换到tasks中的其他任务
	print(2)

@asyncio.coroutine 
def func2):
	print (3)
	yield from asyncio.sleep(2）# 遇到IO耗时操作，自动化切换到tasks中的其他任务
	print (4)
	
tasks = [
	asyncio.ensure_future( func1() ), 
	asyncio.ensure_future( func2 () )
]

loop = asyncio.get_event_loop ()
loop.run_until_complete(asyncio.wait (tasks))

The function decorated by the @asyncio.coroutine decorator is a coroutine function, and the coroutine function must be placed in the event loop created by asyncio.get_event_loop ().
Note: Coroutines created through asyncio will automatically switch to other tasks in the event loop when encountering IO time consumption.

4. async, await keywords

import asyncio

async def func10):
	print (1)	＃ 网络I0请求：下载一张图片
	await asyncio.sleep(2)	＃ 遇到I0耗时操作，自动化切换到tasks中的其他任务
	print (2)
	
async def func20:
	print (3)	＃ 网络Io请求：下载一张图片
	await asyncio.sleep(2） # 遇到I0耗时操作，自动化切换到tasks中的其他任务
	print (4)
tasks = [
	asyncio.ensure_future( func1() ), asyncio.ensure_future ( func2() )
]
loop = asyncio.get_event_loop ()
loop. run_until_complete(asyncio.wait(tasks))

Coroutine meaning

在一个线程中如果遇到IO等待时间，线程不会阻塞，利用空闲时间去处理其他事情。

For example, if you download three pictures through a picture link, the traditional synchronization code will wait for the previous picture to be downloaded before starting the download of the next picture. When using the coroutine to download, after starting the download of the first picture, it will not wait until the download is completed before downloading the second picture. Instead, it will initiate the download of the second and third pictures. This method is based on the legendary Asynchronous programming with coroutines.
The pseudo code is as follows:

import aiohttp
import asynico

async def down_img(session, url):
	print("发送请求: ", url)
	async with session.get(url) as response:
		content = await response.content.read()
		with open(file_name, mode='wb') as f:
			f.write(content)
		print('下载完成:' ,url)

async def main()
	async with aiohttp.ClientSession() as session:
		url_list = [url1,url2,url3]
		tasks = [asyncio.create_task(down_img(session, url)) for url in url_list]
		await asyncio.wait(tasks)

if __name__ == 'main'
	async.run( main() )
	

Asynchronous programming

event loop

It can be understood as an infinite loop to detect and execute certain codes.

＃ 伪代码
任务列表 =〔任务1，任务2，任务了，……〕

while True :
	可执行的任务列表，已完成的任务列表 二去任务列表中检查所有的任务，将’可执行’和’已完成"的任务返回
	
	for 就绪任务 in 已准备就绪的任务列表：
		执行已就绪的任务
	
	for 已完成的任务 in 已完成的任务列表：
		在任务列表中移除 已完成的任务
	
	如果 任务列表 中的任务都已完成，则终止循环

# 去生成或获取一个事件循环
loop = asyncio.get_event_loop ()
# 将任务放到 ’任务列表‘
loop. run_until_complete(任务)

Get started quickly

Coroutine function, async def function name when defining the function.
Coroutine object, the coroutine object obtained by executing the coroutine function ().

import asyncio

# 协程函数
async def func():
	pass

# 协程对象
result = func()

Note: Executing a coroutine function creates a coroutine object, and the code inside the function will not be executed. If you want to run the code inside the coroutine function, you must hand the coroutine object to the event loop for processing.

import asyncio

# 协程函数
async def func():
	pass

# 协程对象
result = func()

# loop = asyncio.get_event_loop ()
# loop. run_until_complete(result )
asyncio.run( result )	# python3.7 后的写法，等同于上面两行代码。

await keyword

Literally, a means async and wait means waiting, which means waiting for execution.

await + awaitable object (coroutine object, Future, Task object->IO waiting)

Showing 1:

import asyncio

async def func():
	print("开始")
	response = await asyncio.sleep(2)
	print("结束"，response)

asyncio.run( func() )

When executing asyncio.run(func()), "start" will be printed first. When encountering code containing the await keyword in the function, if there are other unfinished tasks in the event loop, it will switch to other tasks until the IO blocking ends. , will execute "End".

import asyncio

async def others():
	print ("start")		# 2 执行协程函数内部代码
	await asyncio.sleep (2)
	print('end')	# 3 协程函数内部代码执行结束
	return '返回值'	
	
async def func() :
	print("执行协程函数内部代码")	# 1 进入主线程
	＃ 遇到I0操作挂起当前协程（任务），等IO操作完成之后再继续往下执行。当前协程挂起时，事件可以去执行其他协程（任务）。
	response = await others()
	print("Io请求结東，结果为：", response)	# 4 Io请求结東，结果为：'返回值'
	
asyncio.run( func() )

From Listing 2, we can see that coroutines can be nested in calls.

Showing 3:

import asyncio


async def others():
    print("start")  
    await asyncio.sleep(2)
    print('end')  
    return '返回值'


async def func():
    print("执行协程函数内部代码")  

    response1 = await others()
    print("Io请求结東，结果1为：", response1)  

    response2 = await others()
    print("Io请求结東，结果2为：", response2)  

asyncio.run(func())

执行协程函数内部代码
start
end
Io请求结東，结果1为： 返回值
start
end
Io请求结東，结果2为： 返回值

Process finished with exit code 0

From Listing 3, we can know that there can be multiple awaits in a coroutine function. And multiple awaits in a single coroutine function are serial, and the next await will not be executed until the previous await has a return value.

Task object

Tasks are used to schedule coroutines concurrently.
When a coroutine is wrapped into a Task with functions like asyncio.create task () the coroutine is automatically scheduled to run soon.

Vernacular: Adding multiple tasks to the event loop.
Tasks are used to schedule coroutines. Task objects are created through asyncio.create_task (coroutine object). This allows the coroutine to join the event loop and wait to be scheduled for execution. In addition to using asyncio.create_task() Western numbers, you can also use the low-level loop.create_task() or ensure_future() functions. It is not recommended to instantiate Task objects manually.

In coroutines, the task object is an abstract concept that represents the execution status of an asynchronous task. It is usually associated with a coroutine object to manage and control the coroutine as it executes. The task object has the following official explanation:

1. Task object is a subclass of Future object, which represents the execution status of an asynchronous operation.
2. Task objects have additional states and methods for managing and controlling the execution and cancellation of coroutines.
3. Task objects can be created from coroutines or dynamically created at runtime.
4. Task objects can be awaited to wait for an asynchronous operation to complete and return its result or raise an exception.
5. Task objects can be canceled to stop the execution of their associated coroutines and asynchronous operations.
6. The Task object can have a callback function attached to it, which will run automatically when the asynchronous operation is completed.

Showing 1:

import asyncio


async def func():
    print(1)
    await asyncio.sleep(2)
    print(2)
    return "返回值，"


async def main():
    print("main开始")
    # 创建Task对象，将当前执行func函数任务添加到事件循环。
    task1 = asyncio.create_task(func())
    # 创建Task对象，将当前执行func函数任务添加到事件循环。
    task2 = asyncio.create_task(func())
    print("main结束")
    # 当执行某协程遇到IO操作时，会自动化切换执行其他任务。
    # 此处的await是等待相对应的协程全都执行完毕井获取结果
    ret1 = await task1
    ret2 = await task2
    print(ret1, ret2)


asyncio.run(main())

main开始
main结束
1
1
2
2
返回值， 返回值，

Process finished with exit code 0

Showing 2:

Listing 1 above is a relatively basic way of writing, which is helpful for understanding the execution of coroutines. In development, the following way of writing is often used.

import asyncio


async def func():
    print(1)
    await asyncio.sleep(2)
    print(2)
    return "返回值，"


async def main():
    print("main开始")
    
    task_list = [
		asyncio.create_task(func(), name='t1'),
		asyncio.create_task(func(), name='t2')
	]
	
    print("main结束")
   
    done,pending = await asyncio.wait(task_list, timeout=None)
    print(done, pending )


asyncio.run(main())

main开始
main结束
1
1
2
2
{
    
    <Task finished name='t1' coro=<func() done, defined at E:\Project\PyCharmProject\Test\test\ts.py:4> result='返回值，'>, <Task finished name='t2' coro=<func() done, defined at E:\Project\PyCharmProject\Test\test\ts.py:4> result='返回值，'>} set()

Process finished with exit code 0

Listing 3:
In addition to Listing 2, there is another way to create a task object, which is just for understanding.

import asyncio

async def func() :
	print(1)
	await asyncio.sleep (2)
	print (2)
	return "返回值"

task_list = [
	func(), 	# 注意这个地方的格式是函数名＋（）,因为此时还没有事件循环。
	func(),
]
done, pending = asyncio.run( asyncio.wait(task_list) )
print (done)

Future object

A Future is a special low-level awaitable object that represents an eventual result of an
asynchronous operation.

Task inherits Future, and the processing of await results inside the Task object is based on the Future object. There is a _state variable inside the Future object to record the status of the Task object, and the await function is implemented through the _state status.

Showing 1:

async def main):
	# 获取当前事件循环
	1oop = asyncio.get_running_1oop0
	# 创建一个任务(Future对象），这个任务什么都不干。
	fut = loop. create_future ()
	# 等待任务最终结果 (Future对象），没有结果则会一直等下去。
	await fut
	
asyncio.run( main() )

Since the Future object fut has no result, the execution of the current program will always be stalled. This example mainly shows that await can also be followed by a Future object.

Showing 2:

import asyncio

async def set_after(fut):
	await asyncio.sleep (2)
	fut.set_result("666")
	
async def main():
	# 获取当前事件循环
	loop = asyncio.get_running_loop()
	# 创建一个任务(Future对象），没鄉定任何行为，则这个任务永远不知道什么时候结束。
	fut = loop.create_future()
	＃ 创建一个任务(Task对象），绑定了set_after函数，西数内部在25之后，会给fut赋值。
	# 即手动设置future任务的最终结果，那么fut就可以结東了。
	await loop.create_task(set_after (fut))
	# 等待 Future对象获取 最终结果，否则一直等下去
	data = await fut
	print (data)

asyncio.run( main() )

In Listing 2, the Future object will not wait until the await fut is executed because the value is set. When we use the task object, the underlying Future will automatically set_result the value returned by the coroutine function, achieving the same effect as above.

concurrent.futures.Future object

Objects used when implementing asynchronous operations using thread pools and process pools

import time
from concurrent. futures import Future
from concurrent.futures.thread import ThreadPoolexecutor from concurrent.futures.process import ProcessPoolexecutor

def func (value):
	time.sleep (1)
	print (value)
	return 123
	
# 创建线程池
pool = ThreadPoolexecutor (max_workers=5)

# 创建进程池
# pool = ProcesspoolExecutor(max_workers=5)

for i in range (10):
	fut = pool.submit (func, i)
	print (fut)

We know that when we use async, the third-party module used in coroutine coding must support coroutines. If there is a third-party module that does not support asynchronous, we can use the concurrent.futures.Future object in the thread pool and process pool. Implement asynchronously.

import time
import asyncio
import concurrent.rutures

def func1 ():
	# 某个耗时操作
	time.sleep (2)
	return "耗时操作"
	
asvnc def main:
	loop = asyncio.get_running_loop ()
	# 1. Run in the default loop's executor ( 默认 ThreadPoolexecutor )
	# 第一步：内部会先调用 ThreadpoolExecutor 的 submit 方法去线程池中申请一个线程去执行func1函数，井返回一个 concurrent.futures.Future对象
	# 第二步：调用asyncio.wrap_future将concurrent.futures.Future对象包装为asycio.Future对象。
	# 因为concurrent.futures.Future对象不支持await语法，所以需要包装为 asycio.Future对象 才能使用。
	fut = loop.run_in_executor(None, func1)
	result = await rut
	print ('default thread pool', result)
	
	# 2. Run in a custom thread pool:
	#with concurrent. futures. ThreadPoolexecutor () as pool:
	#	result = await loop.run_in_executor (pool, func1)
	#	print ('custom thread pool', result)
	# 3. Run in a custom process pool:
	# wilh concurrent. futures. ProcessPoolExecutor) as pool:
	#	result = await loop.run_in_executor
	#	pool, func1)
	#	print ('custom process pool', result)
	
asyncio.run(main()

Mixed case of asynchronous and non-asynchronous modules

import asyncio 
import requests
async def download_image(url):
	#发送网络请求，下载图片（遇到网络下载图片的工0请求，自动化切换到其他任务)
	print（"开始下载："，ur1)
	loop = asyncio.get_event_1oop()
	#requests模块默认不支持异步操作，所以就使用线程池来配合实现了。
	future = loop.run_in_executor(None, requests.get, url)
	response = await future
	print('下载完成'）
	＃ 图片保存到本地文件
	file_name = url.rsplit('_')[-1]
	with open (file_name, mode='w') as file_object:
		file_object.write(response.content)
	
if __name_ == '_ main_ ':
	url_list =[
	"https://www3.autoimg.cn/newsdfs/g26/M02/35/A9/120×90_0_autohomecar__ChsEel2AXQ6A00H_AAFOCMS
	8nzu621.jpg'
	"https://www2.autoimg.cn/newsdfs/g30/M01/3C/E2/120×90_0_autohomecar_chccsv2BBICAUntfAADjJFd
	6800429.jpg',
	'https://www.autoimg.cn/newsdfs/g26/MOB/3C/65/120×90_0_autohomecar__ChcCP12BFCmAI083AAGq7vK
	OsGY193.jpg'
	]
	tasks = [ download_image(url) for url in url_list]
	
	loop = asyncio.get_event_loop()
	loop.run_until_complete( asyncio.wait(tasks) )