Second, in-depth asyncio coroutines (task object, coroutine call the principle of concurrent coroutines)

  Since it began to write blog, write notes before all yourself, so there may be expressed in clear, too winded Hello variety of problems, etc., have any questions or criticisms are welcome in the comments area.

If you are new coroutine, please read the article on acquaintance asyncio coroutine to asyncio have a preliminary understanding.

First, the task object (task assignment)

A reference to the acquaintance asyncio coroutine understand that the task object is an encapsulation of coroutine, which comprises various states, such as blocking state (Suspended), the operating state (running), completion status (DONE);

1. Create a task object in three ways

• The first:loop.create_task(xxx)

• The second:asyncio.ensure_future(xxx)

• Third:asyncio.create_task(xxx)

  In fact, the first method is used inside third method, however, it is noted that this method requires a cycle of events that have been run before creating the task, or will throwRuntimeError:no running event loop

2, add callback

import asyncio
async def get_url(url):
    print('start get_url')
    await asyncio.sleep(2)
    print('end get_url')
    return 'Joshua'

def callback(future):  # 回调函数
    print('Hello {}'.format(future.result()))

if __name__=="__main__":
    loop = asyncio.get_event_loop()
    task = loop.create(get_url('https://www.baidu.com'))
    task.add_done_callback(callback)  # 添加回调
    loop.run_until_complete(task)

3, pass parameters to the callback function

from functools import partial
# partial(偏函数)可以把函数包装成另外一个函数
import asyncio
async def get_url(url):
    print('start get_url')
    await asyncio.sleep(2)
    print('end get_url')
    return 'Joshua'

def callback(url,future):
    # 注意：要想向回调函数传递参数需要将参数放在future前面
    print('Hello {}'.format(future.result()))

if __name__=="__main__":
    loop = asyncio.get_event_loop()
    task = loop.create(get_url('https://www.baidu.com'))
    # 在传递前需要先用partial将函数封装成另外一个函数
    task.add_done_callback(partical(callback,'https://www.baidu.com'))

4, task cancel ()

To request cancellation Task object, which will then be thrown in the next round of the event loop CanceledError;

• Request cancellation:task.cancel()

• Determine whether to cancel:cancelled()

• Determine whether to end: ` done()`

• For example, the following

  import asyncio
  
  async def a():
      print('执行a()')
      await asyncio.sleep(3)
      print('执行a()完成')
  
  async def b():
      print('执行b()')
      await asyncio.sleep(2)
      print('执行b()完成')
  
  if __name__ == '__main__':
      loop = asyncio.get_event_loop()
      task1 = loop.create_task(a())
      task2 = loop.create_task(b())
      task1.cancel()  # 取消task1
      loop.run_until_complete(asyncio.gather(task1, task2))
  
  # 运行结果如下
  >>> 执行b()
  >>> Traceback (most recent call last):
    ... # 省略Traceback中的部分
  >>> concurrent.futures._base.CancelledError
  # 如果不想因为抛出异常而中断运行，可以在`gather`中设置`return_exception=True`
  # 如果设置`return_exception=True`,这样异常会和成功的结果一样处理，并聚合至结果列表，那么运行结果将如下:
  >>> 执行b()
  >>> 执行b()完成

  We can see a()not execute.

5, shield shield canceled

asyncio.shield( AW , **, Loop = None ), wait for protecting an object is canceled

• aw: Objects can wait, if coroutine will be internally packaged as Task

We canceled the first example above, amend as follows:

import asyncio

async def a():
    print('执行a()')
    await asyncio.sleep(3)
    print('执行a()完成')

async def b():
    print('执行b()')
    await asyncio.sleep(2)
    print('执行b()完成')

def c():
    loop = asyncio.get_event_loop()
    task1 = loop.create_task(a())
    task1 = asyncio.shield(task1)  # 设置屏蔽取消操作
    task2 = loop.create_task(b())
    task1.cancel()
    loop.run_until_complete(asyncio.gather(task1, task2, return_exceptions=True))

c()
# 运行结果如下
>>> 执行a()
>>> 执行b()
>>> 执行b()完成

See here still was canceled, little friends are certainly a (sand) face (carved) ignorant (Bo) force (Main), but it really do not blame me ah, it really is output, then why is it so ? We look at shield（）the source code, as follows:

def shield(arg, *, loop=None):
    inner = ensure_future(arg, loop=loop)  # 首先创建一个内部的Task 赋给inner
    if inner.done():  # 如果内部的Task执行完毕则返回inner
        # Shortcut.
        return inner
    loop = futures._get_loop(inner)
    outer = loop.create_future()  # 在事件循环上创建一个外部的Task

    def _done_callback(inner):  # 回调函数，在inner执行完毕后回调
        if outer.cancelled():  # 如果外部Task被取消
            if not inner.cancelled():  # 如果内部Task没被取消
                # Mark inner's result as retrieved.
                inner.exception()  # 根据注释的意思，内部Task的结果会被标记为已检索
            return

        if inner.cancelled():  # 如果内部的Task取消了，则外部的Task也将取消
            outer.cancel()  
        else:
            exc = inner.exception()  # 返回内部Task的异常
            if exc is not None:  # 如果存在异常，则外部Task的异常被设置为内部Task的异常
                outer.set_exception(exc)
            else:  # 否则将外部Task结果设置为内部Task的结果
                outer.set_result(inner.result())

    inner.add_done_callback(_done_callback)
    return outer  # 返回外部Task

Can be seen from the above source, running a real internal Task (inner), all external Task on the event loop comes from internal Task, and we canceled it is external Task on the event loop, not on the inside of the Task any impact, so when all Task execution is complete on the event loop exits, while the inside is still running the Task, which led to cancellation although we set up a shield or was canceled, however, that how to solve it?

The following two methods:

• Let canceled Task Task consuming consuming less than normal, which is to be canceled a()coroutine simulate blocking delay setting than b()small coroutine, so the masked canceled coroutine internal Task other than event loop normal Task execution is completed early; as follows:

  async def a():
      print('执行a()')
      await asyncio.sleep(1)  # 由原来的3改为1，使其小于b()的延时
      print('执行a()完成')
  
  async def b():
      print('执行b()')
      await asyncio.sleep(2)
      print('执行b()完成')

  Execution results are as follows:

  >>> 执行a()
  >>> 执行b()
  >>> 执行a()完成
  >>> 执行b()完成

  This can be seen under the a()coroutine execution completed.

• Let the event loop run continuously, it will not be on the Task event loop execution is completed, the program exits leading to internal Task masked canceled coroutine failed to complete the implementation; as follows:

  import asyncio
  import time
  async def a():
      print('执行a()')
      await asyncio.sleep(3)
      print('执行a()完成')
  
  async def b():
      print('执行b()')
      await asyncio.sleep(2)
      print('执行b()完成')
  
  if __name__ == '__main__':
      loop = asyncio.get_event_loop()
      task1 = loop.create_task(a())
      task2 = loop.create_task(b())
      loop.run_forever()
  # 输出如下
  >>> 执行a()
  >>> 执行b()
  >>> 执行b()完成
  >>> 执行a()完成

6, other methods of task objects (pasted from official documents)

• result()

  Return results of the Task. If the Task object has been completed , the result of which the packet will be returned coroutine (or when the coroutine throws an exception, the exception will be re-initiated.) If the Task object is canceled , this method raises an CancelledErrorexception. If the result of the Task object is not available, this method raises an InvalidStateErrorexception.

• exception()

  Return exceptions Task object. If the packet coroutine raises an exception, the exception will be returned. If the packet coroutine returns normally the method will return None. If the Task object is canceled , this method raises an CancelledErrorexception. If the Task object has not been completed , this method throws an InvalidStateErrorexception.

• remove_done_callback(callback)

  Removed from the list of callback callback specified callback. This method should only be used in low-level callback-based code. For more details, please see the Future.remove_done_callback()documents.

• get_stack(**, limit=None*)

  This stack frame returns a list of the Task object. If the packet is not completed coroutine, where it will return to its pending stack. If the coroutine has been successfully completed or canceled, it returns an empty list. If a coroutine is abnormally terminated, it returns back frame list. From the press frame is always from the old to the new order. Each suspended coroutine returns only one stack frame. The optional limit parameter specifies the maximum number of return frame; default return all frames. Returns a list of the order depends on a return or a stack traceback: returns the latest stack frame, back to return the oldest frame. (This is consistent with the behavior traceback module.)

• print_stack(**, limit=None, file=None*)

  Print this Task stack object or backtracking. This method produces an output similar to the traceback module get_stack()frame acquired. limit arguments are passed directly to get_stack(). file parameter is the output of the written I / O stream; default output is written sys.stderr.

• classmethod all_tasks(loop=None)

  Returns a collection of all tasks of an event loop. By default, it returns the current event loop in all tasks. If the loop is None, it will be used get_event_loop()to obtain the current event loop function. Method, IS the this deprecated and removed by Will BE in Python 3.9. The Use at The asyncio.all_tasks()function INSTEAD.

• classmethod current_task(loop=None)

  Returns the current task or run None. If the loop is None, it will be used get_event_loop()to obtain the current event loop function.

Second, the principle invoked coroutine

To better understand the concurrent execution of the back, so now we have to understand the principles under coroutine call, first give coroutine nested example:

import asyncio
import time
async def compute(x,y):
    await asyncio.sleep(1)
    return x+y

async def print_sum(x,y):
    reuslt = await compute(x,y)
    print(result)

if __name__ == "__main__":
    start = time.time()
    loop = asyncio.get_event_loop()
    loop.run_until_complete(print_sum(1,2))
    loop.close()
    print('Cost:{}'.format(time.time()-start))
    
#输出如下
>>> 3
>>> Cost:1.0007200241088867

Below is a timing chart of this example:

1. Create an Event Loopevent loop, call run_until_completestart event loop and to print_sumcreate a task object Task, Event Loopenter the running state, Task enters the pending state;

2. Task driven transport print_sumoperation to await compute(x,y)shift call sub coroutine compute, print_sumenter

   suspended state, the sub-coroutine computeenters the running state;

3. computeRun to await asyncio.sleep(1)the occurrence of blocking time-consuming, enter the suspended state, this time to communicate directly with the Task, and open up a Task channel (sub coroutine with the caller Task), can be roughly understood as similar to the sub-thread communication and process;

   The advantage of this is to ensure that print_sumis carried out simultaneously, and the overall program is run asynchronously, followed by examples to explain the problem.

4. Task communication with the child after turning coroutine and Event Loopcommunication, Event Loopthere is no other Task will only wait for the blocking operation is completed (sleep 1 second);

5. After waiting for time-consuming operation is complete blocking, Event Loopby Task coroutine sub computechannel wakeup compute, then computeenters the running state;

6. computeCode next step return x+y, done into the running state from the state, while thrown StopIterationexception and carry returnresults;

7. print_sumCapture computethrown exception, and acquires from which computethe results returned, this time print_sumfrom the suspended state to the running state, the next step of the codeprint(result)

8. In print_sumexecuting the printpost-converted done by the state running state, and throws StopIterationan exception;

9. Task captured print_sumafter thrown into an abnormal state runing done by the state;

10. In this case Event Loopenters stopped state;

For example explains Article 3:

import asyncio
import time
async def compute(x,y):
    await asyncio.sleep(1)
    return x+y

async def print_sum(x,y):
    result = await compute(x,y)
    print(result)

if __name__ == "__main__":
    start = time.time()
    loop = asyncio.get_event_loop()
    tasks = [print_sum(1,2),print_sum(1,2)]  # 创建两个协程对象
    loop.run_until_complete(asyncio.gather(*tasks))  # 同时执行两个协程对象
    loop.close()
    print('Cost:{}'.format(time.time()-start))
                            
#输出如下
>>>3
>>>3
>>>Cost:1.0004866123199463

As can be seen from the above examples is not performed when the uptime coroutine plurality of objects, in fact, asyncio.gatherpart of the action is to turn the object coroutine Task and register them to the event loop . In this example, steps 3 and 4 above explanation timing chart is not blocked waiting, call another but turned print_sumthe Task, is an asynchronous operation , so even if this method is theoretically 1000 coroutine execution objects still only Processed 1 second.

Let's look at an example:

import asyncio
import time
async def compute(x,y):
    print('执行compute')
    await asyncio.sleep(1)
    return x+y

async def print_sum(x,y):
    result1 = await compute(x, y)  # 调用子协程
    print(result1)
    result2 = await compute(x, y)  # 调用子协程
    print(result2)

if __name__ == "__main__":
    start = time.time()
    loop = asyncio.get_event_loop()
    loop.run_until_complete(print_sum(1,2))  # 同时执行两个协程对象
    loop.close()
    print('Cost:{}'.format(time.time()-start))
# 输出如下
>>> 执行compute
>>> 3
>>> 执行compute
>>> 3
>>> Cost:2.0158169269561768

In this case you can see the overall run time of 2 seconds, verify that print_sumthe synchronization is running, so the implementation of the two sub-coroutine doubling time consuming, if you want to complete it is still one second how to modify?

Nature is a synchronous operation into asynchronous operation, that is, let coroutine objects into a task object (Task), and the following:

import asyncio
import time
async def compute(x,y):
    print('执行compute')
    await asyncio.sleep(1)
    return x+y

async def print_sum(x,y):
    task1 = asyncio.create_task(compute(x, y))
    task2 = asyncio.create_task(compute(x, y))
    await task1
    print(task1.result())
    await task2
    print(task2.result())

if __name__ == "__main__":
    start = time.time()
    loop = asyncio.get_event_loop()
    loop.run_until_complete(print_sum(1,2))  
    loop.close()
    print('Cost:{}'.format(time.time()-start))
# 输出如下
>>> 执行compute
>>> 执行compute
>>> 3
>>> 3
>>> Cost:1.0004889965057373

In this there is a pit, we pay attention to avoid in writing, as follows:

async def print_sum(x,y):
    await asyncio.create_task(compute(x, y))
    await asyncio.create_task(compute(x, y))
    # 这样写依然是同步运行，要将生成的对象赋给一个变量再进行await，即上个例子那样

Three, asyncio concurrent

In several instances calling coroutine above principle, we actually have access to the knowledge portion of the concurrent, let's explain in detail asyncio concurrency.

asyncio complicated by several ways:

1. asyncio.gather(*aws, loop=None, return_exceptions=False)`

   • aws: You can wait for the object (coroutine, Task, Future), if an object is a coroutine can wait, then the asynico.gatherinterior will automatically convert it to a Task and add event loop. If all objects wait successfully completed, the result will be a list of all the return values from the same polymerization, and aws sequential order.
   • return_exceptionsWhen it is the default value Falsewhen aws if the object is canceled can wait, then throws a CancelledErrorprogram run ends; if it is set Truewhen the exception is thrown, the normal operation of the program, will return results in a list this result may object to waitconcurrent.futures._base.CancelledError()

2. asyncio.wait((aws, *, loop=None, timeout=None, return_when=ALL_COMPLETED)

   • aws: Objects can wait, run concurrently aws may wait a specified target and block the thread until meeting return_when specified conditions;

   • loop: This parameter can not ignore it, it will be described according to the official document python3.10to be removed in;

   • timeout: supports fractional and integer, if you set will be applied to the maximum number of seconds to wait before returning control, and will not lead TimeoutError, if timeout occurs Unfinished Task or Future returns;

   • return_when: Returns the conditions specified timing function:

     FIRST_COMPLETED: Returns a wait completion of the first objects;

     FIRST_EXCEPTION: The first occurrence of an exception return, if no abnormality is returned during waiting for all objects is completed;

     ALL_COMPLETED: By default, all waiting to return when the object is completed

asyncio.gatherAnd asyncio.waitthe difference between:

1. Both are used for concurrent coroutine, which asyncio.gatherreturns operation results coroutine asyncio.waitreturns two Task / Future list (coroutine and coroutine two lists is not completed is completed);

2. asyncio.gatherEquivalent to the entire black box, just to tell you the results of coroutines;

3. asyncio.gatherTask group may be, for example:

   import asyncio
   async def get_url(url):
      print('start get_url')
      await asyncio.sleep(2)
      print('end get_url')
      return 'Joshua'
   
   if __name__=="__main__":
      loop = asyncio.get_event_loop()
      # 将url分组
      group1 = [get_url('https://www.baidu.com') for _ in range(10)]
      group2= [get_url('https://www.baidu.com') for _ in range(10)]
      group1.cancel()  # 将第一组任务取消
      loop.run_until_complete(asyncio.gather(*group1,*group2))

4. asyncio.gatherWill return and complete implementation of the unfinished task, you can do some operations in its operation, such as the above return_whenreturns the actual conditions to control when to return the results if you want to perform their own needs with the result from the completion of the task which method take;

5. Get example returns the results as follows:

   tasks = [Task1,Task2,Task3]
   # gather的获取结果方法
   results = await asyncio.gather(*tasks)  # 注意此处gather方法不支持可迭代对象做参数，需要加*脱掉[]
   
   # wait的获取结果方法
   # 第一种
   done,pending = await asyncio.wait(tasks)  # 标准格式的wait方法
   results = [task.result() for task in done]
   # 第二种
   await asyncio.wait(tasks)
   results = [task.result() for task in tasks]