Due to the well-known GIL, Python's threads cannot exert the parallel computing power of multi-core (of course, with multiprocessing, which can realize multi-process parallelism), it seems rather tasteless. Since under the GIL, only one thread can be running at the same time, for CPU-intensive programs, the switching overhead between threads becomes a drag, and the program with I/O as the bottleneck is exactly what the coroutine is. good at:
Multi-task concurrency (non-parallel), each task suspends (initiates I/O) and resumes (I/O ends) at the appropriate time
Coroutines in Python have come a long way. It has roughly gone through the following three stages:
1 Initial generator deformation yield/send
2 Introduce @asyncio.coroutine and yield from
3 Introduce the async/await switch in the recent Python 3.5 version
from yield
Let's first look at a common code for calculating Fibonacci continuations:
def old_fib(n):
res = [0] * n
index = 0
a = 0
b = 1
while index < n:
res[index] = b
a, b = b, a + b
index += 1
return res
print('-'*10 + 'test old fib' + '-'*10)
for fib_res in old_fib(20):
print(fib_res)If we just need to get the nth digit of the Fibonacci sequence, or just want to generate the Fibonacci sequence based on it, then the above traditional method will be more memory-intensive.
This is where yield comes in handy.
def fib(n):
index = 0
a = 0
b = 1
while index < n:
yield b
a, b = b, a + b
index += 1
print('-'*10 + 'test yield fib' + '-'*10)
for fib_res in fib(20):
print(fib_res)When a function contains a yield statement, python automatically recognizes it as a generator. At this time, fib(20) does not actually call the function body, but generates a generator object instance with the function body.
yield is here to keep the calculation scene of the fib function, suspend the calculation of fib and return b. When fib is placed in the for...in loop, next(fib(20)) will be called every time the loop is invoked to wake up the generator and execute to the next yield statement until the StopIteration exception is thrown. This exception will be caught by the for loop, resulting in a break out of the loop.
Send is here
As you can see from the above program, currently only data flows from fib(20) to the outside for loop through yield; if you can send data to fib(20), then you can implement coroutines in Python.
Therefore, the generator in Python has the send function, and the yield expression also has the return value.
We use this feature to simulate the calculation of a slow Fibonacci sequence:
def stupid_fib(n):
index = 0
a = 0
b = 1
while index < n:
sleep_cnt = yield b
print('let me think {0} secs'.format(sleep_cnt))
time.sleep(sleep_cnt)
a, b = b, a + b
index += 1
print('-'*10 + 'test yield send' + '-'*10)
N = 20
sfib = stupid_fib(N)
fib_res = next(sfib)
while True:
print(fib_res)
try:
fib_res = sfib.send(random.uniform(0, 0.5))
except StopIteration:
breakwhere next(sfib) is equivalent to sfib.send(None), which can make sfib run to the first yield and return. Subsequent sfib.send(random.uniform(0, 0.5)) will send a random number of seconds to sfib as the return value of the currently interrupted yield expression. In this way, we can control the thinking time of the coroutine when calculating the Fibonacci sequence from the "main" program, and the coroutine can return the calculation result to the "main" program, Perfect!
What the hell is yield from?
yield from is used to refactor generators. Simple, it can be used like this:
def copy_fib(n):
print('I am copy from fib')
yield from fib(n)
print('Copy end')
print('-'*10 + 'test yield from' + '-'*10)
for fib_res in copy_fib(20):
print(fib_res)This usage is simple, but far from what yield from is all about. The role of yield from also reflects that the send information can be passed to the inner coroutine like a pipeline, and various exceptions can be handled. Therefore, stupid_fib can also be packaged and used like this:
def copy_stupid_fib(n):
print('I am copy from stupid fib')
yield from stupid_fib(n)
print('Copy end')
print('-'*10 + 'test yield from and send' + '-'*10)
N = 20
csfib = copy_stupid_fib(N)
fib_res = next(csfib)
while True:
print(fib_res)
try:
fib_res = csfib.send(random.uniform(0, 0.5))
except StopIteration:
breakIf there is no yield from, the copy_yield_from here will be particularly complicated (because you have to handle various exceptions yourself).
asyncio.coroutine和yield from
yield from is carried forward in the asyncio module. Look at the sample code first:
@asyncio.coroutine
def smart_fib(n):
index = 0
a = 0
b = 1
while index < n:
sleep_secs = random.uniform(0, 0.2)
yield from asyncio.sleep(sleep_secs)
print('Smart one think {} secs to get {}'.format(sleep_secs, b))
a, b = b, a + b
index += 1
@asyncio.coroutine
def stupid_fib(n):
index = 0
a = 0
b = 1
while index < n:
sleep_secs = random.uniform(0, 0.4)
yield from asyncio.sleep(sleep_secs)
print('Stupid one think {} secs to get {}'.format(sleep_secs, b))
a, b = b, a + b
index += 1
if __name__ == '__main__':
loop = asyncio.get_event_loop()
tasks = [
asyncio.async(smart_fib(10)),
asyncio.async(stupid_fib(10)),
]
loop.run_until_complete(asyncio.wait(tasks))
print('All fib finished.')
loop.close()asyncio is an event loop based module for implementing asynchronous I/O. Through yield from, we can hand over control of the coroutine asyncio.sleep to the event loop, and then suspend the current coroutine; after that, the event loop decides when to wake up asyncio.sleep, and then execute the code backwards.
This may be abstract. Fortunately, asyncio is a module implemented by python, so let's take a look at what asyncio.sleep has done:
@coroutine
def sleep(delay, result=None, *, loop=None):
"""Coroutine that completes after a given time (in seconds)."""
future = futures.Future(loop=loop)
h = future._loop.call_later(delay,
future._set_result_unless_cancelled, result)
try:
return (yield from future)
finally:
h.cancel()First, sleep creates a Future object as an inner coroutine object, which is handed over to the event loop through yield from; second, it registers a callback function by calling the call_later function of the event loop.
By looking at the source code of the Future class, you can see that Future is a generator that implements the __iter__ object:
class Future:
#blabla...
def __iter__(self):
if not self.done():
self._blocking = True
yield self # This tells Task to wait for completion.
assert self.done(), "yield from wasn't used with future"
return self.result() # May raise too.Then when our coroutine yields from asyncio.sleep, the event loop is actually an exercise with the Future object. Every time the event loop calls send(None), it will actually be passed to the __iter__ function call of the Future object; and when the Future has not finished executing, it will yield self, which means temporarily suspending, waiting for the next send (None) wakeup.
When we wrap a Future object to generate a Task object, in the initialization of the Task object, the Future's send(None) is called, and the callback function is set for the Future.
class Task(futures.Future):
#blabla...
def _step(self, value=None, exc=None):
#blabla...
try:
if exc is not None:
result = coro.throw(exc)
elif value is not None:
result = coro.send(value)
else:
result = next(coro)
#exception handle
else:
if isinstance(result, futures.Future):
# Yielded Future must come from Future.__iter__().
if result._blocking:
result._blocking = False
result.add_done_callback(self._wakeup)
#blabla...
def _wakeup(self, future):
try:
value = future.result()
except Exception as exc:
# This may also be a cancellation.
self._step(None, exc)
else:
self._step(value, None)
self = None # Needed to break cycles when an exception occurs.After a preset amount of time, the event loop will call Future._set_result_unless_cancelled:
class Future:
#blabla...
def _set_result_unless_cancelled(self, result):
"""Helper setting the result only if the future was not cancelled."""
if self.cancelled():
return
self.set_result(result)
def set_result(self, result):
"""Mark the future done and set its result.
If the future is already done when this method is called, raises
InvalidStateError.
"""
if self._state != _PENDING:
raise InvalidStateError('{}: {!r}'.format(self._state, self))
self._result = result
self._state = _FINISHED
self._schedule_callbacks()This will change the state of the Future and call back the previously set Tasks._wakeup; in _wakeup, Tasks._step will be called again, at this time, the state of the Future has been marked as complete, so it will no longer yield self, The return statement will trigger a StopIteration exception, which will be caught by Task._step to set the result of the Task. At the same time, the entire yield from chain will also be awakened, and the coroutine will continue to execute.
async和await
After figuring out asyncio.coroutine and yield from, async and await introduced in Python 3.5 are not difficult to understand: they can be understood as a perfect stand-in for asyncio.coroutine/yield from. Of course, from the perspective of Python design, async/await allows coroutines to exist on the surface independently of generators, hides the details under the asyncio module, and makes the syntax clearer.
async def smart_fib(n):
index = 0
a = 0
b = 1
while index < n:
sleep_secs = random.uniform(0, 0.2)
await asyncio.sleep(sleep_secs)
print('Smart one think {} secs to get {}'.format(sleep_secs, b))
a, b = b, a + b
index += 1
async def stupid_fib(n):
index = 0
a = 0
b = 1
while index < n:
sleep_secs = random.uniform(0, 0.4)
await asyncio.sleep(sleep_secs)
print('Stupid one think {} secs to get {}'.format(sleep_secs, b))
a, b = b, a + b
index += 1
if __name__ == '__main__':
loop = asyncio.get_event_loop()
tasks = [
asyncio.ensure_future(smart_fib(10)),
asyncio.ensure_future(stupid_fib(10)),
]
loop.run_until_complete(asyncio.wait(tasks))
print('All fib finished.')
loop.close()Summarize
At this point, the introduction of coroutines in Python is complete. In the sample programs, sleep is used as the representative of asynchronous I/O. In actual projects, coroutines can be used to asynchronously read and write networks, read and write files, and render interfaces. While waiting for the coroutine to complete, the CPU can also Do other calculations. This is where coroutines come in.
The relevant code can be found on GitHub at https://github.com/yubo1911/saber/tree/master/coroutine .