Python and coroutine from Python2-Python3

Coroutine, also called micro-threads, shred, English Coroutine; A statement, then what is the thread: thread lightweight coroutine is a user-state.

Python support for coroutines in python2 also relatively simple, but there are third-party libraries that can be used to start full support in python3, but also become a core function python3, it is worth learning.

Society Production Process

Coroutine, also called micro-threads, shred, English Coroutine; A statement, then what is the thread: thread lightweight coroutine is a user-state.

Coroutine has its own stack and register context. When coroutine scheduled handover, the context save registers and stack to another location, when cut back, context restore a previously saved registers and stack. Thus: coroutine can be left as the last call (i.e., a particular combination of all of the local state), during each reentrant, the equivalent of entering a call state, another way: once on entering the leaving at the logic flow position.

Coroutine advantages:

1) you do not need the overhead of thread context switching

2) locking and synchronizing atomic operation without overhead

3) to facilitate flow of the switching control, simplified programming model

4) high concurrency + scalability + low cost: a CPU million for the support coroutine is not a problem. So it is suitable for high concurrent processing.

Coroutine disadvantages:

1) can not take advantage of multi-core resource: the nature of the coroutine is a single-threaded, it can not spend multiple nuclei, coroutines needs and processes with a single CPU to run on multiple CPU

2) (e.g., when the IO) blocked (Blocking) The operation would block out the entire program

Python2 the coroutine

the yield keyword
Python2 support for coroutines, is achieved by the yield keyword, the following sample code is a common producer - consumer model, the code examples are as follows:

def consumer():

    r = ''

    while True:

        n = yield r

        if not n:

            continue

        print('[CONSUMER] Consuming %s...' % n)

        r = '200 OK'

def produce(c):

    c.next()

    n = 0

    while n < 5:

        n = n + 1

        print('[PRODUCER] Producing %s...' % n)

        r = c.send(n)

        print('[PRODUCER] Consumer return: %s' % r)

    c.close()

if __name__ == '__main__':

    c = consumer()

    produce(c)

Results of the:

Noting consumer is a function Generator (generator), to produce a consumer after passing:

1) first calls c.next () Start generator;

2) Then, once the production of something, by switching c.send (n) to perform consumer;

3) consumer to get the message yield by the process, and the results returned by the yield;

4) produce consumer to get the result of the process, continue to produce the next message;

5) produce decided not to produce a close consumer by c.close (), the end of the whole process.

整个流程无锁，由一个线程执行，produce和consumer协作完成任务，所以称为“协程”，而非线程的抢占式多任务。

传统的生产者-消费者模型是一个线程写消息，一个线程取消息，通过锁机制控制队列和等待，但一不小心就可能死锁。

如果改用协程，生产者生产消息后，直接通过yield跳转到消费者开始执行，待消费者执行完毕后，切换回生产者继续生产，效率极高。

Python对协程的支持还非常有限，用在generator中的yield可以一定程度上实现协程。虽然支持不完全，但已经可以发挥相当大的威力了。

gevent模块
Python通过yield提供了对协程的基本支持，但是不完全。而第三方的gevent为Python提供了比较完善的协程支持。gevent是第三方库，通过greenlet实现协程，其基本思想是：

当一个greenlet遇到IO操作时，比如访问网络，就自动切换到其他的greenlet，等到IO操作完成，再在适当的时候切换回来继续执行。由于IO操作非常耗时，经常使程序处于等待状态，有了gevent为我们自动切换协程，就保证总有greenlet在运行，而不是等待IO。由于切换是在IO操作时自动完成，所以gevent需要修改Python自带的一些标准库，这一过程在启动时通过monkey patch完成。

示例代码如下：

from gevent import monkey; monkey.patch_all()

import gevent

import urllib2

def f(url):

    print('GET: %s' % url)

    resp = urllib2.urlopen(url)

    data = resp.read()

    print('%d bytes received from %s.' % (len(data), url))

gevent.joinall([

        gevent.spawn(f, 'https://www.python.org/'),

        gevent.spawn(f, 'https://www.yahoo.com/'),

        gevent.spawn(f, 'https://github.com/'),

])

执行结果：

从执行结果可以看到，网站访问的顺序是自动切换的。

gevent优缺
使用gevent，可以获得极高的并发性能，但gevent只能在Unix/Linux下运行，在Windows下不保证正常安装和运行。Python创始人Gvanrossum从来不喜欢Gevent，而是更愿意另辟蹊径的实现asyncio（python3中的异步实现）。

1）Monkey-patching。中文「猴子补丁」，常用于对测试环境做一些hack。Gvanrossum说用它就是”patch-and-pray”，由于Gevent直接修改标准库里面大部分的阻塞式系统调用，包括socket、ssl、threading和 select等模块，而变为协作式运行。但是无法保证在复杂的生产环境中有哪些地方使用这些标准库会由于打了补丁而出现奇怪的问题，那么只能祈祷（pray）了。

2）其次，在Python之禅中明确说过：「Explicit is better than implicit.」，猴子补丁明显的背离了这个原则。

3）第三方库支持。得确保项目中用到其他用到的网络库也必须使用纯Python或者明确说明支持Gevent，而且就算有这样的第三方库，也需要担心这个第三方库的代码质量和功能性。

4）Greenlet不支持Jython和IronPython，这样就无法把gevent设计成一个标准库了。

之前是没有选择，很多人选择了Gevent，而现在明确的有了更正统的、正确的选择：asyncio（下一节会介绍）。所以建议大家了解Gevent，拥抱asyncio。

另外，如果知道现在以及未来使用Gevent不会给项目造成困扰，那么用Gevent也是可以的。

Python3中的协程

Gvanrossum希望在Python 3 实现一个原生的基于生成器的协程库，其中直接内置了对异步IO的支持，这就是asyncio，它在Python 3.4被引入到标准库。

下面将简单介绍asyncio的使用：

1）event_loop 事件循环：程序开启一个无限的循环，程序员会把一些函数注册到事件循环上。当满足事件发生的时候，调用相应的协程函数。

2）coroutine 协程：协程对象，指一个使用async关键字定义的函数，它的调用不会立即执行函数，而是会返回一个协程对象。协程对象需要注册到事件循环，由事件循环调用。

3) task Tasks: a coroutine object is a native function may be suspended, the task is further encapsulated to coroutine, wherein the state includes various tasks.

4) future: a guide to future results or perform tasks not performed. There is no essential difference between it and the task

5) async / await Keywords: python3.5 keyword defines a coroutine, async define a coroutine, await for suspending blocking asynchronous call interface.

Code examples are as follows:

import asyncio

import time

now = lambda: time.time()

async def do_some_work(x):

    print('Waiting: {}s'.format(x))

    await asyncio.sleep(x)

    return 'Done after {}s'.format(x)

async def main():

    coroutine1 = do_some_work(1)

    coroutine2 = do_some_work(5)

    coroutine3 = do_some_work(3)

    tasks = [

        asyncio.ensure_future(coroutine1),

        asyncio.ensure_future(coroutine2),

        asyncio.ensure_future(coroutine3)

    ]

    done, pending = await asyncio.wait(tasks)

    for task in done:

        print('Task ret: ', task.result())

start = now()

loop = asyncio.get_event_loop()

task = asyncio.ensure_future(main())

try:

    loop.run_until_complete(task)

    print('TIME: ', now() - start)

except KeyboardInterrupt as e:

    print(asyncio.Task.all_tasks())

    print(asyncio.gather(*asyncio.Task.all_tasks()).cancel())

    loop.stop()

    loop.run_forever()

finally:

    loop.close()

Results of the:

We can see the program execution time is the longest waiting time shall prevail.

Use async coroutine objects can be defined using await can be suspended for a time-consuming operation, like the generator in the yield, the function allows a control. Coroutine encounters await, the event loop will suspend the co-drive the implementation of other coroutine until the other coroutine also suspend or finished, and then perform the next coroutine. Typically some time-consuming operations IO operations, such as network requests, file reading and the like. We use asyncio.sleep function to simulate the IO operation. The purpose is to let these coroutines of asynchronous IO operations.

Asyncio python3 is a powerful built-in library, the above is just a brief usage asyncio are interested, it is worth learning about!