[Notes] python advanced multitasking - coroutine (iterators, generators, yield, greenlet, gevent)

Can for ... in ... Such a data read iteration statement we use is called for the object iterable ( the Iterable ), such as: list, tuple, str other types of data.

Can use the isinstance () determines whether an object is an object Iterable ( available iterative determination ). Such as:

from collections import Iterable

isinstance([11, 22, 33], Iterable)  #  返回True

isinstance(100, Iterable)  # 返回False

The essence iterables that can provide us with a iterator (Iterator) help us traverse using its iteration.
A realization of the __iter__method and __next__the method object is an iterator . ( Note: The iterator will be able iteration, iteration can not necessarily iterators . )
The core functionality is the ability to return an iterator to the next data value by next () function is called. Iterative process, each iteration time (ie for ... in ... every cycle) will return the next data object has been read back data until all data iterated over. In this process iterator recorded for each visit to the first few data, each iteration may be returned to the next data.
Iterables by __iter__method provides an iterator to us at the time iteration an iteration object, in fact, is to first get the object provides an iterator, and then turn to get each data object through this iterator . . That is a have the __iter__object method is an iteration object.
If you want an object is called an object can be iterative, that can be used for , it must implement __iter__ method.

# Note: for loop execution

xxxx_obj in the TEMP for:
# for loop internal code

1 can be determined whether the iteration xxxx_obj
2. If the first step to be iterations iter function is called to obtain object xxxx_obj __iter__ method return value
3. The method of __iter__ return value is an iterator

for item in Iterable cyclic nature: that is, by first acquiring function iter iterables Iterable iterator, then the iterator continuously acquired call next () method to get the next value () and assigned to the Item, when after the end of the cycle encountered StopIteration exception.

Using the isinstance () determines whether an object is Iterator objects ( iterator determination ):

from collections import Iterator

isinstance([], Iterator)  # 返回False

isinstance(iter([11,22]), Iterator)  # 返回True

By () ITER function gets iterables the iterator . You can then be obtained using the continuous iterator next () function to get the next data. iter () function is actually called iterables of __iter__methods, such as:
```
li = [11, 22, 33, 44, 55]
>>> li_iter = iter(li)
>>> next(li_iter)
11
>>> next(li_iter)
22
>>> next(li_iter)
```
After the last iteration of the data has been completed, again calling next () function will throw StopIteration exception .

1.1 Object implements iterative Example 1 (two objects)

description:

To achieve the object for loop, __iter__ class to implement a method, the method returns a reference to the object (the object must __iter __, __ next__ method), values for loop is __next__ return value.

import time
from collections import Iterable
from collections import Iterator


class Classmate(object):
    def __init__(self):
        self.names = list()

    def add(self, name):
        self.names.append(name)

    def __iter__(self):
        """如果想要一个对象称为一个　可以迭代的对象，即可以使用for，那么必须实现__iter__方法"""
        return ClassIterator(self)  # 返回对象的引用（对象要有__iter__、__next__方法）
        # self把自己的引用传给了ClassIterator对象(即ClassIterator类的对象的init的obj指向了Classmate的对象)


class ClassIterator(object):

    def __init__(self, obj):
        self.obj = obj
        self.current_num = 0  # 记录取第几个

    def __iter__(self):
        pass

    def __next__(self):
        if self.current_num < len(self.obj.names):
            ret = self.obj.names[self.current_num]
            self.current_num += 1
            return ret
        else:
            raise StopIteration  # 越界


classmate = Classmate()
classmate.add("张三")
classmate.add("李四")
classmate.add("王五")

# print("判断classmate是否是可以迭代的对象:", isinstance(classmate, Iterable))
# classmate_iterator = iter(classmate)
# print("判断classmate_iterator是否是迭代器:", isinstance(classmate_iterator, Iterator))
# print(next(classmate_iterator))

for name in classmate:
    print(name)
    time.sleep(1)

1.2 Object implements iterative Example 2 (get an object)

import time
from collections import Iterable
from collections import Iterator


class Classmate(object):
    def __init__(self):
        self.names = list()
        self.current_num = 0

    def add(self, name):
        self.names.append(name)

    def __iter__(self):
        """如果想要一个对象称为一个　可以迭代的对象，即可以使用for，那么必须实现__iter__方法"""
        return self  # 返回自己
        # 调用iter(xxobj)的时候 只要__iter__方法返回一个迭代器即可(自己的或别的对象的)，
        # 但是要保证是一个迭代器(即实现了 __iter__  __next__方法)

    def __next__(self):
        if self.current_num < len(self.names):
            ret = self.names[self.current_num]
            self.current_num += 1
            return ret
        else:
            raise StopIteration


classmate = Classmate()
classmate.add("张三")
classmate.add("李四")
classmate.add("王五")

# print("判断classmate是否是可以迭代的对象:", isinstance(classmate, Iterable))
# classmate_iterator = iter(classmate)
# print("判断classmate_iterator是否是迭代器:", isinstance(classmate_iterator, Iterator))
# print(next(classmate_iterator))

for name in classmate:
    print(name)
    time.sleep(1)

1.3 Fibonacci Fibonacci number (iterators)

Fibonacci：

nums = list()

a = 0 
b = 1
i = 0  # 第i轮
while i < 10:
    nums.append(a)
    a, b = b, a+b
    i += 1


for num in nums:
    print(num)

The Fibonacci (with iterator): iterator implemented with, at each iteration generating a mathematical calculation by a number.

class Fibonacci(object):
    def __init__(self, all_num):
        self.all_num = all_num
        self.current_num = 0
        self.a = 0
        self.b = 1

    def __iter__(self):
        return self

    def __next__(self):
        if self.current_num < self.all_num:
            ret = self.a
        
            self.a, self.b = self.b, self.a+self.b
            self.current_num += 1

            return ret  # 返回该轮的 a 
        else:
            raise StopIteration


fibo = Fibonacci(10)


for num in fibo:
    print(num)

Difference: An iterator is returned with a way to generate the data , do not list the number of columns to hold the value, and when to call, when to generation, small footprint.

Note: list, tuple and other types of conversion will be used iterators.

2. Generator

Using the iterator, we can obtain data generated in each iteration according to a specific rule (by next () method). But when we realize an iterator, on the state of the current iteration to the needs of our own record, and then only in accordance with the current state of the data generated. In order to achieve the current state of the record, and with the next () function iteration, we can use a simpler syntax, namely generator (generator). Generator is a special type of iterator .

Creating Builder 2.1

method 1:

The first method is as simple as a list of the formula (Formula derivation list) of [] into ().

In [15]:L = [ x*2 for x in range(5)]

In [16]: L
Out[16]: [0, 2, 4, 6, 8]

In [17]: G = ( x*2 for x in range(5))

In [18]: G
Out[18]: <generator object <genexpr> at 0x7f626c132db0>

L is a list, and G is a generator. It can be directly printed out each element of the list L. For G, in accordance with the method of the iterator is used, i.e. () function, for circulation through the next, list () method and the like.

for x in G:
    print(x)

输出：
0
2
4
6
8

Method 2: it can be implemented as a function. Functions are guaranteed yield, function generator automatically becomes.

The Fibonacci generator implements :

def create_num(all_num):
    print("----1---")
    # a = 0
    # b = 1
    a, b = 0, 1
    current_num = 0
    while current_num < all_num:
        print("----2---")
        # print(a)
        yield a  # 如果一个函数中有yield语句，那么这个就不在是函数，而是一个生成器的模板
        print("----3---")
        a, b = b, a+b
        current_num += 1
        print("----4---")


# 如果在调用create_num的时候，发现这个函数中有yield那么此时，不是调用函数，而是创建一个生成器对象
obj = create_num(10)  # obj即为一个生成器对象

ret = next(obj)
print(ret)
ret = next(obj)
print(ret)


# #  生成器是特殊的迭代器，可迭代，会输出数列
# for num in obj:
#    print(num)

Output:

----1---
----2---
0
----3---
----4---
----2---
1

Note:

By the above output, it can be seen each call generator is not executed from the beginning.

For the first time next () is called, executed from the beginning, output ---- 1 --- , ---- 2 --- perform to yield at will here, " pause ", and returns a.

At this time, a = 0, outputs 0 ;

The second time next () is invoked, the yield continues, the output ---- --- 3, 4 --- ----, ---- 2 --- , loops back to yield at "Pause "........

Note: You can create multiple generator object, multiple generators next () call affect each other.

Fibonacci generator implements, the originally iterator __next__basic logic implemented method implemented into a function, it will return a value of each iteration return replaced the yield , this time newly defined function is no longer function , but a generator of. As long as the yield keyword in the def called generator .

2.2 by abnormality determination producer end, acquires the return value Example:

If you want to get the generator's return statement return value must be captured StopIteration error, the return value is included in the value of StopIteration.

def create_num(all_num):
    # a = 0
    # b = 1
    a, b = 0, 1
    current_num = 0
    while current_num < all_num:
        # print(a)
        yield a  # 如果一个函数中有yield语句，那么这个就不在是函数，而是一个生成器的模板
        a, b = b, a+b
        current_num += 1
    return "ok...."


obj2 = create_num(8)

while True:
    try:
        ret = next(obj2)
        print(ret)
    except Exception as ret:
        print(ret.value)  # 返回值包含在异常的的value中
        break

Output:

0
1
1
2
3
5
8
13
ok....

2.3 send wakeup generator

Except next () function generator continues to wake up, can also be used send () function is performed to wake up. Use send () function is a benefit at the same time the breakpoint wake passing one additional data : generating .send objects (by value) .

def create_num(all_num):
    a, b = 0, 1
    current_num = 0
    while current_num < all_num:
        ret = yield a
        print(">>>ret>>>>", ret)
        a, b = b, a+b
        current_num += 1


obj = create_num(10)

# obj.send(None)  # send一般不会放到第一次启动生成器，如果非要这样做 那么传递None

ret = next(obj)  # 第一次一般用next来启动
print(ret)

# send里面的数据会 传递给第5行，当做yield a的结果，然后ret保存这个结果,,, 
# send的返回结果是下一次调用yield时 yield后面的值 a 
ret = obj.send("hello")
print(ret)

Results of the:

0
>>>ret>>>> hello
1

Explanation:

The first time to start the next, the return value is yield value then a, the output 0;

By starting the second send, send the value passed to the RET, the output >>> ret >>>> hello;

send yield following values return value a, in this case 1, output 1.

2.4 Summary Builder:

Use the keyword function is no longer yield is a function, but a generator. (Yield using a function generator is)
yield keyword has two effects:
- Save the current operating status (breakpoint), and then suspended , the upcoming generator (function) Suspend
- The value of the yield keyword behind the expression as a return value to return, this time can be understood as acts of return
You can use next () function allows the generator to continue from the breakpoint, that wake-up generator (function)
Python3 The generator may use the return value of the final return return run, and Python2 the generator does not allow return to return a return value (i.e., can return to exit from the generator, but does not have any return after expression).

3. coroutine - yield (Learn

Coroutine, also called micro-threads, shred. English Coroutine. A python is in addition (understood as the resources needed) way to multi-task, and a smaller footprint than the smaller thread execution unit. It comes CPU context, just at the right time, to be switched to another coroutine a coroutine. As long as this process to save or restore the CPU context, you can still run the program.

Popular understanding: In a thread of a function, you can save information about the current number of temporary variables and other functions at any place, and then switch to another function execution (non-function call), and the number of switches and when then switch to determine the original function by developers.

yield multitasking instance:

import time


def task_1():
    while True:
        print("---1----")
        time.sleep(0.1)
        yield


def task_2():
    while True:
        print("---2----")
        time.sleep(0.1)
        yield


def main():
    t1 = task_1()
    t2 = task_2()
    # 先让t1运行一会，当t1中遇到yield的时候，再返回到24行，然后
    # 执行t2，当它遇到yield的时候，再次切换到t1中
    # 这样t1/t2/t1/t2的交替运行，最终实现了多任务....协程
    while True:
        next(t1)
        next(t2)
    

if __name__ == "__main__":
    main()

Output: Output of the following cycle

---1----
---2----
---1----
---2----
---1----
---2----

... omitted ...

Explanation: The while loop, next (t1) to suspend execution returns wake at yield, next (t2) to yield the wake pause execution returns .......

(Concurrent: fake multi-tasking)

4. coroutine - greenlet (Learn

In order to better use the coroutine to complete the multi-task, the Python greenlet its module package, so that the task switching becomes easier.

Use the following command module mounted greenlet:

sudo pip3 install greenlet

Use greenlet complete multiple tasks:

from greenlet import greenlet
import time


def test1():
    while True:
        print("---A--")
        gr2.switch()
        time.sleep(0.5)


def test2():
    while True:
        print("---B--")
        gr1.switch()
        time.sleep(0.5)


gr1 = greenlet(test1)  # gr1 是全局变量
gr2 = greenlet(test2)

# 切换到gr1中运行
gr1.switch()

Output:

A-- ---
--- B -
--- A--
--- B -
... omitted ...

Object .switch () object to the switching operation.

5. coroutine - gevent (Key

In practice, with less yield, greenlet, using more GEVENT .
greenlet coroutine has been achieved, but needs manual switching. There is also a more powerful than the python greenlet and can automatically switch the task of the modulegevent .
geventPrinciple: When a greenlet encounters IO (input output means is input and output, such as a network, file operation, etc.) operations, such as access to the network, it automatically switches to the other greenlet, IO until the operation is completed, then switch at the appropriate time come back to continue. Since the IO operation is very time-consuming, often make the program in a wait state, with gevent automatically switch coroutine for us to ensure that there is always greenlet running, instead of waiting for IO.

Installation gevent:

sudo pip3 install gevent

5.1 gevent multi-task:

import gevent
import time


def f1(n):
    for i in range(n):
        print(gevent.getcurrent(), i)
        # time.sleep(0.5)
        gevent.sleep(0.5)   # 模拟一个耗时操作，注意不是time模块中的sleep


def f2(n):
    for i in range(n):
        print(gevent.getcurrent(), i)
        # time.sleep(0.5)
        gevent.sleep(0.5)


def f3(n):
    for i in range(n):
        print(gevent.getcurrent(), i)
        # time.sleep(0.5)
        gevent.sleep(0.5)


print("----1---")
g1 = gevent.spawn(f1, 5)  # 创建gevent对象
print("----2---")
g2 = gevent.spawn(f2, 5)
print("----3---")
g3 = gevent.spawn(f3, 5)
print("----4---")
g1.join()
g2.join()
g3.join()

Output:

----1---
----2---
----3---
----4---
<Greenlet at 0x7f9269843748: f1(5)> 0
<Greenlet at 0x7f9269843948: f2(5)> 0
<Greenlet at 0x7f9269843a48: f3(5)> 0
<Greenlet at 0x7f9269843748: f1(5)> 1
<Greenlet at 0x7f9269843948: f2(5)> 1
<Greenlet at 0x7f9269843a48: f3(5)> 1
<Greenlet at 0x7f9269843748: f1(5)> 2
<Greenlet at 0x7f9269843948: f2(5)> 2
<Greenlet at 0x7f9269843a48: f3(5)> 2
<Greenlet at 0x7f9269843748: f1(5)> 3
<Greenlet at 0x7f9269843948: f2(5)> 3
<Greenlet at 0x7f9269843a48: f3(5)> 3
<Greenlet at 0x7f9269843748: f1(5)> 4
<Greenlet at 0x7f9269843948: f2(5)> 4
<Greenlet at 0x7f9269843a48: f3(5)> 4

Explanation:

When you create gevent object does not execute immediately, call the object .join () when the congestion wait subject executing the (delay operation) encountered a delay, gevent switch tasks started.
Inside the function, encountered gevent.sleep (0.5) delay between switching tasks, time.sleep (0.5) does not delay switch tasks.
Coroutine use the time while waiting for time-consuming to do other things.

Notice: encountered gevent.sleep (0.5) delay between switching tasks, time.sleep (0.5) does not delay switch tasks. Each time consuming operations must be changed if gevent, it will be very troublesome. So the need to gevent patch.

5.2 patch to gevent

When there is time-consuming operation, patching method:

from gevent import monkey

monkey.patch_all()  # 将程序中用到的耗时操作的代码，换为gevent中自己实现的模块

gevent multitasking patch examples:

import gevent
import time
from gevent import monkey

monkey.patch_all()


def f1(n):
    for i in range(n):
        print(gevent.getcurrent(), i)
        time.sleep(0.5)


def f2(n):
    for i in range(n):
        print(gevent.getcurrent(), i)
        time.sleep(0.5)


def f3(n):
    for i in range(n):
        print(gevent.getcurrent(), i)
        time.sleep(0.5)


# print("----1---")
# g1 = gevent.spawn(f1, 5)
# print("----2---")
# g2 = gevent.spawn(f2, 5)
# print("----3---")
# g3 = gevent.spawn(f3, 5)
# print("----4---")
# g1.join()
# g2.join()
# g3.join()

gevent.joinall([
    gevent.spawn(f1, 5),
    gevent.spawn(f2, 5),
    gevent.spawn(f2, 5)
])

Note: When true calling, you can not call an object a target .join () , directly gevent.joinall () passing a list of objects can be created.

6. processes, threads, coroutines small summary

Process is resource allocation unit (a process can have multiple threads);
Thread scheduling unit is the operating system (there are a plurality of thread can coroutine);
The process of switching resources are needed maximum efficiency is very low;
Thread switch resources needed in general, the general efficiency (of course, without considering the GIL);
Coroutine task switching resources is small, high efficiency (long network requests and other time-consuming tasks, may be considered without considering GIL with coroutine);
Multi-process, multi-thread based on the number of cpu core is not as likely to be parallel, but coroutines is in a thread so that concurrent.