Python iteration can get to know a paper, the concept of iterators and generator

About
program ape a thoughtful, lifelong learning practitioners, is currently in a start-up team any team lead, technology stack involves Android, Python, Java, and Go, this is the main technology stack our team.
GitHub: https://github.com/hylinux1024
micro-channel public number: Lifetime developer (angrycode)

In Pythonthe iterable ( Iterable), the iterator ( Iterator) and generator ( Generator) these concepts are often used, these few beginner is often confused with the concept, it is time to put a few concepts to figure out.

0x00 iterable (Iterable)

Simply put, an object (which in Python everything is an object) as long as the realization of the long realized the __iter__()method, then a isinstance()check is a function of Iterablethe object;

E.g

class IterObj:
    
    def __iter__(self):
        # 这里简单地返回自身
        # 但实际情况可能不会这么写
        # 而是通过内置的可迭代对象来实现
        # 下文的列子中将会展示
        return self

The above definition of a class IterObjand implements the __iter__()method, this is a can iteration (the Iterable) Object

    it = IterObj()
    print(isinstance(it, Iterable))  # true
    print(isinstance(it, Iterator))  # false
    print(isinstance(it, Generator)) # false

Remember this class, we will see below this class definition.

Common iterables

In Pythonwhich common iterable have it?

Set or sequence type (e.g. list, tuple, set, dict, str)
File object
Defined in the class __iter__()method objects may be considered as Iterablean object, but the iterative custom objects to be able to foruse the correct cycle, it is necessary to ensure the __iter__()implementation must be correct (i.e., by the built-in iter()turn a function of Iteratorthe object. About IteratorHereinafter Description will be here to leave a pit, just remember that iter()the function is the ability to turn into a iterable iterator object, then in foruse)
Achieved if only implemented in the class __getitem__()object can be iter()converted to the iterator functions but not in themselves an object of iterations. So when an object is able to forrun in cycles, but not necessarily Iterableobject.

About 1,2 point we can verify the following

    print(isinstance([], Iterable))  # true list 是可迭代的
    print(isinstance({}, Iterable))  # true 字典是可迭代的
    print(isinstance((), Iterable))  # true 元组是可迭代的
    print(isinstance(set(), Iterable))  # true set是可迭代的
    print(isinstance('', Iterable))  # true 字符串是可迭代的
    
    currPath = os.path.dirname(os.path.abspath(__file__))
    with open(currPath+'/model.py') as file:
        print(isinstance(file, Iterable)) # true

We look at point 3 ,

    print(hasattr([], "__iter__")) # true
    print(hasattr({}, "__iter__")) # true
    print(hasattr((), "__iter__")) # true
    print(hasattr('', "__iter__")) # true

These built-in set or sequence object has __iter__properties that they have achieved the same name as the method. But this iterables want to forbe in use cycle, then it should be a built-in iter()function calls and converted into Iteratorobjects.
For example, we look at the built-in iterables

    print(iter([])) # <list_iterator object at 0x110243f28>
    print(iter({})) # <dict_keyiterator object at 0x110234408>
    print(iter(())) # <tuple_iterator object at 0x110243f28>
    print(iter('')) # <str_iterator object at 0x110243f28>

They are turned into respective corresponding iterator ( Iterator) object.
Now go back and look at the beginning of the definition of IterObjthe class

class IterObj:
    
    def __iter__(self):
        return self 
        
it = IterObj()
print(iter(it))

We use the iter()function, this time Jiangzai console print out the following information:

Traceback (most recent call last):
  File "/Users/mac/PycharmProjects/iterable_iterator_generator.py", line 71, in <module>
    print(iter(it))
TypeError: iter() returned non-iterator of type 'IterObj'

The type of error occurred, meaning that iter()the function can not be transferred 'non iterator' into iterator type.

Then how can one be iterative ( Iterable) object into a iterator ( Iterator) object?
We'll revise the IterObjdefinition of the class

class IterObj:

    def __init__(self):
        self.a = [3, 5, 7, 11, 13, 17, 19]

    def __iter__(self):
        return iter(self.a)

We define a constructor named in athe list, and also implements __iter__()the method.

Modified class can be iter()invoked function, i.e. may be in foruse cycle

    it = IterObj()
    print(isinstance(it, Iterable)) # true
    print(isinstance(it, Iterator)) # false
    print(isinstance(it, Generator)) # false
    print(iter(it)) # <list_iterator object at 0x102007278>
    for i in it:
        print(i) # 将打印3、5、7、11、13、17、19元素

Thus when defining an iterator object, we pay close attention to __iter__()the internal process logic implemented, in general, are known by a number of iterations may be objects (e.g., the above-mentioned set, sequence, or other files defined correctly iterable) to assist us to achieve

About point 4 means that the described iter()functions can be achieved a __getitem__()target converted into the method iterator object may be in foruse cycle, but if the isinstance()time detecting method, it is not an iterator object.

class IterObj:
    
    def __init__(self):
        self.a = [3, 5, 7, 11, 13, 17, 19]
    
    def __getitem__(self, i):
        return self.a[i]
        
it = IterObj()
print(isinstance(it, Iterable)) # false
print(isinstance(it, Iterator)) # false
print(isinstance(it, Generator)) false
print(hasattr(it, "__iter__")) # false
print(iter(it)) # <iterator object at 0x10b231278>

for i in it:
    print(i) # 将打印出3、5、7、11、13、17、19

This example illustrates the can in forthe object in use, it is not necessarily iterables.

Now we make a summary:

An iteration object is to achieve a __iter__()method of the object
It is to be in foruse cycle, it must meet the iter()call (ie, calling this function will not be wrong, can turn into a proper Iteratorobjects)
It can be assisted to achieve iterables our custom known by iterable.
Object implements a __getitem__()method can be iter()converted into a function Iteratorthat can be forused in the loop, but it is not an iterator object (available isinstance detection method ())

0x01 iterator (Iterator)

Many places have mentioned above Iterator, now we have to fill the pit.
When we can understand the concept of iterations for the iterator it is better understood.
An object implements __iter__()and __next__()methods, then it is an iterator object. E.g

class IterObj:

    def __init__(self):
        self.a = [3, 5, 7, 11, 13, 17, 19]

        self.n = len(self.a)
        self.i = 0

    def __iter__(self):
        return iter(self.a)

    def __next__(self):
        while self.i < self.n:
            v = self.a[self.i]
            self.i += 1
            return v
        else:
            self.i = 0
            raise StopIteration()

In the IterObjdefinition of a list, constructor a, length of the list n, index i.

    it = IterObj()
    print(isinstance(it, Iterable)) # true
    print(isinstance(it, Iterator)) # true
    print(isinstance(it, Generator)) # false
    print(hasattr(it, "__iter__")) # true
    print(hasattr(it, "__next__")) # true

We can find the above-mentioned
collections and objects are iterative sequence but not the iterator

    print(isinstance([], Iterator)) # false
    print(isinstance({}, Iterator)) # false
    print(isinstance((), Iterator)) # false
    print(isinstance(set(), Iterator)) # false
    print(isinstance('', Iterator)) # false

The file object is an iterator

    currPath = os.path.dirname(os.path.abspath(__file__))
    with open(currPath+'/model.py') as file:
        print(isinstance(file, Iterator)) # true

An iterator ( Iterator) objects can not only foruse cycle, built-in functions can also next()be called function. E.g

it = IterObj()
next(it) # 3
next(it) # 5

0x02 generator (Generator)

Now let's look at what is the builder?
A generator is both iterative iterators also

Generator defines two ways:

List Builder
Using yieldthe definition of the function generator

Look at the Case 1

    g = (x * 2 for x in range(10)) # 0～18的偶数生成器 
    print(isinstance(g, Iterable)) # true
    print(isinstance(g, Iterator)) # true
    print(isinstance(g, Generator)) # true
    print(hasattr(g, "__iter__")) # true
    print(hasattr(g, "__next__")) # true
    print(next(g)) # 0
    print(next(g)) # 2

List Builder may not need to consume a lot of memory to generate a huge list, will only be calculated when needed the data.
Look Case 2

def gen():
    for i in range(10):
        yield i

This yieldeffect is equivalent to return, is the order of this function returns the [0,10)natural number can be produced by next()or forloop to iterate.
When the program encounters yieldkeyword, the generator function returns, until once again perform the next()function, it will continue from the last execution point function returns, that is, yieldsave the information on the location of the function execution, variables, and so on exit, when executed again, on this yieldcontinue down the exit place.
In Pythonutilizing these features of the generator may be implemented coroutine. Coroutine can be understood as a lightweight thread, thread it with respect to the treatment of high concurrency scenarios has many advantages.

Here with a look of realization coroutine producer - consumer model

def producer(c):
    n = 0
    while n < 5:
        n += 1
        print('producer {}'.format(n))
        r = c.send(n)
        print('consumer return {}'.format(r))


def consumer():
    r = ''
    while True:
        n = yield r
        if not n:
            return
        print('consumer {} '.format(n))
        r = 'ok'


if __name__ == '__main__':
    c = consumer()
    next(c)  # 启动consumer
    producer(c)

This code performs the following effects

producer 1
consumer 1 
producer return ok
producer 2
consumer 2 
producer return ok
producer 3
consumer 3 
producer return ok

Coroutine achieved CPUeffect switching between the two functions to achieve concurrency.

0x04 quote

https://docs.python.org/3.7/