python iterators, generators

In python, we often use for ... in ...it to traverse list, set, dict, str, etc.
for ... in ...Essentially it does two things:

Call iter () to get the iterator;
Call next() to know StopIteration exception; ( next () in python3)

iterator

Let's first understand a few concepts:

Iterable: Iterable object
Iterator: iterator

Let's first look at the implementation of Iterable

from collections import Iterable

help(Iterable)

class Iterable(__builtin__.object)
 |  Methods defined here:
 |
 |  __iter__(self)
 |
 |  ----------------------------------------------------------------------
 |  Class methods defined here:
 |
 |  __subclasshook__(cls, C) from abc.ABCMeta
 |
 |  ----------------------------------------------------------------------
 |  Data descriptors defined here:
 |
 |  __dict__
 |      dictionary for instance variables (if defined)
 |
 |  __weakref__
 |      list of weak references to the object (if defined)
 |
 |  ----------------------------------------------------------------------
 |  Data and other attributes defined here:
 |
 |  __abstractmethods__ = frozenset(['__iter__'])
 |
 |  __metaclass__ = <class 'abc.ABCMeta'>
 |      Metaclass for defining Abstract Base Classes (ABCs).
 |
 |      Use this metaclass to create an ABC.  An ABC can be subclassed
 |      directly, and then acts as a mix-in class.  You can also register
 |      unrelated concrete classes (even built-in classes) and unrelated
 |      ABCs as 'virtual subclasses' -- these and their descendants will

Look at the implementation of Iterator again

from collections import Iterator

help(Iterator)

class Iterator(Iterable)
 |  Method resolution order:
 |      Iterator
 |      Iterable
 |      __builtin__.object
 |
 |  Methods defined here:
 |
 |  __iter__(self)
 |
 |  next(self)
 |      Return the next item from the iterator. When exhausted, raise StopIteration
 |
 |  ----------------------------------------------------------------------
 |  Class methods defined here:
 |
 |  __subclasshook__(cls, C) from abc.ABCMeta
 |
 |  ----------------------------------------------------------------------
 |  Data and other attributes defined here:
 |
 |  __abstractmethods__ = frozenset(['next'])
 |
 |  ----------------------------------------------------------------------
 |  Data descriptors inherited from Iterable:
 |
 |  __dict__
 |      dictionary for instance variables (if defined)
 |
 |  __weakref__
 |      list of weak references to the object (if defined)
 |
 |  ----------------------------------------------------------------------
 |  Data and other attributes inherited from Iterable:
 |
 |  __metaclass__ = <class 'abc.ABCMeta'>
 |      Metaclass for defining Abstract Base Classes (ABCs).
 |
 |      Use this metaclass to create an ABC.  An ABC can be subclassed
 |      directly, and then acts as a mix-in class.  You can also register
 |      unrelated concrete classes (even built-in classes) and unrelated
 |      ABCs as 'virtual subclasses' -- these and their descendants will
 |      be considered subclasses of the registering ABC by the built-in
 |      issubclass() function, but the registering ABC won't show up in
 |      their MRO (Method Resolution Order) nor will method
 |      implementations defined by the registering ABC be callable (not
 |      even via super()).

From the perspective of inheritance relationship, all Iterators are Iterables (iterable objects).
From the perspective of implementation, Iterator adds the next() method.

Determine whether it is Iterator or Iterable

Anything that can be looped is Iterable;
Anything that can be next() is Iterator;
list, tuple, dict, str, set are not Iterators, but an Iterator object can be returned by iter ()

from collections import Iterator, Iterable

isinstance([1,], Iterator)    // False
isinstance((1,), Iterator)    // False
isinstance({}, Iterator)      // False
isinstance("abc", Iterator)   // False
isinstance(set([]), Iterator) // False

isinstance([1,], Iterable)    // True
isinstance((1,), Iterable)    // True
isinstance({}, Iterable)      // True
isinstance("abc", Iterable)   // True
isinstance(set([]), Iterable) // True

dir([])                       // 没有 next() 方法
dir([].__iter__())            // 有 next() 方法

Builder

After the iterator is over, let's talk about the generator. Here is the introduction in Liao Xuefeng's blog:

With list comprehensions, we can directly create a list. However, due to memory constraints, the list capacity is definitely limited.
Moreover, creating a list with 1 million elements not only takes up a lot of storage space, but
if we only need to access the first few elements, then most of the space occupied by the latter elements will be wasted.

So, if the list elements can be calculated according to a certain algorithm, can we continue to calculate the subsequent elements in the process of looping?
This saves a lot of space by not having to create a complete list. In Python, this mechanism of computing while looping is
called a generator.

The creation of generators is simple and can be created by comprehending a list:

 g = (x * x for x in range(10))  // 使用 [] 返回的是 list, () 返回的是 generator

Another way is through yieldkeyword generation.

First look at the implementation of the generator:

<genexpr> = class generator(object)
 |  Methods defined here:
 |
 |  __getattribute__(...)
 |      x.__getattribute__('name') <==> x.name
 |
 |  __iter__(...)
 |      x.__iter__() <==> iter(x)
 |
 |  __repr__(...)
 |      x.__repr__() <==> repr(x)
 |
 |  close(...)
 |      close() -> raise GeneratorExit inside generator.
 |
 |  next(...)
 |      x.next() -> the next value, or raise StopIteration
 |
 |  send(...)
 |      send(arg) -> send 'arg' into generator,
 |      return next yielded value or raise StopIteration.
 |
 |  throw(...)
 |      throw(typ[,val[,tb]]) -> raise exception in generator,
 |      return next yielded value or raise StopIteration.
 |
 |  ----------------------------------------------------------------------
 |  Data descriptors defined here:
 |
 |  gi_code
 |
 |  gi_frame
 |
 |  gi_running

It can be found that there are more generators than iterators send, and so throwon.

send

Here we focus on the next sendmethod . We know that when using an iterator, it will exit when it encounters a yieldkeyword
, and will continue to execute in the next iteration. Let's look at an example:

def MyGenerator():
    value = yield 1
    value = yield value

gen = MyGenerator()
print gen.next()         // print 1
print gen.next()         // print None

Let's take a look at the specific implementation process:

Call the next() method, go to yield 1exit , note that the assignment operation of value has not been reached at this time (ie: value = yield 1 only executes the right part)
Call the next() method to continue the last code execution (ie: value = yield 1 only executes the assignment part on the right)
Since yield does not return a value, value = None
Return None, and print

Modify the above example:

def MyGenerator():
    value = yield 1
    value = yield value

gen = MyGenerator()
print gen.next()         // print 1
print gen.send(2)        // print 2

The send method specifies the return value of the last suspended yield statement, which is a bit abstract. Let's look at the execution process:

Call the next() method, go to yield 1exit , note that the assignment operation of value has not been reached at this time (ie: value = yield 1 only executes the right part)
Call the send(2) method to continue the last code execution (ie: value = yield 1 only executes the assignment part on the right)
value uses the value passed by send, that is: value = 2
return 2, and print

coroutine

Coroutines yieldare send()implemented using the and generator methods.