Python3 iterators and generators

Iterator

Iteration is one of the most powerful features of Python is a way to access the collection elements.

Iterator can remember the object is a traverse position.

Iterator object began a visit from the first element of the collection until all the elements are accessed session is over. Iterator can only move forward not backward.

Iterator has two basic methods: ITER () and Next () .

Strings , lists, tuples, sets, dictionaries, range (), file handles , etc. iterables (iterable) can be used to create an iterator :

Internal comprising the __iter __ () method is iterables, follow a protocol iteration.
Iterables. The __iter __ () or ITER ( iterables ) into iterator

>>> list = [1,2,3,4]
>>> = iter (list) # create iterator object
The next element >>> next (it) # output iterator
1
>>> next(it)
2
>>>

Iterator object can be used for conventional traverse statement:

>>> list = ['a', 'b', 'c', 'd']
>>> = iter (list) # create iterator object
>>> for x in it:
	print(x, end=" ")

	
a b c d 
>>>

You can also use next () function:

>>> lst = [2,6,8,9]
>>> = iter (lst) # create iterator object
>>> 
>>> while True:
	try:
		print(next(it))
	except StopIteration:
		break

	
2
6
8
9
>>>

Creating an iterator

The class as a need to implement an iterator use two methods in a class the __iter __ () and the __next __ () .

If you already understand object-oriented programming, we know there is a class constructor, Python's constructor __init __ () , which will be executed when the object is initialized.

the __iter __ () method returns a special iterator object, the iterator object implements the __next __ () method and by StopIteration abnormality flag iteration is completed.

the __next __ () method (Python 2 is in the Next ()) returns the next iteration object.

Create a Returns iterator (counter), the initial value of 1, gradually increasing 1:

class Counter:
  def __iter__(self):
    self.a = 1
    return self
 
  def __next__(self):
    x = self.a
    self.a += 1
    return x
 
myclass = Counter()
myiter = iter(myclass)
 
print(next(myiter))
print(next(myiter))
print(next(myiter))
print(next(myiter))
print(next(myiter))

# Execution output is:
1
2
3
4
5

StopIteration

　　StopIteration exception is used to complete the identification iteration, prevent an infinite loop in __next __ () method, we can set the completion of a specified number of cycles after the trigger StopIteration exceptions to the end of iteration.

>>> str1 = "Python"
>>> strObj = str1.__iter__()
>>> strObj.__next__()
'P'
>>> strObj.__next__()
'Y'
>>> strObj.__next__()
't'
>>> strObj.__next__()
'h'
>>> strObj.__next__()
'O'
>>> strObj.__next__()
'n'
>>> strObj.__next__()
Traceback (most recent call last):
  File "<pyshell#33>", line 1, in <module>
    strObj.__next__()
StopIteration
>>>

So how to determine whether an object is iterable?

Whether it contains internal __iter__ method:
Means collections of Iterable, Iterator determines the type

>>> tup = (1,2,3)
>>> type(tup)
<class 'tuple'>
>>> dir (tup) when # arguments, and returns the attribute parameter, the method list.
['__add__', '__class__', '__contains__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', 
'__getnewargs__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__iter__', '__le__', '__len__', '__lt__', '__mul__', '__ne__', '__new__',
 '__reduce__', '__reduce_ex__', '__repr__', '__rmul__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'count', 'index']
>>> print('__iter__' in dir(tup))
True
>>>

>>> dic = {1:'dict', 2:'str', 3:'list', 4:'tuple', 5:'set', 6:'range()',7:'flie handler'}
>>> isinstance(dic, Iterable)
True
>>> isinstance(dic, Iterator)
False
>>> 
>>> ran = range(6)
>>> type(ran)
<class 'range'>
>>> isinstance(ran, Iterable)
True
>>> isinstance(ran, Iterator)
False
>>>

Builder

　　In Python, using the yield function is called a generator ( Generator ).

　　The difference is that with an ordinary function, the generator is a return iterator function can only be used for iterative operations, more simply understood generator is a iterator .

　　During the call to the generator running, each encounter yield when the function will pause and save all of the current operating information, return yield value, and the next execution of the next () continue to operate from its current position method.

　　Calling a generator function, it returns an iterator object .

　　yield Vs return：

 After the return return function status is terminated, and the yield will save the current execution state function, upon return, the function has returned to a previously saved state continues.

termination function return, yield not terminate generator function.
Returns a value, return to the executive function return value, the yield is given next () Return value

　　The following example uses yield achieved Fibonacci columns:

>>> def fib (max): # generator function - Fibonacci
	a, b, n = 0, 1, 0
	while n < max:
		Use yield yield b #
		a, b = b, a + b
		n = n + 1

>>> f = fib (6) # call the fab (5) does not perform fab function, but returns an iterable objects!
>>> f # Python interpreter will treat it as a generator
<generator object fib at 0x000001C6CB627780>
>>> 
>>> for n in fib(5):
	print(n)

	
1
1
2
3
5
>>> 
>>> f = fib(5)
>>> next (f) # next function using the values from the generator, can contribute to the next use of the generator
1
>>> next(f)
1
>>> next(f)
2
>>> next(f)
3
>>> next(f)
5
Continue next (g) when >>> next (f) # When the function has no more yield, met StopIteration
Traceback (most recent call last):
  File "<pyshell#37>", line 1, in <module>
    next(f)
StopIteration
>>> 
>>> fwrong = fib(6)
>>> fwrong.next () # Python2 syntax, Python3 will complain
Traceback (most recent call last):
  File "<pyshell#40>", line 1, in <module>
    fwrong.next () # Python2 syntax, Python3 will complain
AttributeError: 'generator' object has no attribute 'next'
>>>

　　send transmission data to the Builder. send action corresponds Next , but may also pass data to the generator while the generator continues to drive.

>>> import numbers
>>> def gen_sum():
	total = 0
	while True:
		num = yield
		if isinstance(num, numbers.Integral):
			Total = a +
			print('total: ', total)
		elif num is None:
			break
	return total

>>> g = gen_sum()
>>> g
<generator object gen_sum at 0x0000026A6703D3B8>
>>> g.send (None) # corresponds to the next (g), pre-activated generators
>>> g.send(2)
total:  2
>>> g.send(6)
total:  8
>>> g.send(12)
total:  20
>>> g.send (None) # stop generator
Traceback (most recent call last):
  File "<pyshell#40>", line 1, in <module>
    g.send(None)
StopIteration: 20
>>> 
>>> try:
	g.send (None) # stop generator
except StopIteration as e:
	print(e.value)

	
None
>>>

yield from Keyword

　　yield from an iterable into an iterator returns, it can be said, yield from keywords can return directly to a generator

>>> def func():
	lst = ['str', 'tuple', 'list', 'dict', 'set']
	yield lst

	
>>> gen = func()
>>> next(gen)
['str', 'tuple', 'list', 'dict', 'set']
>>> for i in gen:
	print(i)

	
>>> # yield from an iterable into an iterator returns
>>> def func2():
	lst = ['str', 'tuple', 'list', 'dict', 'set']
	yield from lst

	
>>> gen2 = func2()
>>> next(gen2)
'str'
>>> next(gen2)
'tuple'
>>> for i in gen2:
	print(i)

	
list
dict
set
>>>

>>> lst = ['H','e','l']
>>> dic = {'l':'vvvvv','o':'eeeee'}
>>> str1 = 'Python'
>>> 
>>> def yield_gen():
    for i in lst:
        yield i
    for j in dic:
        yield j
    for k in str1:
        yield k

        
>>> for item in yield_gen():
	print(item, end='')

	
HelloPython
>>> 
>>> l = ['H','e','l']
>>> d = {'l':'xxxxx','o':'ooooo'}
>>> s = 'Java'
>>> 
>>> def yield_from_gen():
	yield from l
	yield from d
	yield from s

	
>>> for item in yield_from_gen():
	print(item, end='')

	
Hello Java
>>>

Why Builder

Easier to use, the smaller the amount of code to use memory more efficiently. such as:

The list is to allocate all of the memory space in the establishment of,
While the generator is only used only when needed, like a record represents an infinite stream. A bit like a database operation a single record using the cursor .

If we want to read and use content far more than memory, but necessary to treat the contents of all of the stream, then the generator is a good choice,
For example, you can let the generator returns the current processing status, because it can save the state, then the next time you can deal directly.

Coroutine

　　According to Wikipedia, the definition given in " coroutine is a computer program component to produce a non-preemptive multitasking subroutine, coroutine allow different entry points at different locations pause or starts executing the program." From a technical point of view, "coroutine is that you can suspend function." If you understand it as "like a generator, like," So you wanted to right.

Yield achieved using coroutine

Based on yield asynchronous #
def consumer():
    '' 'Task 1: receiving data, processing the data' ''
    while True:
        x=yield

def producer():
    '' 'Task 2: Production Data' ''
    g=consumer()
    next(g)
    for i in range(10000000):
        g.send(i)

producer()

Implemented using coroutine yield from

import datetime
import heapq # stack module
import time


class Task:
    def __init__(self, wait_until, coro):
        self.coro = coro
        self.waiting_until = wait_until

    def __eq__(self, other):
        return self.waiting_until == other.waiting_until

    def __lt__(self, other):
        return self.waiting_until < other.waiting_until


class SleepingLoop:

    def __init__(self, *coros):
        self._new = coros
        self._waiting = []

    def run_until_complete(self):
        for coro in self._new:
            wait_for = coro.send(None)
            heapq.heappush(self._waiting, Task(wait_for, coro))
        while self._waiting:
            now = datetime.datetime.now()
            task = heapq.heappop(self._waiting)
            if now < task.waiting_until:
                delta = task.waiting_until - now
                time.sleep(delta.total_seconds())
                now = datetime.datetime.now()
            try:
                print('*'*50)
                wait_until = task.coro.send(now)
                print('-'*50)
                heapq.heappush(self._waiting, Task(wait_until, task.coro))
            except StopIteration:
                pass


def sleep(seconds):
    now = datetime.datetime.now()
    wait_until = now + datetime.timedelta(seconds=seconds)
    print('before yield wait_until')
    actual = yield wait_until # return a data type of time datetime
    print('after yield wait_until')
    return actual - now


def countdown(label, length, *, delay=0):
    print(label, 'waiting', delay, 'seconds before starting countdown')
    delta = yield from sleep(delay)
    print(label, 'starting after waiting', delta)
    while length:
        print(label, 'T-minus', length)
        waited = yield from sleep(1)
        length -= 1
    print(label, 'lift-off!')


def main():
    loop = SleepingLoop(countdown('A', 5), countdown('B', 3, delay=2),
                        countdown('C', 4, delay=1))
    start = datetime.datetime.now()
    loop.run_until_complete()
    print('Total elapsed time is', datetime.datetime.now() - start)


if __name__ == '__main__':
    main()

　　Results of the:

A waiting 0 seconds before starting countdown
before yield wait_until
B waiting 2 seconds before starting countdown
before yield wait_until
C waiting 1 seconds before starting countdown
before yield wait_until
**************************************************
after yield wait_until
A starting after waiting 0:00:00
A T-minus 5
before yield wait_until
--------------------------------------------------
**************************************************
after yield wait_until
C starting after waiting 0:00:01.001511
100 t-minus 4
before yield wait_until
--------------------------------------------------
**************************************************
after yield wait_until
A T-minus 4
before yield wait_until
--------------------------------------------------
**************************************************
after yield wait_until
B starting after waiting 0:00:02.000894
B T-minus 3
before yield wait_until
--------------------------------------------------
**************************************************
after yield wait_until
100 t-minus 3
before yield wait_until
--------------------------------------------------
**************************************************
after yield wait_until
A T-minus 3
before yield wait_until
--------------------------------------------------
**************************************************
after yield wait_until
B T-minus 2
before yield wait_until
--------------------------------------------------
**************************************************
after yield wait_until
100 t-minus 2
before yield wait_until
--------------------------------------------------
**************************************************
after yield wait_until
A T-minus 2
before yield wait_until
--------------------------------------------------
**************************************************
after yield wait_until
B T-minus 1
before yield wait_until
--------------------------------------------------
**************************************************
after yield wait_until
100 t-minus 1
before yield wait_until
--------------------------------------------------
**************************************************
after yield wait_until
A T-minus 1
before yield wait_until
--------------------------------------------------
**************************************************
after yield wait_until
B lift-off!
**************************************************
after yield wait_until
C lift-off!
**************************************************
after yield wait_until
A lift-off!
Total elapsed time is 0:00:05.005168

asyncio module

　　asyncioIs Python 3.4 standard library version introduced directly built-in support for asynchronous IO's.

　　With asynciosupplied @asyncio.coroutinecan be a generator labeled coroutine type, then coroutine inside with yield froma call to another coroutine asynchronous operation.

　　asyncioThe programming model is a message loop . We from the asyncioacquisition module in a direct EventLoopreference to the need to perform and then thrown into the coroutine EventLoopis executed, on the realization of asynchronous IO.

coroutine+yield from

import asyncio

@asyncio.coroutine
def hello():
    print("Nice to learn asyncio.coroutine!")
    # Asynchronous call asyncio.sleep (1):
    r = yield from asyncio.sleep(1)
    print("Nice to learn asyncio.coroutine again !")


# Get EventLoop:
loop = asyncio.get_event_loop()
# Coroutine execution
loop.run_until_complete(hello())
loop.close()

Nice to learn asyncio.coroutine !
Nice to learn asyncio.coroutine again !

　In order to simplify and better identify asynchronous IO, from Python 3.5 begun to introduce new syntax asyncand awaitallows coroutine code more concise and readable.

　Please note, asyncand awaitagainst coroutine new syntax to use the new syntax, only need a simple two-step replacement:

To @asyncio.coroutinebe replaced async;
To yield frombe replaced await.

async+await

　　In coroutine function, it may be suspended by its own grammar await coroutine, and waits until the completion of another coroutine returns the result:

import asyncio


async def hello():
    print("Nice to learn asyncio.coroutine!")
    # Asynchronous call asyncio.sleep (1):
    await asyncio.sleep(1)
    print("Nice to learn asyncio.coroutine again !")


# Get EventLoop:
loop = asyncio.get_event_loop()
# Coroutine execution
loop.run_until_complete(hello())
loop.close()

Perform multiple tasks

import threading
import asyncio


async def hello():
    print('Hello Python! (%s)' % threading.currentThread())
    await asyncio.sleep(1)
    print('Hello Python again! (%s)' % threading.currentThread())

loop = asyncio.get_event_loop()
tasks = [hello(), hello()]
loop.run_until_complete(asyncio.wait(tasks))
loop.close()

　result:

Hello Python! (<_MainThread(MainThread, started 4536)>)
Hello Python! (<_MainThread(MainThread, started 4536)>)
Hello Python again! (<_MainThread(MainThread, started 4536)>)
Hello Python again! (<_MainThread(MainThread, started 4536)>)

Get Return value

import threading
import asyncio


async def hello():
    print('Hello Python! (%s)' % threading.currentThread())
    await asyncio.sleep(1)
    print('Hello Python again! (%s)' % threading.currentThread())
    return "It's done"

loop = asyncio.get_event_loop()
task = loop.create_task(hello())
loop.run_until_complete(task)
K = task.result ()
print (right)