peddled

gevent coroutine-based network library, based on rapid cycling event libev based lightweight greenlet execution unit, and reuse the concept of the Python standard library, supporting socket, ssl, tripartite library to provide synchronization by playing monkey patch form written in a way to support asynchronous code, dns resolved by the thread pool or c-ares, built-in tcp udp http server that supports multi-process, thread pool.

gevent and eventlet

2009年当时eventlet不支持livevent
当时eventlet的monkey-patch有bug导致socket操作会挂起
its Hub API is geared towards pure Python event loops. Making it work smoothly would require significant changes in the interface and implementation of every other hub. The socket module bugs were also specific to event loop, so even though I fixed them for pyevent, they were still present in every other hub.

gevent on the basis of libevent

Of course, the latest implementation is to use a libev

libevent event loop, using epoll, kqueue, dns is asynchronous, while it provides a http server. The eventlet is to maintain a pure python's event loop (recently supported epoll).

In addition to efficiency, and integrated signal processing, other use libevent library can also be integrated to come, dns resolve also the asynchronous, wsgi libevent server is also running on the server that comes with the (soon)

The same standard of style and python

For example, like the event so thread

difference

No eventlet.db_pool
no multi-process eventlet.processes
if it is used on the reactor twister

Event Loop

gevent tells the operating system when the data arrives to inform it, then you can proceed with the other gevent coroutine, for example, has received the data.

Unlike other network library, gevent implicitly start a coroutine in circulation, without calling dispatch or run or something, when to block it, it will get hub instance and switch to one, that is, the control to the hub If at this time will not automatically create a hub.

Switching rules

Note that only a greenlet give up control to perform other (call blocking function will switch back to the hub), which for io frequent system of course no problem, but for the frequent cpu system, or call some function to bypass the libev.

Locks, semaphores, what is actually useless, while the event, asyncresult, queue, etc. are still useful.

use

Create a Greenlet then call its start function, or directly call the spawn functions. Then, after giving up the current coroutine execution and control switches to. If an exception is triggered greenlet execution, not out of bounds greenlet, but the default is to print stacktrace stderror.

join the waiting greenlet exit, kill greenlet interrupt the execution, get get it produces abnormal return value or re raise.

It can be derived Greenlet, change its str way to change the traceback information displayed, but was first invoked in the initialization function Greenlet. The init (Self).

class MyNoopGreenlet(Greenlet): def __init__(self, seconds): Greenlet.__init__(self) self.seconds = seconds def _run(self): gevent.sleep(self.seconds) def __str__(self): return 'MyNoopGreenlet(%s)' % self.seconds

kill may be accompanied by a custom exception, it is possible that the coroutine will catch this exception can be attached timeout parameter, may be directly specified block = False performed asynchronously.

A little summary of the gevent

The basic concept 0x00

gevent is based libev a python of the asynchronous frame and greenlet.

libev is a high-performance event loop (event loop) implementation.
Event loop (also called IO multiplexing), is to solve the blocking problem, a method for concurrent implementation. Simply put, that event loopwill capture, process iochange events: experience blocking, to come out; the end of the obstruction continues. This depends on the underlying system selectfunctions and their upgraded version: polland epoll. "In-depth understanding of computer systems," a book, which has depth.

greenletIt is a pythoncoroutine management, switching module. By greenletYou can explicitly switch between different tasks.

0x01 Actor mode

ActorIt is Erlangthe essence of language. It emphasizes concurrency mechanism based on message passing.
We imitate this article establish a model actor. The core code is as follows:

import genvet
# 主队列
queue = gevent.queue.JoinableQueue()
while True: # 持续获取数据 try: data = queue.get(timeout=5) except Empty: gevent.sleep(0.5) continue # 交付给Actor if data: gl = Actor(data) g1.start() g1.join()

ActorIs our custom a Greenletsubclass, the core code is as follows:

class Actor(Greenlet):
    def __init__(self, data): self.data = data Greenlet.__init__(self) def _run(self): result = do_something(self.data) return result

In this way, our Actorcan accept data from the message queue, through start(), _run()is called.

Because of geventand monkey patchpresence, you can basically in a synchronized manner, write asynchronous code. Such wording made redundant. However, we can achieve a good separation of business and make the code clearer and easier to maintain and extend.

0x02 implement a callback

If you also want to Actoradd a callback. That is, etc. After he finished, then some processing.
Then we can Actormodify the models are as follows:

class Actor(Greenlet):
    def __init__(self, data): self.data = data Greenlet.__init__(self) def _run(self): # 通过self.link添加回调函数。 self.link(callback) result = do_something(self.data) return result

You can Greenlet().link()give your coroutine add a callback. The callback function receives only one parameter, i.e. in this example the coroutine ( g1). We will see the _run()function, in fact, returned a result resultof. Can we get that value it in the callback function. In fact, it is possible, this value is existed g1.valuein.

Again, this design may look a lot more than. But with the increase of framework features, such excess will design your code to become more flexible.

Python had to learn to master the library, gevent and asyncio use Comments

一, peddled

python program to achieve multi-tasking scheduler under a single thread, a thread in simple terms, the AB has two tasks to perform, but when A encounter time-consuming operation (network latency, file read and write, etc.), this A time gevent will continue, but will also begin the task B, if A and B simultaneously perform over time-consuming operations in the face of time-consuming operation, and continue gevent A.

Example of use:

Wherein gevent.sleep () is gevent carrying delay encountered when gevent This delay will automatically switch, here gevent.sleep () in place of some time-consuming operations, such as read and write the database, HTTP request, the file of write and other operations, gevent provides us with a very simple process of co-operation.

Two, asyncio

The use of asyncio, feel, and have the same purpose gevent

1, the basic concept:

event_loop event loop: understood as a pool circulation, which kept some of the async keyword defined coroutine function, you can perform only into the circulating pool

coroutine coroutine: coroutine object refers to a function defined using async keyword, it will not execute the function call immediately, but will return a coroutine object. Coroutine objects need to be registered to the event loop, called by the event loop.

task Tasks: a coroutine object is a native function may be suspended, the task is further encapsulated to coroutine, various states of the tasks contained therein.

Future : indicative of future results or perform tasks not performed. There is no essential difference between it and the task

async / await keywords: python3.5-defined keywords for coroutines, async definition of a coroutine, await for suspending blocking asynchronous call interface.

2. Create a task

Coroutine object can not be run directly into tasks required to run the package, the above is () method packaged in a task (implicit packaging) by run_until_complete, there are two ways explicitly Packing:

After creating the task, task before joining the event loop is pending state, run after joining loop is running the state, loop Calling End is Done, finished running a finished state , although essentially coroutine function and task refers to things are the same, but With task state coroutine function.

Wherein loop.run_until_complete () accepts a future parameter , futurn specifically refers to a coroutine function, and a task future subclass, so we do not declare a task directly into the coroutine function can be executed.

3, binding callback function

By task.add_done_callback task of binding (callback) method callback, the callback function receives a future target parameters such as Task, by internally future.result () function returns the obtained value coroutine.

4, await (hang time consuming operation)

Multitasking declared coroutine function, also registered in the loop, and his execution is the order of execution, because there is no statement that operate in asynchronous function is time-consuming operation, it will execute the order. await action is to tell the controller This step is time-consuming, the async coroutine objects can be defined, we can use await pending for the time-consuming operation

The above is not executed asynchronously executed, but the order of execution, but changed to the following form and that is asynchronous execution:

Visible interval of three tasks is almost negligible, can be used to note here await the success of pending correspondence should be the following three:

Native asynchronous function (the coroutine)

A types.coroutine () a modified generator, the generator may return coroutine object.

__Await object contains the method returns an iterator

So even if saync modification requests do not support asynchronous method, but requires special asynchronous network requests library aiohttp.

5, aiohttp

aiohttp need to be installed separately and used with asyncio library, look at the case

Difference in time a few basic task is negligible, that pro-test request will send one thousand 11 seconds to complete, really to the force.

6, with the use of multi-process

asyncio, aiohttp need to meet aiomultiprocess library use, requires at least version 3.6, using the example on github affix the library, currently in validation:

7, multiple concurrent coroutine

Use loop.run_until_complete (syncio.wait (tasks)) can also be used loop.run_until_complete (asyncio.gather (* tasks)), the former incoming task list, task will unpack operation.

7, nested coroutine

It suggests that it is a call to another coroutine coroutine, but the results related to the functions of the two coroutine processed and returned.

It is invoked coroutine returns the result of the following three ways;

8, stop the task coroutine

There are two ways to achieve the end of the task: off single task, close the Loop , the main function involving:

asyncio.Task.all_tasks () Gets the event loop task list

KeyboardInterrupt stop abnormal capture (Ctrl + C)

loop.stop () to stop the task cycle

task.cancel () to cancel a single task

loop.run_forever()

loop.close () event loop is closed, or will restart

gevent learning