Reactor model and Proactor

One, background

Introduced in front of I / O multiplexing model, that has I / O multiplexing, with epoll can have the server and sent hundreds of thousands of simultaneous connections, but also to maintain a relatively high TPS, is it enough? For example, when using epoll now they are generally from a task, whether to patrol the event, and then notify the processing, and the ideal way is the best mechanism for callback able to provide a programming framework, so that the program more these structures, on the other hand, if you want after each event notification, do have the opportunity to be delegated to a separate thread to run inside, and the thread can notify the completion of the state back to the main task, "asynchronous" into the system must be introduced. Therefore, this chapter introduces the next "programming framework."

Two, Reactor model

Reactor pattern is treated concurrent I / O more common mode, for synchronous I / O, the central idea is registered to process all I / O events on a central I / O multiplexer, while the main thread / process blocked on multiplexer; once the I / O event comes or ready (or socket file descriptor read, write), multiplexer and return to pre-registered corresponding I / O event processor corresponding to distribute.

Reactor is an event-driven mechanism, and differs from the ordinary function call that: the application is not active calls an API to complete the deal, but on the contrary, Reactor Retrograde event handling process, the application needs to provide the appropriate interfaces and registered with the Reactor, if the corresponding event occurs, Reactor will take the initiative to call interface applications that are registered, these interfaces also known as "callbacks." With "Hollywood principle" to describe Reactor very appropriate: Do not call us, we'll call you.

Reactor mode and Observer mode is very similar in some ways: When a subject is changed, all dependency bodies are notified. However, the observer pattern source associated with a single event, and the reaction mode is associated with a plurality of event sources.

In Reactor mode, there are five key players:

1. Descriptor (handle): resources provided by the operating system, for identifying each event, as Socket descriptors, file descriptors, etc. values ​​of the signal. In Linux, it is represented by an integer. Events can be external, such as a connection request from a client, data and the like. Events can also be from within, such as signal, timer events.
   
2. Synchronous event demultiplexer (event demultiplexer): incoming event is random, asynchronous, can not predict when the program receives a client connection request or receive a signal. So wait and recycling program to handle the event, this is the event loop. In the event loop, waiting for events in general use I / O multiplexing implementation. On linux systems usually select, poll, epoll system calls, to wait for the occurrence of one or more events. I / O framework library will generally be various I / O system calls multiplexed encapsulated into a unified interface, called an event demultiplexer. The caller will be blocked until the separator has a separate set of descriptors event. This can refer to the previous I / O multiplexer model .
   
3. Event handler (event handler): Event Processor I / O is typically framework library interface consists of one or more template function thereof. These templates describe the function and operation of an event related to the application, users need to extend it to implement its own event handler that specific event handler. Therefore, the event handler callback function is generally declared as virtual functions to support the user to expand.
   
4. Specific event handler (concrete event handler): is to implement an event processor interface. It implements a service provided by the application. Specific a sum of each event handler descriptor associated. It uses descriptors to identify service events, identify the application provides.
   
5. Reactor Manager (reactor): defines the interface for application control event scheduling, registration and application, delete an event processor and associated descriptors. It is scheduled event processor core. Reactor Manager uses synchronous event separator to an event waiting to happen. Once the event occurs, Reactor separate each event manager first, then schedules an event handler, and finally call the relevant template function to handle this event.

2.1 application scenarios

• Scene: coach on the road, someone on the train was off, but the passengers always want to be able to rest on the bus.
• Traditional practices: Every once in a while (or each station), driver or conductor for each passenger asks if off.
• Reactor practice: the car is subject (Reactor) passengers access after passengers on the train, the conductor Registration (acceptor), the following passengers can rest went to sleep, when it reaches the passenger's destination to be reached (specified event occurs, the passengers to get off the site), the conductor to wake up to.

2.2 more vivid example

This section mainly from: https://blog.csdn.net/russell_tao/article/details/17452997
traditional programming methods
is like to the banking hall, a long queue in front of each window, a window clerk who after one solution to the customer's request. A salesman can enjoy thinking about the issues raised by customer order A, for example:
"I want to buy 20000 XX financial products."
"Look, and 50,000 from the sale."
"Etc., under the current check my balance."
"Balance 50,000. " "
then buy 50,000 it. "
clerk began to enter information.
"Yes, XX financial products annual interest rate of 8%?" ,
"It is expected 8% minimum guaranteed interest-free."
"Early did not say, bye, I buy the balance of treasure."
Clerk expressionless delete the information has been entered transaction rollback.
"next!"

IO multiplexing method
using the IO multiplexing is a master salesman began to challenge the limits, to customers who sign a hand in the operating room large, dense mass of customers who are in the hall, placards questions apply when there is a problem, the master sharp-eyed named designated person to ask questions, after which customers quickly get a reply masters, take some time to think, look up their silver bags, LD under advice to the next question again, until the complete satisfactory answers exit the hall. There was just no time Masters A guide to fill a particular transfer form, came to redeem baht B, B to the exchange after a single, C handle set turn again to live, then D and F in competition for limited ball-point pen: for example, harmonious phenomenon, master of business was stopped temporarily wait.

This is based on event-driven programming IO multiplexing compared to the traditional 1 thread 1 request to have the difficulty of the design point, customers are God, you can not go wrong, not favoritism.

When there is no Reactor, we may design approach is this: to master each customer's questions are recorded when the customer A question, asked what A has done before them first, called in context, then according to Check out the questions and the context of the current banking regulations, targeted to answer a, and the answer is also recorded. After successfully answered all the questions A and delete all record A.

2.3 in the program

A moment, a total of 100,000 concurrent server connections, at this time, a complex IO returned 100 active connections waiting to be processed by the call interface. According to this first connector 100 to find its corresponding object, it is not difficult, the epoll return connection data structure has such a pointer can be used. Then, the processing cycle each connection, to identify the state of the object context at the moment, then the use of read, write the acquired IO network such operation contents, in context status query at this time should be selected which traffic processing method, the method invokes the corresponding after completion of the operation, if the end of the request, delete the object and its context.

In this way, we sank into a process-oriented programming, and in application-oriented, rapid response to the mobile Internet era of the king, to do so sooner or later have to play with his death. Our main concern different types of requests, in different states for different request command to select a different business approach. This will lead to an increase as the type of request, the request to increase the state's request to increase the command of the main program the complexity of rapid expansion, resulting in more and more difficult to maintain, hard to force the programmer no longer dare to take new demands, reconstruction .

Reactor is a way to solve the problem of software engineering, it may not be elegant, the development efficiency is not the highest, but its efficiency and process-oriented use of IO multiplexing is almost equivalent, therefore, whether it is nginx , memcached, redis and so synonymous with these high-performance components, are duty-bound headlong into the arms of the reactor.

Reactor mode in software engineering level, event-driven framework to isolate specific business, the idea of ​​separation between OO different types of requests. Usually, Reactor only use IO multiplexing driving handle network events, but also to achieve timer processing time of an event driven (process timeout process or timed tasks requested), as the following diagram:

The picture has the meaning 5:00:

• OO thinking based processing applications, different types of request processing is isolated. For example, type A user registration request is a request, the request is a query type B user avatar, then when we put new avatar user multiple resolution images, change the code processing logic B type of request, the request does not involve A type of code Modifications.
  
• Logical application processing request, completely separated from the event distribution frame. What does that mean? That write application processing, not to manage IO multiplexing when to call and what to call epoll_wait not to manage, it returns to handle multiple socket connection. Application code, only care how to read, transmit data on the socket, how to handle the business logic. Event distribution frame has an abstract event interface, all applications must implement the abstract event interface, and only then separated by this abstract and application framework.
  
• Available on the Reactor registration, remove the event method, for use by code and distribute event method is usually invoked cycle only, if available to the application code calls, or use by the framework brute direct cycle, which is a free framework.
  
• IO multiplexer is an abstract, which may be specific to select, the epoll may be, they must be collected only in a moment active connection of all connections to be monitored.
  
• Timers are also used by the Reactor object, it must provide at least four methods, including adding, deleting timer event, which is called by the application code. Recently timeout is needed, which will be used reactor object that is used to confirm select or blocking timeout when epoll_wait execution, prevent IO wait influence the process timed events. Traversing frame is used by the reactor for processing a timed event.

2.4 Reactor several models

Reference: http://gee.cs.oswego.edu/dl/cpjslides/nio.pdf
in web services, many of them involving basic operations: read request, decode request, process service, encod reply, send reply and so on.

1 single-threaded mode

This is the simplest of Reactor single-threaded model. Reactor thread is a versatile, responsible for demultiplexing socket, Accept new connection and dispatches the request to the processor in the chain. This model is applicable to the service processing chain processor can quickly complete the assembly of the scene. However, this single-threaded model does not take full advantage of multi-core resources, so not much practical use.

2 Multi-threaded mode (single-Reactor)

The model in the event handler (Handler) chain is partly based on multi-threaded (thread pool), also commonly used model backend.

More than three-threaded mode (multiple Reactor)

Compared to the second model, which is the Reactor divided into two parts, mainReactor is responsible for monitoring and to accept a new connection, and then build the socket assigned to subReactor through a multiplexer (Acceptor). subReactor responsible socket demultiplexing connected, read and write data network; business processing functions to the worker thread pool to complete. Typically, the number equivalent to the number of available subReactor CPU.

Three, Proacotr model

It is Proactor and asynchronous I / O related.
In Reactor mode , waiting for an event separated by a plurality of event or status of the operation application may occur (such as read and write file descriptors, or socket can be read), put the event separator pass event registered in advance. the processor (or event handler callback function), the latter do the actual reading and writing operations.

在Proactor模式中，事件处理者(或者代由事件分离者发起)直接发起一个异步读写操作(相当于请求)，而实际的工作是由操作系统来完成的。发起时，需要提供的参数包括用于存放读到数据的缓存区，读的数据大小，或者用于存放外发数据的缓存区，以及这个请求完后的回调函数等信息。事件分离者得知了这个请求，它默默等待这个请求的完成，然后转发完成事件给相应的事件处理者或者回调。

可以看出两者的区别：Reactor是在事件发生时就通知事先注册的事件（读写由处理函数完成）；Proactor是在事件发生时进行异步I/O（读写由OS完成），待IO完成事件分离器才调度处理器来处理。

举个例子，将有助于理解Reactor与Proactor二者的差异，以读操作为例（类操作类似）。
在Reactor（同步）中实现读：

• 注册读就绪事件和相应的事件处理器
  
• 事件分离器等待事件
  
• 事件到来，激活分离器，分离器调用事件对应的处理器。
  
• 事件处理器完成实际的读操作，处理读到的数据，注册新的事件，然后返还控制权。

Proactor（异步）中的读：

• 处理器发起异步读操作（注意：操作系统必须支持异步IO）。在这种情况下，处理器无视IO就绪事件，它关注的是完成事件。
• 事件分离器等待操作完成事件
  
• 在分离器等待过程中，操作系统利用并行的内核线程执行实际的读操作，并将结果数据存入用户自定义缓冲区，最后通知事件分离器读操作完成。
  
• 事件分离器呼唤处理器。
  
• 事件处理器处理用户自定义缓冲区中的数据，然后启动一个新的异步操作，并将控制权返回事件分离器。

四、常见的I/O编程框架

对比几个常见的I/O编程框架：libevent，libev，libuv，aio，boost.Asio。

4.1 libevent

libevent是一个C语言写的网络库，官方主要支持的是类linux操作系统，最新的版本添加了对windows的IOCP的支持。在跨平台方面主要通过select模型来进行支持。
设计模式 ：libevent为Reactor模式；
层次架构：livevent在不同的操作系统下，做了多路复用模型的抽象，可以选择使用不同的模型，通过事件函数提供服务；
可移植性 ：libevent主要支持linux平台，freebsd平台，其他平台下通过select模型进行支持，效率不是太高；
事件分派处理 ：libevent基于注册的事件回调函数来实现事件分发；
涉及范围 ：libevent只提供了简单的网络API的封装，线程池，内存池，递归锁等均需要自己实现；
线程调度 ：libevent的线程调度需要自己来注册不同的事件句柄；
发布方式 ：libevent为开源免费的，一般编译为静态库进行使用；
开发难度 ：基于libevent开发应用，相对容易，具体可以参考memcached这个开源的应用，里面使用了 libevent这个库。

4.2 libev

libevent/libev是两个名字相当相近的I/O Library。既然是库，第一反应就是对api的包装。epoll在linux上已经存在了很久，但是linux是SysV的后代，BSD及其衍生的MAC就没有，只有kqueue。

libev v.s libevent。既然已经有了libevent，为什么还要发明一个轮子叫做libev？

http://www.cnblogs.com/Lifehacker/p/whats_the_difference_between_libevent_and_libev_chinese.html
http://stackoverflow.com/questions/9433864/whats-the-difference-between-libev-and-libevent

上面是libev的作者对于这个问题的回答，下面是网摘的中文翻译：

就设计哲学来说，libev的诞生，是为了修复libevent设计上的一些错误决策。例如，全局变量的使用，让libevent很难在多线程环境中使用。watcher结构体很大，因为它们包含了I/O，定时器和信号处理器。额外的组件如HTTP和DNS服务器，因为拙劣的实现品质和安全问题而备受折磨。定时器不精确，而且无法很好地处理时间跳变。

总而言之，libev试图做好一件事而已（目标是成为POSIX的事件库），这是最高效的方法。libevent则尝试给你全套解决方案（事件库，非阻塞IO库，http库，DNS客户端）libev 完全是单线程的，没有DNS解析。

libev解决了epoll, kqueuq等API不同的问题。保证使用livev的程序可以在大多数 *nix 平台上运行(对windows不太友好)。但是 libev 的缺点也是显而易见，由于基本只是封装了 Event Library，用起来有诸多不便。比如 accept(3) 连接以后需要手动 setnonblocking 。从 socket 读写时需要检测 EAGAIN 、EWOULDBLOCK 和 EINTER 。这也是大多数人认为异步程序难写的根本原因。

4.3 libuv

libuv是Joyent给Node做的I/O Library。libuv 需要多线程库支持，其在内部维护了一个线程池来 处理诸如getaddrinfo(3) 这样的无法异步的调用。同时，对windows用户友好，Windows下用IOCP实现，官网http://docs.libuv.org/en/v1.x/

4.4 boost.Asio

Boost.Asio类库，其就是以Proactor这种设计模式来实现。
参见：Proactor（The Boost.Asio library is based on the Proactor pattern. This design note outlines the advantages and disadvantages of this approach.），
其设计文档链接：http://asio.sourceforge.net/boost_asio_0_3_7/libs/asio/doc/design/index.html
http://stackoverflow.com/questions/11423426/how-does-libuv-compare-to-boost-asio

4.5 linux aio

linux有两种aio(异步机制)，一是glibc提供的(bug很多，几乎不可用)，一是内核提供的(BSD/mac也提供)。当然，机制不等于编程框架。

最后，本文介绍的同步Reactor模型比较多，后面的章节会以boost.Asio库为基础讲解为什么需要异步编程