Get into the habit of writing together! This is the 12th day of my participation in the "Nuggets Daily New Plan · April Update Challenge", click to view the details of the event .
But let Longcheng Fei will be there, and do not teach Huma to ride the Yin Mountains.
1 Introduction
The understanding of NIO and blocking was described in the previous article, and the concept of multiplexing and Reactor is continued in this article.
2 Multiplexers
Even if nio solves the blocking problem, invalid polling will cause the CPU to idle and waste resources. Using the IO multiplexing technology, when the kernel prepares the data, it notifies the application process to obtain the data, which solves this problem. In the linux operating system, it is divided into three multiplexers of select/poll/epoll according to the different modes of operation. All sockets are monitored by the kernel kernel When the data is ready, a system call is initiated, that is, the system call copies the data from the kernel to the user process.
The feature of I/O multiplexing is that a process can wait for multiple file descriptors at the same time through a mechanism, and any one of these file descriptors (socket descriptors) enters the read-ready state, the select() function to return. The ability to handle multiple connection requests at the same time is its greatest advantage.
2.1 select function operation
select is a system call function provided by the operating system, through which an array of file descriptors can be sent to the operating system, and the operating system can traverse it to determine which file descriptor can be read and written, and then tell us to deal with it. The operation is as follows shown:
select is a calling function provided by the operating system. Through this function, a group of fds can be passed to the operating system. The operating system traverses the fds and returns the number of prepared file descriptors to the user thread. The user thread then traverses the fds one by one to see which one fd is already in the ready state, and then go to the processing.
The characteristics of select are as follows:
-
1 用户需要将监听的 fd list 传入到操作系统内核中，内核来完成遍历操作并将解决返回，这样在高并发场景数组的复制操作下会过多的消耗资源。select 的这一操作仅解决了系统的上下文切换的开销，遍历数组是依旧存在的。select 返回结果是就绪的 fd 个数，用户线程还需要判断哪个 fd 处于就绪状态。
-
2 select 可以传入一组 socket 然后等待内核的处理结果,但是其 list 大小只有 1024 个，每次调用 select 都需要将 fd 数组从用户态复制到内核态，其开销比较大。调用 select 后返回的是就绪 fd 数量，还需要用户再次遍历。
2.2 epoll 操作
针对 select 的缺点，poll 为了增加单次监听 socket 的个数，采用了链表的结构，放弃了数组的结构，但是其核心需要遍历的缺点依然没有解决。针对 select 和 poll 的缺点，epoll 应运而生，其核心主要包括三个方法：
# 在内核开辟一个区域用来存放需要监听的fd
epoll_create
# 向内核中添加、修改、删除需要监控的fd
epoll_ctl
# 返回已经就绪的fd
epoll_wait
复制代码epoll 示意图如下所示：
总结核心内容如下所述：
- 1.内核中存储了一份文件描述符 fd 的集合，无需用户每次都从用户态传入，只需要告诉内核修改的部分就可以。
- 2.内核中不再通过轮询的方式找到就绪的文件描述符 fd，而是通过异步 IO 事件进行唤醒，这里涉及到数据的操作模式。
- 3.内核会将有 IO 事件发生的文件描述符 fd 返回给用户，用户不需要自己进行遍历。
2.2.1 epoll 内部的数据结构
epoll 创建了一个空间用来存储监听的 fd，这些监听的 fd 是按照红黑树的数据结构进行组织，eventpoll 中是 rbr 指向该红黑树，而已经准备好的数据则使用双向链表的结构进行存储，存放在 eventpoll 通过 rdllist 指向该链表。
2.2.2 epoll 的数据操作模式
epoll 的数据操作有两种模式：水平模式 LT（level trigger）和边缘模式 ET（edge trigger）。LT 是 epoll 的默认操作模式，两者的区别在于当触发 epoll_wait 函数时，是否会清空 rdllist。
-
1 LT 模式： epoll_wait 函数检测到有事件发生时需要通知应用程序，但是应用程序不一定及时进行处理，当 epoll_wait 函数再次检测到该事件的时还会通知应用程序，直到事件被处理。可以理解为 mq 发送消息的 at least once 模型。 -
2 ET 模式：epoll_wait 函数检测到事件发生只会通知应用程序一次，后续 epoll_wait 函数将不再监控该事件。因此 ET 模式降低了同一个事件被 epoll 触发的次数，效率比 LT 模式高。可以理解为 mq 发送消息的 exactly once 模型。
3 IO 线程模型
前文是从内核的角度来分析数据收发的模型，在本节中将从用户空间的角度来看数据的收发情况。通过 IO 多路复用就可以监听 IO 事件，通过 dispatch 不断分发事件就像一个反应堆一样，看起来像不断的产生 IO 事件，因此我们称这种模式为 Reactor 模型。Reactor 就是利用 NIO 对 IO 线程进行不同的分工：
- 1 前文中提到的 IO 多路复用器，进行 IO 事件的注册和监听。
- 2 将监听到的就绪 IO 事件通过 dispatch 分发到具体的处理事件 handler 进行 IO 事件处理。
Single Reactor means that there is only one epoll object to listen for all events, including connection events, read and write events. There is only one thread to execute epoll_wait to obtain the ready Socket, and the subsequent operations on the Socket are also processed by this thread. When a new connection comes in, the connection event is triggered, which is handled by the Acceptor, and when there is an IO event, it is handled by the Handler. The main task of the Acceptor is to build a Handler. After obtaining the SocketChannel related to the client, it will be bound to the corresponding Handler. After the corresponding SocketChannel has read and write events, it can be processed. Some of the processes of the handler include decode, compute, encode operations. The single Reactor single thread model is shown in the following figure:
Since a single thread cannot give full play to the advantages of a multi-core cpu, it is not used much, so there is a single-Reactor multi-threading model. When processing business logic, that is, IO read/write or single thread, the decode, compute, encode operations are It is handled by the thread pool, and its model is shown in the following figure:
The master-slave multi-threaded Reactor further divides the tasks. The master Reactor is mainly responsible for monitoring socket events to handle connections. After the connection is established, the socket channel is registered to the slave subReactor. The subReactor maintains its own selector, and after completing the read and write operations of the data, the data processing is handed over to the worker thread for processing. Under this model, the work of each module is more specific, the coupling degree is lower, the performance and stability are also greatly improved, and the number of supported concurrent clients can reach millions. In netty, the specific model implementation is as follows:
// 配置单Reactor单线程
EventLoopGroup eventGroup = new NioEventLoopGroup(1);
ServerBootstrap serverBootstrap = new ServerBootstrap();
serverBootstrap.group(eventGroup);
//配置单Reactor多线程
EventLoopGroup eventGroup = new NioEventLoopGroup();
ServerBootstrap serverBootstrap = new ServerBootstrap();
serverBootstrap.group(eventGroup);
//配置主从Reactor多线程
EventLoopGroup bossGroup = new NioEventLoopGroup(1);
EventLoopGroup workerGroup = new NioEventLoopGroup();
ServerBootstrap serverBootstrap = new ServerBootstrap();
serverBootstrap.group(bossGroup, workerGroup);
复制代码4 Summary
In this article, I mainly describe IO-related operations, mainly involving the understanding of NIO non-blocking concept, as well as the understanding of several models of multiplexing based on the kernel, and finally the content of the IO thread model.