Related video analysis
epoll's network model, from redis, memcached to nginx, work out
the asynchronous principle of linux server performance optimization and realize
handwriting a user-mode network protocol stack, which instantly improves your network skills
Preface
Obsessive-compulsive disorder cannot tolerate this extremely circumstantial concept without giving an explanation. Synchronous/asynchronous, blocking/non-blocking, blocking IO/non-blocking IO/synchronous IO/asynchronous IO/IO multiplexing (IO Multiplexing), select/poll/epoll these The concept has troubled me for a long time, and my personal understanding at this stage is given below.
Synchronous/asynchronous and blocking/non-blocking understanding
Thread is a single sequential control flow in program execution, the smallest unit of program execution flow, and the basic unit of processor scheduling and dispatch. Use threads to execute program flow to understand synchronous and asynchronous, blocking and non-blocking. Synchronous and asynchronous focuses on whether the flow execution process needs to wait for the results of external calls, while blocking non-blocking focuses on the impact of external calls on the flow itself.
Synchronous and asynchronous
During the execution of the thread, an external call is generated. If you need to wait for the call to return to continue the thread flow, it is called synchronization. The situation where the thread flow can continue to execute without waiting for the result to return is called asynchronous.
Distinguish: the thread flow down execution does not need to wait for the result of the system call.
Blocking and non-blocking
In the process of thread execution, after an external call is generated, will the thread flow be "blocked"? The "blocking" corresponds to blocking, and vice versa is non-blocking.
Five IO models of Linux
The description of synchronous/asynchronous, blocking/non-blocking in the previous section can only be said to be able to distinguish them, if it is not in the computer field but in life, the truth is similar. However, some professional terms in the computer need to be viewed in a special context, such as the Linux IO model mentioned below. It is recommended to understand the model itself, rather than the words synchronous/asynchronous and blocking non-blocking, because you will find Even the non-blocking model has a blocking part. The difference between synchronous IO and asynchronous IO is whether the process will be blocked during IO operations.
The final source of the model description below is:
"UNIX®️ Network Programming Volume 1, Third Edition: The Sockets Networking" by Richard Stevens, Section 6.2 "I/O Models"
A synchronous I/O operation causes the requesting process to be blocked until that I/O operation completes;
An asynchronous I/O operation does not cause the requesting process to be blocked;
Synchronous IO model
Judging by blocking in IO operations, 4 of the 5 IO models belong to synchronous IO, which are blocking IO model, non-blocking IO model, IO multiplexing model, and signal-driven IO model.
blocking IO
Blocking IO is the default setting of socket. Its model is shown in the figure below. The program calls recvfrom to generate a system call. After the kernel receives the call request, there are two steps. The first is to wait for the data to be ready, and the second is to transfer the data from the kernel. The space is copied to the user space and then OK is returned. The user space will not continue the execution of the program flow until the system call returns.
non-blocking IO
The non-blocking IO model of socket needs to be set separately. The non-blocking IO model is shown below. After the kernel receives the system call, if the data is not ready to return error immediately, the user process will continue to generate system calls after receiving the error until the data is ready and copied to user space.
[Article benefits] C/C++ Linux server architects are required to learn materials plus group 812855908 (data include C/C++, Linux, golang technology, Nginx, ZeroMQ, MySQL, Redis, fastdfs, MongoDB, ZK, streaming media, CDN, P2P, K8S, Docker, TCP/IP, coroutine, DPDK, ffmpeg, etc.)
IO multiplexing
select/poll/epoll corresponds to the IO multiplexing model. The advantage is that it can monitor multiple sockets. The model diagram is shown below. The user process calls select to generate a system call. The kernel monitors all the sockets that select is responsible for. Once a socket data is ready, the kernel returns, and the user goes to recvfrom to generate a system call to read the data from the kernel space to the user space.
signal-driven IO
The user program registers a signal handler, and then continues to do the following things. When the kernel data is ready, it will send a signal, and the program calls recvfrom to make a system call to copy the data from the kernel space to the user space.
Asynchronous I/O model
Judging by the non-blocking in IO operation, only asynchronous IO is available among the five IO models.
asynchronous IO
The asynchronous IO model is as follows. Aio_read generates a system call. After the data is ready, the kernel copies the data from the kernel space to the user space and returns a signal to inform the read data success. After the program calls aio_read, it continues to execute other parts until the signal is received. Call the handler for processing.
Model comparison
Kernel has two processes, waiting for data to be ready and copying data to user space. User program blocking generally has two situations, select blocking and socket IO blocking. The comparison of the IO models in 5 is as follows.
select/poll/epoll
Select/poll/epoll can monitor multiple file descriptors fd at the same time, and will return these fd when the read and write operations of fd are completed, which can correspond to the system call in the IO multiplexing model to query whether fd is ready for data.
select
int select(int nfds, fd_set *readfds, fd_set *writefds,fd_set *exceptfds,struct timeval *timeout);
typedef struct {
unsigned long *in, *out, *ex;
unsigned long *res_in, *res_out, *res_ex;
} fd_set_bits;
Select uses a structure in the user layer to identify the monitored fd and the status of the monitoring. Each fd is represented by 1 bit, 1 means the file is monitored, and 0 means no monitoring. The bit array pointed to by in, out, and ex represents the corresponding read, write, and abnormal file descriptors. The bit array pointed to by res_in, res_out, res_ex represents the detection result of the corresponding read, write, and abnormal file descriptor.
1. This structure is copied to the kernel layer.
2. Register the callback function __pollwait for all fd.
3. Call the poll method of fd to traverse all the fd of the entire FD_SESIZET and check whether it needs to be monitored. If the monitoring fd is felt For things of interest (file read and write operations are completed or abnormal, refer to the user mode preset settings), the poll method returns a mask that describes whether the read and write operations are ready, and assigns a value to fd_set according to the mask mask. poll->poll_wait->__pollwait will hang the current process to the waiting queue of fd in the inode of the corresponding file.
4. If a round of traversal is unsuccessful, it will be suspended until a timeout or a device driver wakes it up from the waiting queue after its own resource can be read and written, and then a new round of traversal is performed.
5. Copy fd_set from kernel space to user space and delete the process from each waiting queue.
poll
int poll(struct pollfd *fds, nfds_t nfds, int timeout);
struct pollfd {
int fd;
short events;
short revents;
};
The implementation mechanism of poll is similar to that of select. The difference is that in the use of poll, the user mode directly provides information about the fd that needs to be monitored. The pollfd structure records the monitored fd and its status.
struct poll_list {
struct poll_list *next;
int len;
struct pollfd entries[0];
};
In addition, poll uses the poll_list structure to record the monitored fd. Each poll_list node contains a pollfd array. After the parameters are copied to the kernel, the poll_list is traversed. In other words, the pollfd array is traversed. Like select, all fd are traversed.
epoll
int epoll_create(int size);
int epoll_ctl(int epfd, int op, int fd, struct epoll_event *event);
int epoll_wait(int epfd, struct epoll_event *events, int maxevents, int timeout);
Epoll refines the process into a set of system calls, including 1 epoll_create, multiple epoll_ctrl, and 1 epoll_wait. The kernel adds a file system "eventpollfs" for epoll operations. Each or more fd to be monitored has a corresponding eventpollfs file system inode node. The main information is stored in the eventpoll structure, and the monitored file is important The information is stored in the epitem structure. When executing epoll_create and epoll_ctrl, the user state information is saved to the kernel state. Even if epoll_wait is called repeatedly afterwards, it will not repeatedly copy parameters like select/poll, scan fd, and repeatedly put the current process in/out waiting queue.
epoll_create
The function of epoll_create() is to create an inode node of the eventpollfs file system. The main information is stored in the eventpoll structure. Eventpoll records important information of the inde node of the eventpollfs file system. The member rbr saves all the descriptors monitored by the epoll file node. , The organization method is a red-black tree, and the important information of the monitored file is stored in the epitem structure.
epoll_ctl
Epoll_ctl implements a series of operations. It calls ep_find() to obtain the epitem structure from the red-black tree in eventpoll, and selects different operations according to the op parameters. When op is EPOLL_CTL_ADD, epitem cannot be found in the red-black tree of eventpoll in general, so call ep_insert to create an epitem structure and insert it into the corresponding red-black tree.
init_poll_funcptr(&epq.pt, ep_ptable_queue_proc);
…
revents = tfile->f_op->poll(tfile, &epq.pt);
…
Then ep_insert calls init_poll_funcptr to register a callback function ep_ptable_queue_proc, which will be executed when f_op->poll is called. The Ep_ptable_queue_proc function allocates an epoll waiting queue node epoll_entry. On the one hand, it hangs it in the file operation waiting queue, and on the other hand it hangs it in the epitem queue. In addition, it also registers a waiting queue callback function ep_poll_callback, ep_poll_callback is called before completing the operation and waking up the current waiting process, and will put epitem in the completion queue of eventpoll, and then wake up the waiting process.
epoll_waite
The job of epoll_wait is to wait for the file operation to complete and return. Its main body is ep_poll(). This function checks whether there are any completed events in eventpoll, and returns the result if any. If not, call schedule_timeout() to go to sleep until the process is awakened again or timed out.
Compared
The weakness of select/poll is that it needs to poll and traverse fd, which is expensive when there are many fd monitoring; while epoll relies on the callback function, and the overhead is large when there are too many active fd.
