epoll understanding and application
The select multiplexing method has a long history. Therefore, after using this technology, no matter how the performance of the program is optimized, it is impossible to connect hundreds of clients at the same time (of course, the hardware performance is different, and the difference is also very large). This select method is not suitable for the modern development environment where Web server-side development is the mainstream, so you need to learn epoll under the Linux platform.
Reasons for the slow speed of select-based I/O multiplexing technology
In Chapter 12, the select-based IO multiplexing server has been implemented. It is easy to analyze the unreasonable design from the code. The most
important two points are as follows.
□Common loop statement for all file descriptors after calling the select function.
□Every time you call the select function, you need to pass the monitoring object information to the function.
The above two points can be confirmed from lines 45, 49 and 54 of the echo server example in Chapter 12. After calling the select function, instead of collecting the changed file descriptors together, the changed file descriptors can be found by observing the changes of the fd_set variable as the monitoring object (lines 54 and 56 of the example), so the loop statement for all monitoring objects cannot be avoided. Moreover, the fd_set variable as the monitoring object will change, so the original information should be copied and saved before calling the select function (refer to line 45 of the example), and the new monitoring object information should be passed each time the select function is called.
What do you think are the bigger barriers to performance? Is it a common loop statement for all file descriptor objects after calling the select function? Or is it the monitoring object information that needs to be passed each time?
Thinking at the code level, it's easy to think of it as a loop. But compared to the loop statement, the bigger obstacle is passing the monitoring object information each time. Because passing object information has the following meaning: "Every time the select function is called, the monitoring object information is passed to the operating system."
The transfer of data from the application program to the operating system will cause a great burden on the program, and it cannot be solved by optimizing the code, so it will become a fatal flaw in performance.
This shortcoming of the select function can be remedied in the following way: "The monitoring object is only passed to the operating system once, and when the monitoring scope or content changes, only the changed items are notified."
In this way, there is no need to pass the monitoring object information to the operating system every time the select function is called, but the premise is that the operating system supports this processing method (the degree and method of each operating system support are different). The support method of Linux is epoll, and the support method of Windows is IOCP.
select also has advantages
After knowing these contents, some people may be disappointed with the select function, but everyone should master the select function. The epoll method in this chapter is only supported under Limux, that is to say, the improved IO multiplexing model is not compatible. Instead, most operating systems support the select function. As long as the following two conditions are met or required, you should not stick to epoll even on the Linux platform.
□There are few server-side users
□Programs should be compatible
There is no model applicable to all situations. You should understand the pros and cons of each model.
Functions and structures necessary to implement epoll
The epoll function that can overcome the shortcomings of the select function has the following advantages, which are just the opposite of the shortcomings of the previous select function.
□There is no need to write a loop statement for all file descriptors for the purpose of monitoring state changes.
□When calling the epoll_wait function corresponding to the select function, there is no need to pass the monitoring object information each time.
The following introduces the three functions required in the implementation of the epoll server. I hope that you can understand the functions of these functions in combination with the advantages of the epoll function.
□epoll_create: Create a space to save the epoll file descriptor.
□epoll_ctl: Register or unregister file descriptors like space.
□epoll_wait: Similar to the select function, it waits for the file descriptor to change.
In the select mode, the fd_set variable is directly declared in order to save the monitor target file descriptor. However, in the epoll mode, the operating system is responsible for saving the monitoring object file descriptor, so it is necessary to request the operating system to create a space for saving the file descriptor. The function used at this time is epoll_create.
In addition, in order to add and delete the monitoring object file descriptor, FD_SET, FD_CLR functions are required in the select method. But in the epoll mode, the operating system is requested to complete through the epoll_ctl function. Finally, the select function is called in select mode to wait for the change of the file descriptor, while the epoll_wait function is called in epoll. In addition, in the select mode, the fd_set variable is used to check the state change of the monitoring object (whether the event occurs or not), while in the epoll mode, the changed (event-occurring) file descriptors are collected separately through the following structure epoll_event.
struct epoll_event{
__uint32_t events;
epoll_data_t data;
}
typedef union epoll_data{
void *ptr;
int fd;
__uint32_t u32;
__uint64_t u64;
}epoll_data_t;After declaring a large enough epoll_event structure array, when passed to the epoll_wait function, the changed file descriptor information is filled in the array. Therefore, there is no need to loop through all file descriptors like the select function.
epoll_create
epoll was introduced from the Linux2.5.44 kernel, so you need to verify the Linux kernel version before using epoll. Let's take a closer look at the epoll_create function.
#include<sys/epoll.h>
int epoll_create(int size);//成功时返回epoll文件描述符，失败时返回-1
size //epoll示例的大小The file descriptor storage space created when the epoll_create function is called is called the "epoll routine". Another point to note is that the size passed in is not actually used to determine the size of the epoll routine, but is only for reference by the operating system.
The resources created by the epoll_create function are the same as sockets and are also managed by the operating system. Therefore, the function returns a file descriptor as in the case of creating a socket. When termination is required, the close function is also called the same as other file descriptors.
epoll_ctl
#include<sys/epoll.h>
int epoll_ctl(int epfd,int op,int fd,struct epoll_event* event);
//成功时返回0，失败时返回-1
epfd //用于注册监视对象epoll例程的文件描述符
op //用于指定监视对象的添加，删除或更改等操作
fd //需要注册监视对象文件描述符
event //监视对象的事件类型Next are two simple examples
epoll_ctl(A,EPOLL_CTL_ADD,B,C);
epoll_ctl(A,EPOLL_CTL_DEL,B,NULL);The meaning of the first sentence is: "The file descriptor B is registered in the epoll routine A, and the main purpose is to monitor the events in the parameter C"
The meaning of the second sentence is: "Delete file descriptor B from epoll routine A"
As can be seen from the above call statement, when deleting from the monitoring object, the monitoring type (event information) is not required, so NULL is passed to the fourth parameter.
Next, we will introduce the constants and meanings that can be passed to the second parameter of epoll_ctl.
□ EPOLL_CTL_ADD: Register the file descriptor to the epoll routine.
□ EPOLL_CTL DEL: Delete the file descriptor from the epoll routine.
□ EPOLL _CTL_MOD: Change the occurrence of the concerned event of the registered file descriptor.
The following explains the fourth parameter of the epoll_ctl function that you are not familiar with. Its type is the epoll_event structure pointer mentioned before. As mentioned earlier, the epoll_event structure is used to save the collection of file descriptors where events occur. But it can also be used to register concerned events when registering file descriptors in the epoll routine! (There are two functions) The definition of the epoll_event structure in the function is not conspicuous, so the application of the structure in the epoll_ctl function is explained through the call statement.
struct epoll_event event;
....
event.events=EPOLLIN;//事件种类
event.data.fd=sockfd;
epoll_ctl(epfd,EPOLL_CTL_ADD,sockfd,&event);
....The above code registers sockfd to the epoll routine epfd, and generates corresponding events when data needs to be read. Next, the constants that can be saved in the member events of epoll_event and the type of events referred to are given.
□ EPOLLIN: When data needs to be read.
□ EPOLLOUT: The output buffer is empty and the data can be sent immediately.
□ EPOLLPRI: The case of receiving OOB data.
□ EPOLLRDHUP: Disconnected or half-closed, which is very useful in edge-triggered mode.
□ EPOLLERR: An error condition occurred.
□ EPOLLET: Get event notification in edge-triggered way.
□ EPOLLONESHOT: After an event occurs, the corresponding file descriptor will no longer receive event notifications. Therefore, you need to pass EPOLL_CTL_MOD to
the second parameter of the epoll_ctl function to set the event again.
Multiple of the above parameters can be passed simultaneously via bitwise OR operation. About "Edge Trigger" will be explained separately later, just remember EPOLLIN for now.
epoll_wait
Finally, the epoll_wait function corresponding to the select function is introduced, which is called last by default among epoll related functions.
#include<sys/epoll.h>
int epoll_wait(int epfd,struct epoll_event*events,int maxevent,int timeout);
epfd //表示事件发生监视范围的epoll例程的文件描述符
events //保存发生事件的文件描述符集合的结构体地址
maxevents //表示第二个参数中可以保存的最大事件数
timeout //以1/1000秒为单位的等待时间，传递-1时，一直等待直到发生事件The function is called as follows. It should be noted that the buffer pointed to by the second parameter needs to be allocated dynamically.
int event_cnt;
struct epoll_event* ep_events;
....
ep_events=malloc(sizeof(struct epoll_event)*EPOLL_SIZE);//EPOLL_SIZE是宏常量
....
event_cnt=epoll_wait(epfd,ep_event,EPOLL_SIZE,-1);
....After calling the function, return the number of file descriptors where the event occurred, and save the set of file descriptors where the event occurred in the buffer pointed to by the second parameter. So there is no need to insert a loop over all file descriptors like select does.
Echo server based on epoll
#include<stdio.h>
#include<stdlib.h>
#include<string.h>
#include<unistd.h>
#include<arpa/inet.h>
#include<sys/socket.h>
#include<sys/epoll.h>
#define BUF_SIZE 100
#define EPOLL_SIZE 50
void error_handling(char *message);
int main(int argc,char *argv[]){
int serv_sock,clnt_sock;
struct sockaddr_in serv_addr,clnt_addr;
socklen_t addr_sz;
int str_len,i;
char buf[BUF_SIZE];
struct epoll_event *ep_events;
struct epoll_event event;
in epfd,event_cnt;
if(argc!=2){
printf("Usage : %s <port>\n",argv[0]);
exit(1);
}
serv_sock=socket(PF_INET,SOCK_STREAM,0);
memset(&serv_addr,0,sizeof(serv_addr));
serv_addr.sin_family=AF_INET;
serv_addr.sin_addr.s_addr=htonl(INADDR_ANY);
serv_addr.sin_port=htons(argv[1]);
if(bind(serv_sock,(struct sockaddr*)&serv_addr,sizeof(serv_addr))==-1)
error_handling("bind() error");
if(listen(serv_sock,5)==-1)
error_handling("listen() error");
epfd=epoll_create(EPOLL_SIZE);
ep_events=malloc(sizeof(struct epoll_event)*EOLL_SIZE);
event.events=EPOLLIN;
event.data.fd=serv_sock;
epoll_ctl(epfd,EPOLL_CTL_ADD,serv_sock,&event);
while(1){
event_cnt=epoll_wait(epfd,ep_events,EPOLL_SIZE,-1);
if(event_cnt==-1){
puts("epoll_wait() error");
break;
}
for(i=0;i<event_cnt;;++i){
if(ep_events[i].data.fd==serv_sock){
addr_sz=sizeof(clnt_addr);
clnt_sock=accept(serv_sock,(struct sockaddr*)&clnt_addr,sizeof(addr_sz));
event.events=EPOLLIN;
event.data.fd=clnt_sock;
epoll_ctl(epfd,EPOLL_CTL_ADD,clnt_sock,&event);
printf("connected client: %d \n",clnt_sock);
}
else{
str_len=read(ep_events[i].data.fd,buf,BUF_SIZE);
if(str_len==0){ //传输完成
printf("close client: %d \n",ep_events[i].data.fd);
close(ep_events[i].data.fd);
epoll_ctl(epfd,EPOLL_CTL_DEL,ep_events[i].data.fd,NULL);
}
else{
write(ep_events[i].data.fd,buf,str_len);//回声
}
}
}
}
close(serv_sock);
close(epfd);
return 0;
}
void error_handling(char *message){
fputs(buf,stderr);
fputc('\n',stderr);
exit(1);
}
The demonstration of the above code is similar to the select function code in the previous chapter, and can be understood in combination with the code ideas in the previous chapter.
Condition-triggered and edge-triggered
Some people often cannot correctly distinguish between Level Trigger and Edge Trigger when learning epoll,
but only when they understand the difference between the two can they fully master epoll.
The difference between conditional triggering and edge triggering is the point in time when the event occurs
First, an example is given to help you understand conditional triggering and edge triggering. Observing the following dialogue, you can understand the characteristics of the conditional trigger event through the content of the dialogue.
Son: "Mom, I got all A's on this final exam."
Mom: "Great!"
Son: "My total score ranks first in the class."
Mom: "Good job!"
Son: "But my total score is only tenth in the grade"
Mom: "Don't be discouraged, it's fine!"
It can be seen from the above dialogue that the son has been reporting to his mother since he said the final exam , which is the principle of conditional triggering. I organize it as follows: "In the conditional trigger mode, as long as the input buffer has data, it will always be notified."
For example, when the server-side input buffer receives 50 bytes of data, the server-side operating system will notify the event (registered to the changed file descriptor). However, when there are 30 bytes left after the server reads 20 bytes, the event will still be registered. That is to say, in the condition trigger mode, as long as there is data left in the input buffer, it will register again in the event mode. Next, introduce the event characteristics of edge triggering through the following dialogue.
Son: "Mom, I have a final exam."
Mom: "How did you do in the exam?"
son:"........."
Mom: "Talk! Did you fail the exam?"
As can be seen from the above dialogue, the event is only registered once when the input buffer receives data in the edge trigger.
Even if data remains in the input buffer , it will not be registered again.
Master the event characteristics of conditional triggering
Next, use the code to understand the event registration method triggered by the condition. By default, epoll works in a condition-triggered manner, so you can verify the condition-triggered features through this instance.
#include <"与之前示例的头文件声明一致，故省略。">
#define BUF_SIZE 4
#define EPOLL_SIZE 50
void error_handling(char *buf);
int main(int argc, char *argv[]){
int serv_sock, clnt_sock;
struct sockaddr_in serv_adr, clnt_adr;
socklen_t adr_sz;
int str_len, i;
char buf[BUF_SIZE];
struct epoll_event *ep_events;
struct epoll_event event;
int epfd, event_cnt;
if(argc!=2){
printf("Usage : %s <port>\n", argv[0]);
exit(1);
}
serv_sock=socket(PF_INET, SOCK_STREAM, 0);
memset(&serv_adr, 0, sizeof(serv_adr));
serv_adr.sin_family=AF_INET;
serv_adr.sin_addr.s_addr=htonl(INADDR_ANY)
serv_adr.sin_port=htons(atoi(argv[1]));
if(bind(serv_sock,(struct sockaddr*) &serv_adr, sizeof(serv_adr))==-1)
error_handling("bind() error");
if(listen(serv_sock,5)==-1)
error_handling("listen() error");
epfd=epoll_create(EPOLL_SIZE);
ep_events=malloc(sizeof(struct epoll_event)*EPOLL_SIZE);
event.events=EPOLLIN;
event.data.fd=serv_sock;
epoll_ctl(epfd, EPOLL_CTL_ADD, serv_sock, &event);
while(1){
event_cnt=epoll_wait(epfd,ep_events，EPOLL_SIZE，-1);
if(event_cnt==-1){
puts("epoll_wait() error");
break;
}
puts("return epoll_wait");
for(i=0; i<event_cnt; i++){
if(ep_events[i].data.fd==serv_sock){
adr_sz=sizeof(clnt_adr);
clnt_sock=accept(serv_sock, (struct sockaddr*)&clnt_adr, &adr_sz);
event.events=EPOLLIN;
event.data.fd=clnt_sock;
epoll_ctl(epfd, EPOLL_CTL_ADD, clnt_sock,&event);
printf("connected client: %d \n", clnt_sock);
}
else{
str_len=read(ep_events[i].data.fd, buf, BUF_SIZE);
if(str_len==0){ //关闭套接字
printf("closed client: %d \n", ep_events[i].data.fd);
close(ep_events[i].data.fd);
epoll_ctl(epfd, EPOLL_CTL_DEL, ep_events[i].data.fd, NULL);
}
else{
write(ep_events[i].data.fd,buf,str_len);//回声
}
}
}
}
close(serv_sock);
close(epfd);
return 0;
}
void error_handling(char *buf){
//与之前实例相同，故省略
}The above example differs from the previous as follows.
□ Reduce the buffer size used when calling the read function to 4 bytes (line 2)
□ Insert a statement to verify the number of epoll_wait function calls (line 50)
to reduce the buffer size to prevent the server from reading the received data at one time. In other words, after calling the read function, there is still data to be read in the input buffer. And it will therefore register a new event and will loop out the "return epoll_wait" string when returning from the epoll_wait function.
It can be seen from the running results that whenever the client data is received, the event will be registered, and therefore the epoll_wait function will be called multiple times. Let's change the above example to be edge-triggered, which requires some extra work. But I want to validate the way the edge-triggered model registers events with minimal changes. Change line 57 of the above example to the following to run the server and client:
event.events = EPOLLIN|EPOLLET;
After the change, the following facts can be verified: "When receiving data from the client, the 'return epoll_wait' string is output only once, which means that only one event is registered." Although the above facts can be verified, an error will occur when the client runs. Have you encountered this kind of problem? Can you analyze the cause yourself? While there's no need to confuse this at this point, understanding the nature of edge triggering should make it possible to analyze the cause of the error.
Two points that must be known in the server-side implementation of edge triggering
The following two points are must-know contents for implementing edge triggering.
□ Verify the cause of the error through the errno variable.
□In order to complete non-blocking (Non-blocking) I/O, change the socket characteristics.
Linux socket-related functions generally notify that an error has occurred by returning -1. Although it is known that an error has occurred,
the cause of the error cannot be known from these contents alone. Therefore, to provide additional information when an error occurs, Linux declares the following
global variables:
int errno;In order to access this variable, the error.h header file needs to be introduced. In addition, when an error occurs in each function, the value saved in the errno variable is different, and it is not necessary to remember all possible values. In the process of learning each function, master it one by one, and refer to it when necessary. This section only introduces the following types of errors: "The read function returns -1 when it finds that there is no data to read in the input buffer, and saves the EAGAIN constant in errno."
The usage of errno will be given later by example. The following explains how to change the socket to non-blocking mode. Linux provides the following methods (used in Chapter 13) to change or read file attributes.
#include<fcntl.h>
int fcntl(int filedes,int cmd,...);
//成功是返回cmd参数相关值，失败时返回-1
filedes //目标的文件描述符。
cmd //表示函数调用的目的。As can be seen from the above statement, fcntl has a variable parameter form. If F_GETFL is passed to the second parameter , the file descriptor attribute (int type) pointed to by the first parameter can be obtained. Conversely, if F_SETFL is passed, the file descriptor attributes can be changed . If you want to change the file (socket) to non-blocking mode, you need the following two statements.
int flag = fcntl(fd, F_GETFL, 0);
fcntl(fd，F_SETFL，flag|O_NONBLOCK);Obtain the previously set attribute information through the first statement, and add the non-blocking O_NONBLOCK flag on this basis through the second statement. When the read&write function is called, a non-blocking file will be formed regardless of whether data exists. The scope of application of the fcntl function is very wide. You can either summarize all applicable situations at one time when learning system programming, or master them one by one every time you need them.
Implementing an edge-triggered echo server-side
The reason why the method of reading the cause of the error and the method of socket creation in non-blocking mode is introduced is that both are closely related to the server-side implementation of edge triggering.
First explain why you need to confirm the cause of the error through errno: "In the edge trigger mode, the event is only registered once when receiving data." Because of this feature, once an
input-related event occurs, all data in the input buffer should be read. Therefore, it is necessary to verify
whether the input buffer is empty. (Otherwise the socket will not be able to logout)
That being the case, why do you need to turn the socket into non-blocking mode? In the edge-triggered mode, the read&write function that works in a blocking manner may cause a long pause on the server side (waiting for data to arrive). Therefore, the non-blocking read&write function must be used in the edge trigger mode. An echo server-side example working in an edge-triggered manner is given next.
#include<“添加fcntl.h、errno.h，其他与之前示例的头文件声明一致。“>
#include <fcntl.h>
#include <errno.h>
#define BUF SIZE 4
#define EPOLL_SIZE 50
void setnonblockingmode(int fd);
void error_handling(char *buf);
int main(int argc, char *argv[]){
int serv_sock,clnt_sock;
struct sockaddr_in serv_adr, clnt_adr;
socklen_t adr_sz;
int str_len, i;
char buf[BUF_SIZE];
struct epoll_event *ep_events;
struct epoll_event event;
int epfd, event_cnt;
if(argc!=2){
printf("Usage : %s <port>\n",argv[0]);
exit(1);
}
serv_sock=socket(PF_INET, SOCK_STREAM, 0);
memset(&serv_adr, 0, sizeof(serv_adr));
serv_adr.sin_family=AF_INET;
serv_adr.sin_addr.s_addr=htonl(INADDR_ANY);
serv_adr.sin_port=htons(atoi(argv[1]));
if(bind(serv_sock,(struct sockaddr*) &serv_adr, sizeof(serv_adr))==-1)
error_handling("bind() error");
if(listen(serv_sock,5)==-1)
error_handling("listen()error");
epfd=epoll_create(EPOLL_SIZE);
ep_events=malloc(sizeof(struct epoll_event)*EPOLL_SIZE);
setnonblockingmode(serv_sock);
event.events=EPOLLIN;
event.data.fd=serv_sock;
epoll_ctl(epfd, EPOLL_CTL_ADD, serv_sock, &event);
while(1){
event_cnt=epoll_wait(epfd, ep_events, EPOLL_SIZE,-1);
if(event_cnt==-1){
puts("epoll_wait() error");
break;
}
puts("return epoll_wait");
for(i=0; i<event_cnt; i++){
if(ep_events[i].data.fd==serv_sock){
adr_sz=sizeof(clnt_adr);
clnt_sock=accept(serv_sock,(struct sockaddr*)&clnt_adr,&adr_sz);
setnonblockingmode(clnt_sock);
event.events=EPOLLIN|EPOLLET;
event.data.fd=clnt_sock;
epoll_ctl(epfd，EPOLL_CTL_ADD, clnt_sock,&event);
printf("connected client: %d \n", clnt_sock);
}
else{
while(1){
str_len=read(ep_events[i].data.fd, buf, BUF_SIZE);
if(str_len==0){ // 客户端关闭连接
printf("closed client: %d \n", ep_events[i].data.fd);
close(ep_events[i].data.fd);
epoll_ctl(epfd, EPOLL_CTL_DEL, ep_events[i].data.fd, NULL);
break;
}
else if(str_len<0){
if(errno==EAGAIN)break;
else {
write(ep_events[i].data.fd, buf, str_len); // 回声
}
}
}
}
}
close(serv_sock);
close(epfd);
return 0;
}
void setnonblockingmode(int fd){
int flag=fcntl(fd,F_GETFL,0);
fcntl(fd,F_SETFL,flag|O_NONBLOCK);
}
void error_handling(char *buf){
//示例和之前的示例相同，故省略
}
Conditional trigger and edge trigger are stronger and weaker
We have a good understanding of conditional triggering and edge triggering from a theoretical and code perspective, but this alone does not understand
the advantages of edge triggering over conditional triggering. In the edge trigger mode, the following can be achieved: "The time point of receiving data and processing data can be separated!"
Although relatively simple, it is a very accurate and powerful illustration of the advantages of edge triggering. Regarding the meaning of this sentence, you
will have a deeper understanding when you open different types of programs in the future. At this stage, the following scenario is given to help everyone understand: There is a server and three clients. These three clients are A, B, and C. They send their corresponding parts of data to the server respectively. The server needs to combine these data and send them to any host in the forward order of A, B, and C.
Then the operation process of the corresponding server is as follows:
□The server receives data from clients A, B, and C respectively.
□The server reassembles the received data in the order of A, B, and C.
□The combined data will be sent to any host.
In order to complete the process, if the program can be run according to the following process, the implementation on the server side is not difficult.
□The client connects to the server in the order of A, B, and C, and sends data to the server in sequence.
□Clients that need to receive data should connect to the server and wait before clients A, B, and C.
However, in reality, the following situations may frequently occur. In other words, the following situations are more realistic.
□Clients C and B are sending data to the server, but A has not yet connected to the server.
□Clients A, B, and C send data out of sequence.
□The server has received the data, but the target client to receive the data has not yet connected to the server.
Therefore, even if the input buffer receives data (register corresponding events), the server side can also decide
the time point to read and process the data, which brings great flexibility to the implementation of the server side. Compared with conditional triggering, if you try to separate the time of receiving data and processing data, a corresponding event will be generated every time the epoll_wait function is called. And the number of events will also accumulate, can the server accept it?