Detailed Explanation of Linux Network Communication epoll (10) -【Linux Communication Architecture Series】

Series Article Directory

C++ skill series
Linux communication architecture series
C++ high-performance optimization programming series
Deep understanding of software architecture design series
Advanced C++ concurrent thread programming

Looking forward to your attention! ! !

现在的一切都是为将来的梦想编织翅膀，让梦想在现实中展翅高飞。
Now everything is for the future of dream weaving wings, let the dream fly in reality.

1. Introduction to epoll technology

(1) I/O multiplexing technology is used to monitor data transmission and reception on multiple TCP connections, whileepoll is a typical technology that uses I/O multiplexing on Linux and supports high concurrency. Traditional select and poll are also I/O multiplexing technologies, but these two technologies are limited by internal implementation and do not support high concurrency. If more than 1000 clients are connected at the same time, the performance will drop significantly. (epoll technology was introduced from linux kernel 2.6).

(2) The performance of epoll technology can be said to be very amazing, it isThe core technology that enables a single computer to support millions or even hundreds of thousands of concurrent, is far superior to other I/O models or I/O functions (such as select and poll functions). Due to the internal implementation problems of the system, technologies such as select and poll, when the number of concurrency (clients are connected at the same time) exceeds 1000~2000 The performance begins to decline sharply, but epoll technology has no such problem at all (the performance will not decrease significantly as the number of concurrency increases). Of course, the higher the number of concurrency, the more memory is required. Therefore, even if the sharp increase in the number of concurrency has little effect on performance, the memory is always limited. In other words, the number of concurrency is always limited, which is impossible. Unlimited increase.

(3) Even if there are 100,000 concurrent connections (100,000 clients remain connected to the server at the same time), it is usually impossible for these 100,000 connections to send and receive data at the same time.Generally, only dozens or hundreds of connections are sending and receiving data at the same time, and other connections may be in a state of only connecting but not sending and receiving data.如果以100ms为间隔判断一次，可能这100ms内只有100个活跃连接（有数据收发的连接），把这100个活跃连接的数据放在一个专门的地方，后续到这个专门的地方来，只需要处理100条数据。Is it stress-free to handle? This is how epoll handles it. And select and poll judge in turn whether there is any data sent from these 100,000 connections (in fact, only 100 connections have data), and if there is data, it will be processed. It is not difficult to imagine that checking 100,000 connections each time is a huge waste of resources and time compared to checking 100 connections each time, so when the number of concurrency exceeds 1000 to 2000, the select and poll technologies (or this function, This model) performance will drop dramatically.

(4) Many server programs that handle network communications aremulti-Progress(Each process corresponds to a client connection), there are alsoMultithreading(Each thread corresponds to a client connection), but the number of processes or threads increases, even if the consumption of the processes or threads is not counted, the frequent switching of time slices/contexts between processes or threads consumes a lot of performance. The epoll technology is a simple, crude and effective technology that uses an event-driven mechanism to collect and process events only in a separate process or thread, without the cost of switching between processes or threads.

2. Working principle of epoll

2.1 epoll_create function - [ create an epoll object ]

When the user process calls epoll_create, the kernel will create a struct eventpoll kernel object and associate it with the open file list of the current process.

2.1.1 epoll_create format

• The format of the epoll_create function is as follows：
  int epoll_create(int size);

2.1.2 epoll_create function

• Create an epoll object, return an object (file) descriptor to identify the epoll object, and then send and receive data by operating the descriptor; the object must be closed eventually, because it is a descriptor,
  or a handle, Always be closed;
  the size in the format must be greater than 0 to avoid unexpected problems;

2.1.3 Principle of epoll_create

Figure 2_1 epoll structure

• Find the source code of this function implementation in the source code: **Generate an eventpoll object** (imagine the system generates a structure) There are many members in the eventpoll object, and here we only focus on the sum of them . 000
  struct eventpoll *ep = (struct eventpoll * )calloc(1, sizeof(struct eventpoll));

rbrrdlist

  ① rbr . This member can be understood as a red-black tree root node (pointer). A red-black tree is a data structure used to save data, generally storing "key/value (key/value) pairs". The characteristic of the red-black tree is that it can find and retrieve the value (value) very quickly according to the given key (key). The key here is generally a number, and the value may represent a batch of data. If value is a data structure, you can quickly find value (a structure with a batch of data) by searching in the red-black tree with a number (key). Because the search speed of the red-black tree is fast and the efficiency is high, the red-black tree is introduced in the epoll technology.
  使用红黑树，为了支持对海量连接的高效查找、插入、删除，eventpoll内部使用一颗红黑树，通过这棵树来管理用户进程下添加进来的所有socket连接。
  

  ② rdlist . This member can be understood as a header pointer representing a doubly linked list. A doubly linked list is also a data structure, which is characterized by a very fast sequential access to the nodes inside, just go down (traverse) along its chain. Compared with the red-black tree above, the red-black tree randomly finds any node quickly, and the doubly linked list visits each node sequentially, each with its own characteristics and uses. ③ wq. Waiting queue linked list.
  就绪的描述符链表。当有连接就绪的时候，内核会把就绪的连接放到rdlist链表里。这样应用进程只需要判断链表就能找出就绪连接，而不是去遍历整颗树。


软中断数据就绪的时候会通过wq来找到阻塞在epoll对象上的用户进程。

• Summarize the epoll_create function:
  (1) Create an event_poll structure object and save it by the system.
  (2) The rbr member in the object is initialized to point to the root of a red-black tree (with this root, you can insert nodes into the red-black tree, or insert data).
  (3) The rdlist member in the object is initialized to point to the root of a doubly linked list (with this root, nodes (data) can be inserted into the doubly linked list).

Next, let's see how the system uses the eventpoll structure object to handle up to millions of concurrency.

2.2 epoll_ctl function - [Add/delete and modify a (socket) management link to the epoll object]

2.2.1 epoll_ctl format

• The format of the epoll_create function is as follows：
  int epoll_ctl(int efpd, int op, int socketid, struct epoll_event *event);

2.2.2 epoll_ctl function

• Add a socket and socket-related events to the epoll object descriptor. The epoll object has been used to monitor the data exchange on the socket (that is, the TCP connection). When there is data exchange, the system will notify the program.
  We will use the epoll_ctl function to add events that need attention (interest) in the program (about 7 to 8 events in the entire system) to the epoll object descriptor, and the system will notify the program when these events arrive.
  ①The 参数efpd。epoll object descriptor returned from epoll_create.

  ② 参数op。An operation type (macro definition)
  EPOLL_CTL_ADD: Add the associated event on the sockid.
  EPOLL_CTL_MOD: Modify the associated events on sockid.
  EPOLL_CTL_DEL: delete the associated event on sockid.
  add eventLater, when such an event comes, the system will notify the program to handle it. The so-called adding event is to add a node to the red-black tree. After each client connects to the server, the server will create a corresponding socket (the return value of the accept function) to communicate with the client, because the operating system will ensure that the socket value of each connection to the server is not repeated, so the system will The node will be added to the red-black tree with the socket value as the key (the key requirements of the red-black tree cannot be repeated).
  modify eventIt is to modify some values ​​​​in the red-black tree nodes. So if you want to modify the event, you must first call EPOLL_CTL_ADD to add the event to the red-black tree. If you originally added 3 events in the epoll object descriptor, and now you want to modify it to only focus on 2 events, you need to call EPOLL_CTL_MOD.
  delete eventIf you originally paid attention to 3 events, but now want to reduce 1 event to focus on 2 events, you need to call EPOLL_CTL_MOD instead of EPOLL_CTL_DEL. The real action of EPOLL_CTL_DEL is to delete the node from the red-black tree (not close the TCP connection), which will prevent the program from receiving all event notifications on the TCP connection , so this item is only used when needed.

  ③A 参数sockid。TCP connection. When adding an event (adding a node to the red-black tree), it is to use the socketid as the key to add a node to the red-black tree.

  ④ 参数event。Pass information to the epoll_ctl function. If you want to add some events, you can pass the specific events into the epoll_ctl function through the event parameter.
  Event Type:
  EPOLLIN: A situation where data needs to be read.
  EPOKKOUT: The output buffer is empty and data can be sent immediately.
  EPOLLPRI: In case OOB data is received.
  EPOLLRDHUP: Disconnected or half-closed condition, which is useful in edge-triggered mode.
  EPOLLERR: An error condition has occurred.
  EPOLLET: Get notified of events in an edge-triggered manner.
  EPOLLONESHOT: After an event occurs, the corresponding file descriptor no longer receives event notifications. Therefore, you need to pass EPOLL_CTL_MOD to the second parameter of the epoll_ctl function to set the event again.

2.2.3 Principle of epoll_ctl

• ① Let's look at the source code implementation. If an EPOLL_CTL_ADD is passed in, first check whether the node already exists in the red-black tree, if yes, return directly, if not, the program goes down. If it is confirmed that the red-black tree does not have the node, an epitem object is generated at this time, and the epitem object is a node added to the red-black tree later.
  Figure 2_2_3 is about to insert a node into the red-black tree, the key of the node is stored in the member sockfd, the event to be added is stored in the member event, and then the node is inserted into the red-black tree. For a red-black tree, each node must record its own left subtree, right subtree, and parent node. In the figure, the rbn member itself is a structure type, which includes points to the left subtree, right The pointer member of the node. If multiple users connect to the server in the future, many nodes need to be added to this red-black tree, and these nodes must also be connected to each other.
  In short, for each node of the red-black tree, through rbn members, if there is a parent node, it points to the parent node, if there is a child node, it points to the child node, if there are both parent nodes and child nodes, it points to both the parent node and the child nodes.
  Who adds nodes to the red-black tree?
  It is actually epoll_ctl (EPOLL_CTL_ADD), and each red-black tree node actually represents a TCP connection.
  ② If an EPOLL_CTL_MOD is passed in, find the existing red-black tree node and modify some data (event) in the node.
  ③ If EPOLL_CTL_DEL is passed in, find the existing red-black tree node, delete the node from the red-black tree, and release the corresponding memory. After deleting a node from the red-black tree, the TCP connection corresponding to the node There is no way to know what happened.

• Summary: EPOLL_CTL_ADD is equivalent to adding a node to the red-black tree; EPOLL_CTL_MOD is equivalent to modifying the node of the red-black tree; EPOLL_CTL_DEL is equivalent to deleting the node from the red-black tree.
  Therefore, each connected client should call epoll_ctl to add a red-black tree node to the red-black tree . If there are 1 million concurrent connections, there will be 1 million nodes on the red-black tree.
  Now, these 1 million connections have been added to the red-black tree, and events of interest to related programs have also been added to the nodes of the red-black tree. When these events occur on some TCP connections (such as connection, disconnection) , when there is data sending and receiving, etc.), the operating system will notify the program.
  How does the program receive notifications from these operating systems? Next, let's look at the epoll_wait function.

2.3 epoll_wait function - [waiting for I/O events on its management connection]

2.2.1 epoll_wait format

• The format of the epoll_create function is as follows：
  int epoll_wait(int epfd, struct epoll_event *events, int maxevents, int timeout);

2.2.2 epoll_wait function

• Block for a short period of time and wait for the event to occur, return the event collection, and obtain the kernel's time notification. In other words, it is to traverse the doubly linked list, copy the data related to the node in the doubly linked list, and delete the node from the doubly linked list. Because all sockets (TCP connections) with data are recorded in the doubly linked list.
  ①The 参数efpdepoll object object descriptor returned from epoll_create.
  ② 参数eventsAn array with a length of maxevents, indicating that calling the epoll_wait function can collect at most maxevents ready (ready) read and write events. The actual read and write events are determined by the return value of this function (in other words, the number of TCP connections that have events occurred is returned, but due to memory limitations, events may occur on 100 TCP connections, but the returned number is 80 — — less than 100).
  ③The 参数timeoutlength of waiting for blocking.
  Generally speaking, this function is to go to the doubly linked list, take the connection that has an event in the connection that is connected at the same time at the moment, and then use functions such as read, write or send, recv to send and receive data. As long as a certain socket is in the doubly linked list, a certain/certain event must have occurred in the socket, in other words, only the socket in which a certain/certain event has occurred will appear in the doubly linked list.
  This is why epoll is efficient, because epoll only traverses a small part of the socket connections where events occur each time (these sockets are in this doubly linked list), instead of traversing all socket connections one by one to determine whether the event has arrived.
  Epitem is a red-black tree node and also a doubly-linked list node, so this epitem node is designed very cleverly and is very versatile. It can be added to the red-black tree as a node of the red-black tree, and can also be used as a doubly-linked list A node is added to the doubly linked list, so when the node is fetched from the doubly linked list through the epoll_wait function, the epitem node is still taken out.
  The rdlink member has 2 pointers, so that the epitem node can be inserted into the doubly linked list.
  If events are received on 3 TCP connections, then these 3 TCP connections must all stay in the doubly linked list (of course they also stay in the red-black tree at the same time).

2.2.3 Principle of epoll_wait

• In the source code, find the source code of the function implementation,
  ① while loop, used to wait for a short period of time (such as 100ms). The nodes (socket connections) of events that occur within this short period of time will be placed in a doubly linked list by the operating system.
  ② After the waiting time is up, determine the number of events returned to the caller program that called this function (epoll_wait) this time.
  Use a while loop to return the information of this batch of events to the caller program. Note that the node returned to the caller program is removed from the doubly linked list (the node is always stored in the red-black tree, but whether it is in the doubly linked list depends on whether the node has received an event). In addition, the rdy member in the epitem structure is used to mark whether the node exists in the doubly linked list, so when it is removed from the doubly linked list, the rdy member is set to 0.

2.4 The kernel adds nodes to the doubly linked list

• The epoll_wait function actually fetches nodes from the doubly linked list, so who inserted these nodes into the doubly linked list? Although the operating system (kernel). When does the operating system insert nodes into the doubly linked list? Obviously, when an event arrives on a certain TCP connection (these events are registered in the red-black tree by the programmer using epoll_ctl), the operating system will insert nodes into the doubly linked list.
  When writing code, what events will cause the operating system to insert nodes into the doubly linked list? Generally, there are 4 situations.
  (1) When the client completes the three-way handshake, the operating system will insert a node into the doubly linked list. At this time, the server often calls the accept function to remove the connection from the completed connection queue.
  (2) When the client closes the connection, the operating system will insert a node into the doubly linked list, and the server will also call close to close the corresponding socket.
  (3) When the client sends data, the operating system will insert a node into the doubly linked list, then the server can call send or recv to receive the data.
  (4) When the data can be sent, the operating system will insert a node into the doubly linked list, then the server can call send or write to send data to the client. (If the client receives data slowly and the server sends data blocks, then the server has to wait for the client to receive a batch of data before sending the next batch, so as to prevent the client from "choking to death")

3. ET (edge ​​trigger), LT (level trigger) mode in-depth

• LT：是水平触发，属于低速模式，如果该事件没有处理完，就会被一直触发。
• ET：边缘触发，属于高速模式，该事件通知只会出现1次。

It is generally believed that the efficiency of ET is very high, but the programming of ET is very difficult.

The client sample code is convenient for the demonstration of the following running results:

#include<stdio.h>
#include<stdlib.h>
#include<string.h>
#include<unistd.h>
#include<arpa/inet.h>
#include<sys/socket.h>

#define BUF_SIZE 1024
void error_handling(char *message);

int main(int argc, char *argv[])
{
    
    
	int sock;
	char message[BUF_SIZE];
	int str_len;
	struct sockaddr_in serv_adr;
	FILE *readfp;
	FILE *writefp;
	if(argc != 3){
    
    
		printf("Usage: %s <IP> <port>\n", argv[0]);
		exit(1);
	}
	sock = socket(PF_INET, SOCK_STREAM, 0);
	if(sock == -1)
		error_handling("socket() error");
	
	memset(&serv_adr, 0, sizeof(serv_adr));
	serv_adr.sin_family = AF_INET;
	serv_adr.sin_addr.s_addr = inet_addr(argv[1]);
	serv_adr.sin_port = htons(atoi(argv[2]));

	if(connect(sock, (struct sockaddr *)&serv_adr, sizeof(serv_adr)) == -1)
		error_handling("connect() error!");
	else
		puts("Connected...........");
	
	readfp = fdopen(sock, "r");
	writefp = fdopen(sock, "w");
	while(1)
	{
    
    
		fputs("Input message(Q to quit): ", stdout);
		fgets(message, BUF_SIZE, stdin);
		if(!strcmp(message, "q\n") || !strcmp(message, "Q\n"))
			break;
		
		fputs(message, writefp);
		fflush(writefp);
		fgets(message, BUF_SIZE, readfp);
		printf("Message from server: %s", message);
	}
	fclose(writefp);
	fclose(readfp);
	return 0;
}

void error_handling(char *message)
{
    
    
	fputs(message, stderr);
	fputc('\n', stderr);
	exit(1);
}

3.1 epoll instance - level trigger

调用read函数后，输入缓冲区中仍有数据需要读取。而且会因此注册新的事件并从epoll_wait函数返回时将循环输出“return epoll_wait”字符串。
If the event is not processed, it will be triggered all the time.

code show as below:

#include<stdio.h>
#include<stdlib.h>
#include<string.h>
#include<unistd.h>
#include<arpa/inet.h>
#include<sys/socket.h>
#include<sys/epoll.h>
#include<fcntl.h>
#include<errno.h>
//为了验证边缘触发的工作方式，将缓冲区设置为4字节
#define BUF_SIZE 4 
#define EPOLL_SIZE 50
void error_handling(char *buf);

int main(int argc, char *argv[])
{
    
    
	int serv_sock, clnt_sock;
	struct sockaddr_in serv_adr, clnt_adr;
	socklen_t adr_sz;
	int str_len, i;
	char buf[BUF_SIZE];
	
	struct epoll_event *ep_events;
	struct epoll_event event;
	int epfd, event_cnt;
	
	if(argc != 2){
    
    
		printf("Usage : %s <port> \n", argv[0]);
		exit(1);
	}
	
	serv_sock = socket(PF_INET, SOCK_STREAM, 0);
	memset(&serv_adr, 0, sizeof(serv_adr));
	serv_adr.sin_family = AF_INET;
	serv_adr.sin_addr.s_addr = htonl(INADDR_ANY);
	serv_adr.sin_port = htons(atoi(argv[1]));
	
	if(bind(serv_sock, (struct sockaddr*)&serv_adr, sizeof(serv_adr)) == -1)
		error_handling("bind() error");
	if(listen(serv_sock, 5) == -1)
		error_handling("listen() error");
	
	// --- epoll_create: 创建保存epoll文件描述符的空间，成功时返回epoll文件描述符，失败时返回-1
	//参数：size（int size） 表示文件描述符保存空间的大小
	epfd = epoll_create(EPOLL_SIZE);
	// 表示保存发生事件的文件描述符集合的结构体地址
	ep_events = malloc(sizeof(struct epoll_event) * EPOLL_SIZE);
	
	//发生需要读取数据情况（事件）时
	event.events = EPOLLIN; 
	event.data.fd = serv_sock;
	// --- epoll_ctl: 向epoll空间注册并销毁文件描述符，成功时返回0，失败时返回1
	//参数：epfd（int epfd） 表示用于注册监视对象的epoll例程的文件描述符
	//参数：op（int op）表示用于指定监视对象的添加、删除或更改等操作
	//参数：fd（int fd）表示需要注册的监视对象文件描述符
	//参数：event（epoll_event event）表示监视对象的事件类型
	epoll_ctl(epfd, EPOLL_CTL_ADD, serv_sock, &event);

	while(1)
	{
    
    
		// --- epoll_wait: 等待文件描述符发生变化，成功时会返回文件描述符数，失败时返回-1 ----
		//参数：epfd（int epfd） 表示事件发生监视范围的epoll例程的文件描述符
		//参数：epevents（epoll_event events）表示指向缓冲区保存发生事件的文件描述符集合的结构体地址
		//参数：EPOLL_SIZE（int maxevents）表示第二个参数中可以保存的最大事件数
		//参数：-1（int timeout）表示以1/1000秒为单位的等待时间，传递-1时，一直等待直到发生的事件
		event_cnt = epoll_wait(epfd, ep_events, EPOLL_SIZE, -1);
		if(event_cnt == -1)
		{
    
    
			puts("epoll_wait() error");
			break;
		}
		
		//为观察事件发生数而添加的输出字符串的语句
		puts("return epoll_wait");
		for(i = 0; i < event_cnt; i++)
		{
    
    
			if(ep_events[i].data.fd == serv_sock)
			{
    
    
				adr_sz = sizeof(clnt_adr);
				clnt_sock = accept(serv_sock, (struct sockaddr *)&clnt_adr, &adr_sz);
				event.events = EPOLLIN          ;
				event.data.fd = clnt_sock;
				epoll_ctl(epfd, EPOLL_CTL_ADD, clnt_sock, &event);
				printf("connect client: %d \n", clnt_sock);
			}else{
    
    
				str_len = read(ep_events[i].data.fd, buf, BUF_SIZE);
				if(str_len == 0) //close request!
				{
    
    
					epoll_ctl(epfd, EPOLL_CTL_DEL, ep_events[i].data.fd, NULL);
					close(ep_events[i].data.fd);
					printf("closed client: %d \n", ep_events[i].data.fd);
					break;
					//read函数返回-1且errno值为EAGAIN时，意味着读取了输入缓冲区中的全部数据
				}else{
    
    
					write(ep_events[i].data.fd, buf, str_len); //echo!
				}
			}
		}
	}
	close(serv_sock);
	close(epfd);
	return 0;
}

void error_handling(char *buf)
{
    
    
	fputs(buf, stderr);
	fputc('\n', stderr);
	exit(1);
}

Figure 3_1 epoll level trigger running results

It can be seen from the running results that whenever the client data is received, the event will be registered, and therefore the epoll_wait function will be called multiple times.

3.2 epoll instance - edge trigger

边缘触发方式中，接收数据时仅注册1次该事件函数。
Because of this feature, once an input-related event occurs, all data in the input buffer should be read. Hence the need to verify that the input buffer is not empty.

read函数返回-1时，变量errno中的值为EAGAIN时，说明没有数据可读。
That being the case, why do you need to turn the socket into non-blocking mode? In the edge-triggered mode, the read & write functions that work in a blocking manner may cause a long pause on the server side. Therefore, non-blocking read & write functions must be used in the edge trigger method, which may cause a long pause on the server side. Therefore, non-blocking read & write functions must be used in the edge trigger mode.

Two things you must know about edge triggering:
(1) Verify the cause of the error through the errno variable.
(2) In order to complete non-blocking (Non-blocking) I/O, change the socket properties.

code show as below:

#include<stdio.h>
#include<stdlib.h>
#include<string.h>
#include<unistd.h>
#include<arpa/inet.h>
#include<sys/socket.h>
#include<sys/epoll.h>
#include<fcntl.h>
#include<errno.h>
//为了验证边缘触发的工作方式，将缓冲区设置为4字节
#define BUF_SIZE 4 
#define EPOLL_SIZE 50
void setnonblockingmode(int fd);
void error_handling(char *buf);

int main(int argc, char *argv[])
{
    
    
	int serv_sock, clnt_sock;
	struct sockaddr_in serv_adr, clnt_adr;
	socklen_t adr_sz;
	int str_len, i;
	char buf[BUF_SIZE];
	struct epoll_event *ep_events;
	struct epoll_event event;
	int epfd, event_cnt;
	if(argc != 2){
    
    
		printf("Usage : %s <port> \n", argv[0]);
		exit(1);
	}
	serv_sock = socket(PF_INET, SOCK_STREAM, 0);
	memset(&serv_adr, 0, sizeof(serv_adr));
	serv_adr.sin_family = AF_INET;
	serv_adr.sin_addr.s_addr = htonl(INADDR_ANY);
	serv_adr.sin_port = htons(atoi(argv[1]));
	if(bind(serv_sock, (struct sockaddr*)&serv_adr, sizeof(serv_adr)) == -1)
		error_handling("bind() error");
	if(listen(serv_sock, 5) == -1)
		error_handling("listen() error");
	
	// --- epoll_create: 创建保存epoll文件描述符的空间，成功时返回epoll文件描述符，失败时返回-1
	//参数：size（int size） 表示文件描述符保存空间的大小
	epfd = epoll_create(EPOLL_SIZE);
	
	// 表示保存发生事件的文件描述符集合的结构体地址
	ep_events = malloc(sizeof(struct epoll_event) * EPOLL_SIZE);

	//设置非阻塞模式
	setnonblockingmode(serv_sock);
	
	//发生需要读取数据情况（事件）时
	event.events = EPOLLIN; 
	event.data.fd = serv_sock;
	
	// --- epoll_ctl: 向epoll空间注册并销毁文件描述符，成功时返回0，失败时返回1
	//参数：epfd（int epfd） 表示用于注册监视对象的epoll例程的文件描述符
	//参数：op（int op）表示用于指定监视对象的添加、删除或更改等操作
	//参数：fd（int fd）表示需要注册的监视对象文件描述符
	//参数：event（epoll_event event）表示监视对象的事件类型
	epoll_ctl(epfd, EPOLL_CTL_ADD, serv_sock, &event);

	while(1)
	{
    
    
		// --- epoll_wait: 等待文件描述符发生变化，成功时会返回文件描述符数，失败时返回-1 ----
		//参数：epfd（int epfd） 表示事件发生监视范围的epoll例程的文件描述符
		//参数：epevents（epoll_event events）表示指向缓冲区保存发生事件的文件描述符集合的结构体地址
		//参数：EPOLL_SIZE（int maxevents）表示第二个参数中可以保存的最大事件数
		//参数：-1（int timeout）表示以1/1000秒为单位的等待时间，传递-1时，一直等待直到发生的事件
		event_cnt = epoll_wait(epfd, ep_events, EPOLL_SIZE, -1);
		if(event_cnt == -1)
		{
    
    
			puts("epoll_wait() error");
			break;
		}
		
		//为观察事件发生数而添加的输出字符串的语句
		puts("return epoll_wait");
		for(i = 0; i < event_cnt; i++)
		{
    
    
			if(ep_events[i].data.fd == serv_sock)
			{
    
    
				adr_sz = sizeof(clnt_adr);
				clnt_sock = accept(serv_sock, (struct sockaddr *)&clnt_adr, &adr_sz);
				//将accept函数创建的套接字改为非阻塞模式
				setnonblockingmode(clnt_sock);
				//向EPOLLIN添加EPOLLET标志，将套接字事件注册方式改为边缘触发
				event.events = EPOLLIN|EPOLLET;
				
				event.data.fd = clnt_sock;
				epoll_ctl(epfd, EPOLL_CTL_ADD, clnt_sock, &event);
				printf("connect client: %d \n", clnt_sock);
			}else{
    
    
				while(1)
				{
    
    
					//边缘触发方式中，发生事件时需要读取输入缓冲区中的所有数据，因此需要循环调用read函数
					str_len = read(ep_events[i].data.fd, buf, BUF_SIZE);
					if(str_len == 0)
					{
    
    
						epoll_ctl(epfd, EPOLL_CTL_DEL, ep_events[i].data.fd, NULL);
						close(ep_events[i].data.fd);
						printf("closed client: %d \n", ep_events[i].data.fd);
						break;
						//read函数返回-1且errno值为EAGAIN时，意味着读取了输入缓冲区中的全部数据
					}else if(str_len < 0){
    
    
						if(errno == EAGAIN)
							break;
					}else{
    
    
						write(ep_events[i].data.fd, buf, str_len);
					}
				}
			}
		}
	}
	close(serv_sock);
	close(epfd);
	return 0;
}
//设置非阻塞模式
void setnonblockingmode(int fd)
{
    
    
	// --- int fcntl(int filedes, int cmd, . . .); ---
	//fcntl成功时返回cmd参数相关值，失败时返回-1
	//参数：int filedes 表示更改目标文件描述符
	//参数：int cmd 表示函数调用的目的

	//从上述声明中可以看到，fcntl具有可变参数形式。如果向第二个参数传递F_GETFL，可以获得第一个参数所指的文件描述符属性（int 型）。
	//反之，如果传递F_SETFL,可以更改文件描述符属性，

	//将文件（套接字）改为非阻塞模式
	//获取之前设置的属性信息
	int flag = fcntl(fd, F_GETFL, 0);
	//添加非阻塞O_NONBLOCK标志
	fcntl(fd, F_SETFL, flag|O_NONBLOCK); 
	//此时，调用read & write 函数时，无论是否存在数据，都会形成非阻塞文件（套接字）
}

void error_handling(char *buf)
{
    
    
	fputs(buf, stderr);
	fputc('\n', stderr);
	exit(1);
}

Run the edge trigger mode of the client and server, and the results are as follows:

Figure 3_2 epoll edge trigger running results

The above running results need to pay attention to the number of messages sent by the client and the number of epoll_wait function calls on the server. The client sends data 5 times from the connection request to the disconnection, and the server generates 5 events accordingly.

3.3 Which is better, level trigger or edge trigger?

Advantages of edge triggering:
可以分离接受数据和处理数据的时间点。

Figure 3_3 Understanding edge triggering

The operation process is as follows:

• The server receives data from clients A, B, and C respectively.
• The server reassembles the received data in the order of A, B, and C.
• The combined data will be sent to any host.

In order to complete the process, if the program can be run according to the following process, the implementation of the server is not difficult.

• The client connects to the server in the order of A, B, and C, and sends data to the server in sequence.
• Clients that need to receive data should connect to the server and wait before clients A, B, and C.

However, in reality, the following situations may frequently occur. In other words, the following situations are more realistic.

• Clients C and B are sending data to the server, but A is not yet connected to the server.
• Clients A, B, and C send data out of order.
• The server has received the data, but the target client to receive the data has not yet connected to the server.

Therefore, even if the input buffer receives data (register corresponding events), the server side can also decide the time point to read and process the data, which brings great flexibility to the implementation of the server side.

Is there no way to distinguish between data reception and processing in conditional triggering?
It is not impossible, but in the case of input buffer receiving data, if it is not read (delayed processing), the corresponding event will be generated every time the epoll_wait function is called. And the events will also accumulate, can the server side bear it? This is impossible in reality (this province is not reasonable, so it is something you don't want to do at all).