This article mainly introduces the programming of TCP concurrent network, focusing on the epoll implementation of io multiplexing
1. TCP/IP network communication process
To complete a complete TCP/IP network communication process, a series of functions need to be used. These functions include bind, listen, accept, recv/send, etc. Here's how they work together:
- Create a socket (socket): use the socket function to create a socket, specify the protocol family and socket type.
- Bind address (bind): Bind the local address to the socket so that the client can access the server through this address.
- Listen for connection requests (listen): Set the socket to the listening state and specify the maximum number of waiting connections (backlog).
- Accept connection request (accept): When a client initiates a connection request, use the accept function to create a new socket for communication with the client.
- Read and write data (recv/send): Use the newly created socket for data transmission, including reading data from the client and sending data to the client.
- Close the connection (close): After the communication is over, you need to use the close function to close the socket to release resources.
There are two processing methods for the request in step 4: one thread one request and the epoll method.
2. epoll implementation of io multiplexing
epoll is a mechanism for high-performance I/O multiplexing in the Linux operating system. It can monitor multiple file descriptors at the same time, and when an event such as reading or writing occurs on any one of the file descriptors, it will trigger the corresponding callback function for processing.
The flow of the TCP server program based on epoll is as follows:
-
Create a listening socket, use the socket() function to create a socket and set related parameters (such as address reuse, etc.).
-
Bind the listening socket to the local IP address and port number, and use the bind() function to bind the socket to the specified address.
-
Start listening for connection requests, use the listen() function to mark the socket as a passive listening state, and set the maximum number of connections that can be processed at the same time.
-
Create an epoll instance, use the epoll_create() function to create an epoll instance, and set the event type to be monitored.
-
Add the listening socket to the epoll instance, and use the epoll_ctl() function to add the file descriptors and corresponding events to be monitored to the epoll instance. The event type is generally EPOLLIN for readable events or EPOLLERR for error events.
-
Enter the main loop to process client requests, use epoll_wait() to wait for the kernel to notify the ready event, and get the list of ready file descriptors. Then traverse the list of file descriptors and process accordingly according to the event type corresponding to each file descriptor. If it is a new connection request, call accept() to receive the connection and add it to the epoll listening queue; otherwise, directly read data or close the connection.
-
Close the listening socket and the connected client socket, clean up resources and exit the program.
In short, in the TCP server program implemented by epoll, by using the epoll instance, multiple client connection requests can be processed at the same time, and when new data arrives, the program can be notified in time for corresponding processing.
In addition, epoll also provides two working modes: ET (edge trigger) and LT (level trigger).
- Horizontal trigger mode
In horizontal trigger mode, if the events on the file descriptor have not been processed, epoll will continue to notify the application that there are still events to be processed on the file descriptor. In this case, if the application does not respond in time and read the data, epoll will keep notifying the application that there is data to read on the file descriptor. - Edge-triggered mode
In edge-triggered mode, epoll notifies the application whenever a new event occurs on the file descriptor (such as data read or connection established). However, after the notification, if the application does not immediately respond and read all the data, epoll will not notify again that there is new data to read on that file descriptor.
Generally speaking, the edge trigger mode is more efficient than the level trigger mode, and can avoid the problem of high CPU usage caused by repeated monitoring. However, care needs to be taken when using edge-triggered mode to read all data in a timely manner and ensure that each event is properly processed.
And, compared with select, although both are I/O multiplexing mechanisms under Linux, they have some important differences:
-
The limit on the number of monitored file descriptors is different: in Linux, the maximum number of file descriptors supported by the select function is 1024 by default, while epoll does not have this limit and can monitor thousands of file descriptors.
-
The copying method of the file descriptor set is different: when using select, all the file descriptors to be monitored need to be copied from the user space to the kernel space for each call, and the result needs to be copied back from the kernel space to the user space after the result is returned. This will bring a large performance overhead. When using epoll, you only need to add the file descriptor to be monitored to a kernel event table to complete the registration, and when a ready event occurs, directly notify the application for processing.
-
The processing method for non-blocking sockets is different: when using select, for non-blocking sockets, we need to manually set it to non-blocking mode and poll whether the read and write operations are ready. However, epoll implements the edge-triggered mode by setting the EPOLLET flag, and for a non-blocking socket, it only needs to wait for it to return the EAGAIN error code to know that it is already in a non-blocking state.
Generally speaking, compared with select, epoll has the advantages of being more efficient, more flexible, and easier to expand, and has better performance and scalability when dealing with a large number of concurrent connections.
#include <stdio.h>
#include <time.h>
#include <string.h>
#include <stdlib.h>
// #include <winsock2.h>
// #include <mswsock.h>
// #include <windows.h>
// #include <sys/types.h>
// #include <unistd.h>
// #include <fcntl.h>
#include <netinet/tcp.h>
#include <arpa/inet.h>
#include <pthread.h>
#include <errno.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/epoll.h>
#define BUFFER_LENGTH 1024
#define EPOLL_SIZE 1024
void *client_routine(void *arg){
int clientfd=*(int *)arg;
while (1){
char buffer[BUFFER_LENGTH]={
0};
int len=recv(clientfd,buffer,BUFFER_LENGTH,0);
if (len < 0){
//非阻塞状态下读到空数据
close(clientfd);
break;
}
else if(len == 0) {
//断开连接
close(clientfd);
break;
}
else{
printf("Recv: %s, %d btye(s)\n",buffer,len);
}
}
}
int main(int argc,char *argv[]){
if (argc < 2) {
printf("Param Error \n");
return -1;
}
int port=atoi(argv[1]);//atoi将一个字符串转换为对应的整数值
int sockfd=socket(AF_INET,SOCK_STREAM,0);
struct sockaddr_in addr;
memset(&addr,0,sizeof(struct sockaddr_in));
addr.sin_family=AF_INET;
addr.sin_port=htons(port);
addr.sin_addr.s_addr=INADDR_ANY;
if (bind(sockfd,(struct sockaddr*)&addr,sizeof(struct sockaddr_in))<0){
perror("bind");
return -2;
}
if(listen(sockfd,5)<0){
perror("listen");
return -3;
}
#if 0
// 一请求一线程
while (1){
struct sockaddr_in client_addr;
memset(&client_addr,0,sizeof(struct sockaddr_in));
socklen_t client_len =sizeof(client_addr);
/*调用 accept() 函数后，它会一直阻塞等待直到有新的客户端连接请求到达为止。
当有新的连接请求到达时，它会返回一个新产生的套接字文件描述符，并且将该连接对应的客户端地址信息存储在 addr 指向的结构体中*/
int clientfd=accept(sockfd,(struct sockaddr *)&client_addr,&client_len);
pthread_t thread_id;
pthread_create(&thread_id,NULL,client_routine,&clientfd);
}
#else
/*使用epoll的基本流程如下：
1,创建一个epoll实例，可以通过调用 epoll_create() 函数来创建。
2,向 epoll 实例中添加需要监控的文件描述符及其事件类型，可以通过调用 epoll_ctl() 函数进行操作。
3,调用 epoll_wait() 函数等待监控对象上发生事件，并处理活跃的文件描述符及其事件类型。
4,处理完活跃文件描述符的相关操作后，返回到第三步继续等待新的事件发生。
*/
int epfd=epoll_create(1);
struct epoll_event events[EPOLL_SIZE] = {
0}; //创建一个结构体数组 events 用于存储 epoll_wait() 返回的事件列表。
struct epoll_event ev;
ev.events=EPOLLIN; //创建一个新的 epoll_event 结构体 ev 并设置其关注的事件类型为 EPOLLIN （表示等待读事件）
ev.data.fd = sockfd;
epoll_ctl(epfd,EPOLL_CTL_ADD,sockfd,&ev); //使用 epoll_ctl() 函数将 sockfd 文件描述符加入到 epfd 实例中，并关联上面创建的 ev 结构体。
while (1){
int nready=epoll_wait(epfd,events,EPOLL_SIZE,5); //
if (nready == -1) continue; //表示5秒内，没有事件，继续监听
int i=0;
for (i=0;i<nready;i++){
/*判断当前事件所对应的文件描述符是否为监听套接字 sockfd。如果是，则说明有新的客户端连接请求到来了，
需要通过 accept() 函数获取新产生的客户端连接并添加到 epoll 实例中；
否则，说明是已经建立好连接的客户端发送了数据，需要通过 recv() 函数接收数据并进行相应处理。*/
if(events[i].data.fd == sockfd){
/*当有新的连接请求到来时（即 sockfd 上有 EPOLLIN 事件），使用 accept() 函数接受连接，
并将其加入 epoll 实例中关注该套接字上是否有输入事件。*/
struct sockaddr_in client_addr;
memset(&client_addr,0,sizeof(struct sockaddr_in));
socklen_t client_len =sizeof(client_addr);
int clientfd=accept(sockfd,(struct sockaddr *)&client_addr,&client_len);
ev.events=EPOLLIN | EPOLLET; //EPOLLET 则表示将 I/O 事件设置为边缘触发模式。
ev.data.fd=clientfd;
epoll_ctl(epfd,EPOLL_CTL_ADD,clientfd,&ev);
}
else{
//当某个客户端套接字上出现可读事件时（即该文件描述符在 events 中对应的元素有 EPOLLIN 标志），则调用 recv() 函数从该套接字中读取数据
int clientfd=events[i].data.fd;
char buffer[BUFFER_LENGTH]={
0};
int len=recv(clientfd,buffer,BUFFER_LENGTH,0);
if (len < 0){
//出现了异常情况或者非阻塞状态下没有更多数据可读
//关闭该套接字并将其从 epoll 实例中删除
close(clientfd);
ev.events=EPOLLIN;
ev.data.fd=clientfd;
epoll_ctl(epfd,EPOLL_CTL_DEL,clientfd,&ev); //从 epoll 实例中删除 clientfd 对应的文件描述符，并且停止监听该套接字上的事件。
}
else if(len == 0) {
//对方已经断开连接
//关闭该套接字并将其从 epoll 实例中删除
close(clientfd);
ev.events=EPOLLIN;
ev.data.fd=clientfd;
epoll_ctl(epfd,EPOLL_CTL_DEL,clientfd,&ev);
}
else{
printf("Recv: %s, %d btye(s)\n",buffer,len);
}
}
}
}
#endif
return 0;
}