Article directory
1. Basic concepts of five IO models
First of all, IO is waiting + data copying. Remember that we implemented the read/recv interface for the server before. We said at that time that if there is data in this interface, then read/recv will return after the copy is completed. If there is no data, then It will block waiting, and the purpose of waiting is to wait for the resources to be ready and copy the data once there are resources.
1. Blocking I/O
When the process calls recvfrom to make a system call to read the data in the kernel, if the data is not ready, recv will directly block and wait for the data to be ready. Once the data is ready, the data will be copied from the kernel to the user space, and the copy will return success instructions.
2. Non-blocking IO
When the process calls recvfrom to make a system call to read the data in the kernel, if the data is not ready, then recv will return an error code, because it is non-blocking, so it takes a while to ask whether the kernel data is ready, and it can be used at other times Let this process do some other things, such as printing logs or something, just ask at intervals whether the data is ready, if not ready, send an error code, copy the data from the kernel to the user space and return a success indication when it is ready .
3. Signal driven IO
When the data is not ready, we can let the process capture sigaction, and once it is ready, we capture this signal to copy the data. You can still do other things when you are not ready.
4. IO multiplexing
Note: The principle of multi-way transfer is to wait for multiple file descriptors at a time, so the previous interface cannot be used, and the new select system call must be used. And select, poll, and epoll are all IO intermediate steps. Once they are successful, they can still call recvfrom to copy data. And recvfrom will no longer be blocked during multi-way transfer, as long as the select waits for success, recvfrom will directly copy the data.
5. Asynchronous I/O
The principle of asynchronous IO is to let the system wait for the data. When there is data, it will be copied to the buffer I designated. I am only responsible for getting the data in the buffer. This is equivalent to the fact that the previous IOs all focus on how to cook, while asynchronous IO only focuses on how to eat, and does not care about how the meal comes.
2. Important concepts of IO
1. Synchronous communication vs asynchronous communication (synchronous communication/asynchronous communication)
2. Blocking vs non-blocking
3. Code demonstration of non-blocking IO
First, let's get to know the fcntl interface:
#include <unistd.h>
#include <fcntl.h>
int fcntl(int fd, int cmd, ... /* arg */ );void setNonBlock(int fd)
{
int n = fcntl(fd,F_GETFL);
if (n<0)
{
std::cerr<<"fcntl: "<<strerror(errno)<<std::endl;
return;
}
fcntl(fd, F_SETFL, n | O_NONBLOCK);
}F_GETFD gets the status flag of the file descriptor, and the function returns -1 to indicate that the setting fails
F_SETFL can set the status flag of the file descriptor, such as setting read or set write, as shown in the following figure:
The last O_NONBLOCK is the option set to non-blocking. After we have written the function that sets the file descriptor to be non-blocking, first demonstrate the result of the blocking state, and then demonstrate the result of the non-blocking state:
int main()
{
char buffer[1024];
while (true)
{
printf(">>> ");
fflush(stdout);
ssize_t s = read(0,buffer,sizeof(buffer)-1);
if (s>0)
{
buffer[s] = 0;
std::cout<<"echo# "<<buffer<<std::endl;
}
else if (s == 0)
{
std::cout<<"read end"<<std::endl;
break;
}
else
{
}
}
return 0;
}
We read directly in an infinite loop, first create a buffer, and then read the data in the 0 standard input file descriptor to our own buffer, when the read is successful, put \0 at the end of the file and print it . Seeing the result, we can know that this is a blocking read, because once we do not print the content to the standard input file descriptor, it will be blocked in the read function. Let's take a look at the non-blocking result:
int main()
{
char buffer[1024];
setNonBlock(0);
while (true)
{
printf(">>> ");
fflush(stdout);
ssize_t s = read(0,buffer,sizeof(buffer)-1);
if (s>0)
{
buffer[s] = 0;
std::cout<<"echo# "<<buffer<<std::endl;
}
else if (s == 0)
{
std::cout<<"read end"<<std::endl;
break;
}
else
{
}
sleep(1);
}
return 0;
}First, set descriptor 0 as non-blocking, because it is too fast to print >>> during the test. In order to demonstrate, we sleep for 1 second:
It can be seen that even if we do not input the function to the file descriptor number 0, it will still be executed in an endless loop. The reflected result is that if we do not input, the >>> symbol will continue to be printed, and the >>> symbol will also be printed during our input process. , which is non-blocking! We no longer need to block to the read interface to wait for data input.
Remember at the beginning we said that non-blocking IO will return an error code if the data is not ready. We know that the read interface returns -1 if it fails to read. Let’s verify it below:
From the results, we can see that the error code -1 is indeed returned, and we will print out the cause of the error below:
It can be seen that although -1 is returned, it is not an error but that the resource is not ready. In fact, the operating system has prepared some error codes for us:
For example, EAGAIN means that the resource is not ready, and EINTR means that the data has not been read and is interrupted, which is not an error:
So in fact, the correct way of writing is the above, because in this way we can know that there is no error at this time but the resources are not ready.
The above is the code demonstration of non-blocking IO. Next, we introduce the select interface of IO multi-channel transfer.
4. IO multiplexing select
int select(int nfds, fd_set *readfds, fd_set *writefds,
fd_set *exceptfds, struct timeval *timeout);Because select can wait for multiple file descriptors at a time, and the essence of file descriptors is an array subscript, so the first parameter is the largest file descriptor to be checked + 1, +1 is because the bottom layer will traverse the file descriptor.
readfds, writefds, and exceptfds are respectively the set of read file descriptors, the set of write file descriptors, and the set of exception file descriptors.
timeout is a structure, which is used to set the waiting time of select. Let's see what timeval is:
What does that mean. For example, if we pass timeout={0,0} to indicate a non-blocking monitoring file descriptor, timeout=nullptr indicates a blocking monitoring file descriptor, timeout={5,0} indicates a blocking monitoring file descriptor within 5s, exceeding 5 seconds non-blocking return, and subsequent timeout{5,0} becomes {0,0}.
For example, in the blocking reading code demonstrated at the beginning, if you set 5 and 0 with select, only >>> will be displayed within 5 seconds, waiting for user input, and an error code will be returned after 5 seconds. After returning, it will continue to print like non-blocking >>>
If the return value of select is greater than 0, it means that several file descriptors are ready. If the return value is equal to 0, it means a timeout return. If the return value is less than 0, it means that there is an error in the select call.
In fact, our fd_set type is a bitmap. When a file descriptor read event is ready, the position of the file descriptor in the bitmap is set to 1. Write events and exception events are the same, as shown in the following figure:
When we call it, the user tells the kernel which file descriptors need to be cared about.
When the function is executed, which bit in the bitmap is set to 1 represents which file descriptor event is ready.
The following is the interface for manipulating bitmaps:
void FD_CLR(int fd, fd_set *set); // 用来清除描述词组set中相关fd 的位
int FD_ISSET(int fd, fd_set *set); // 用来测试描述词组set中相关fd 的位是否为真
void FD_SET(int fd, fd_set *set); // 用来设置描述词组set中相关fd的位
void FD_ZERO(fd_set *set); // 用来清除描述词组set的全部位After knowing the above interface, we will implement the select server:
First, we encapsulate the four steps of creating a socket, binding, listening, and obtaining a new connection into functions:
enum
{
SOCKET_ERR = 2,
USE_ERR,
BIND_ERR,
LISTEN_ERR
};
const uint16_t gport = 8080;
class Sock
{
private:
public:
const static int gbacklog = 32;
static int createSock()
{
// 1.创建文件套接字对象
int sock = socket(AF_INET, SOCK_STREAM, 0);
if (sock == -1)
{
logMessage(FATAL, "create socket error");
exit(SOCKET_ERR);
}
logMessage(NORMAL, "socket success %d",sock);
int opt = 1;
setsockopt(sock,SOL_SOCKET,SO_REUSEADDR | SO_REUSEPORT,&opt,sizeof(opt));
return sock;
}
static void Bind(int sock,uint16_t port)
{
struct sockaddr_in local;
bzero(&local, sizeof(local));
local.sin_family = AF_INET;
local.sin_port = htons(port);
local.sin_addr.s_addr = INADDR_ANY; // INADDR_ANY绑定任意地址IP
if (bind(sock, (struct sockaddr *)&local, sizeof(local)) < 0)
{
logMessage(FATAL, "bind socket error");
exit(BIND_ERR);
}
logMessage(NORMAL, "bind socket success");
}
static void Listen(int sock)
{
if (listen(sock, gbacklog) < 0)
{
logMessage(FATAL, "listen socket error");
exit(LISTEN_ERR);
}
logMessage(NORMAL, "listen socket success");
}
static int Accept(int listensock,std::string *clientip,uint16_t& clientport)
{
struct sockaddr_in peer;
socklen_t len = sizeof(peer);
// sock是和client通信的fd
int sock = accept(listensock, (struct sockaddr *)&peer, &len);
// accept失败也无所谓，继续让accept去获取新链接
if (sock < 0)
{
logMessage(ERROR, "accept error,next");
}
else
{
logMessage(NORMAL, "accept a new link success");
*clientip = inet_ntoa(peer.sin_addr);
clientport = ntohs(peer.sin_port);
}
return sock;
}
};We have talked about all the function interfaces of the server above when implementing the TCP server. If you don’t understand it, you can go and see:
namespace select_ns
{
static const int defaultport = 8080;
class SelectServer
{
private:
int _port;
int _listensock;
public:
SelectServer(int port = defaultport)
:_port(port)
,_listensock(-1)
{
}
void initServer()
{
_listensock = Sock::createSock();
Sock::Bind(_listensock,_port);
Sock::Listen(_listensock);
}
void start()
{
for (;;)
{
fd_set rfds;
FD_ZERO(&rfds);
// 把lsock添加到读文件描述符集中
FD_SET(_listensock, &rfds);
struct timeval timeout = {1, 0};
int n = select(_listensock+1,&rfds,nullptr,nullptr,&timeout);
switch (n)
{
case 0:
logMessage(NORMAL,"time out.....");
break;
case -1:
logMessage(WARNING,"select error,code: %d,err string: %s",errno,strerror(errno));
break;
default:
//说明有事件就绪了
logMessage(NORMAL,"get a new link");
break;
}
sleep(1);
/* std::string clientip;
uint16_t clientport = 0;
int sock = Sock::Accept(_listensock,&clientip,clientport);
if (sock<0)
{
continue;
}
//开始进行服务器的处理逻辑 */
}
}
~SelectServer()
{
if (_listensock != -1)
{
close(_listensock);
}
}
};
}The above is a framework for us to implement a select server using the encapsulated interface. In the server startup function, we need to create a file descriptor bitmap read object, and then initialize it to 0 with FD_ZERO. Note that we only demonstrate how to read as a demonstration. In fact, writing and exceptions are the same as reading. Set the blocking read within 1 second, we divide the return value of select into 3 cases, 1. select timeout 2. select error 3. detect that there is an event ready, once there is an event ready, we will print it. Now let's run it:
When there is no connection, it must print time_out, and when there is a connection, it will print get new:
So why are so many get a new printed? This is because we did not process the file descriptor obtained by this select, so the value of the file descriptor in the bitmap is always 1, so it keeps printing. Next, we write a processing function to deal with the ready file descriptor:
void HanderEvent(fd_set &rfds)
{
if (FD_ISSET(_listensock, &rfds))
{
//listensock必然就绪
std::string clientip;
uint16_t clientport = 0;
int sock = Sock::Accept(_listensock, &clientip, clientport);
if (sock < 0)
{
return;
}
logMessage(NORMAL,"accept success [%s:%d]",clientip.c_str(),clientport);
}
}When the listen file descriptor read event is ready, we get a new connection and print the client's ip and port number:
Now let's run it:
We can see that once the new connection is obtained successfully, this time it will not repeatedly print and obtain the new connection as before, but continue to wait for the new connection. This is because the read event is ready and we have processed this event.
After dealing with this point, let's think about how to let select handle other file descriptors. For example, we now need to use the file descriptor returned by accept to communicate. When the client sends data, our server can display this data. In fact, it usually takes Using select requires programmers to maintain an array that stores all legal fds. Let's implement it below:
First we create an array and a default value, which is used to initialize all elements in the array:
fd_num represents the maximum number of file descriptors that can be stored in this array, and this number is as large as fd_set*8.
void initServer()
{
_listensock = Sock::createSock();
if (_listensock == -1)
{
logMessage(NORMAL,"createSock error");
return;
}
Sock::Bind(_listensock,_port);
Sock::Listen(_listensock);
fdarray = new int[fd_num];
for (int i = 0;i<fd_num;i++)
{
fdarray[i] = defaultfd;
}
fdarray[0] = _listensock;
}When we initialize, we need to open space and initialize all values to -1 (why is it a negative number? Because the file descriptor starts from 0, if it is a positive number, it may affect a certain file descriptor), since the space is opened, it is not necessary. It is to be destructed, so there is a destructor. Of course, our listening socket must be managed in an array during initialization:
~SelectServer()
{
if (_listensock != -1)
{
close(_listensock);
}
if (fdarray)
{
delete[] fdarray;
fdarray = nullptr;
}
}In the start function, when an event is ready, we execute the hander function, because we are now using an array to manage all file descriptors, so the hander method becomes as follows:
void HanderEvent(fd_set &rfds)
{
if (FD_ISSET(_listensock, &rfds))
{
//listensock必然就绪
std::string clientip;
uint16_t clientport = 0;
int sock = Sock::Accept(_listensock, &clientip, clientport);
if (sock < 0)
{
return;
}
logMessage(NORMAL,"accept success [%s:%d]",clientip.c_str(),clientport);
// 开始进行服务器的处理逻辑
// 将accept返回的文件描述符放到自己管理的数组中，本质就是放到了select管理的位图中
int i = 0;
for (i = 0; i < fd_num; i++)
{
if (fdarray[i] != defaultfd)
{
continue;
}
else
{
break;
}
}
if (i == fd_num)
{
logMessage(WARNING, "server is full ,please wait");
close(sock);
}
else
{
fdarray[i] = sock;
}
print();
}
}The first step is to judge whether the read event of the listening socket is ready, and only when it is ready, we will do the following operations. When we get the communication socket returned by the new connection, we need to put this socket into the bitmap in select to manage, so first traverse the array to find the legal file descriptor (if the default value is used, then the is illegal), after finding a legal descriptor, we first judge whether the end of the array has been reached in the process of traversing. If it reaches the end of the array, it means that all the file descriptors in the array are legal. At this time, we need to record that the log array is full and we need to wait . If the end of the array is not reached, just place the new file descriptor returned by accept just at the specified position of the array. Later we added a print function for the convenience of seeing the result:
void print()
{
std::cout << "fd list: ";
for (int i = 0; i < fd_num; i++)
{
if (fdarray[i] != defaultfd)
{
std::cout << fdarray[i] << " ";
}
}
std::cout << std::endl;
}This function will only print legal file descriptors. Of course, there is one place that has not been modified. Remember the first parameter of select. This parameter is the largest file descriptor + 1, so the modification is as follows:
First assume that the largest file descriptor is the listening socket, then traverse the array, find a legal file descriptor, add the legal file descriptor to the read bitmap, and then determine whether it is greater than maxfd. Let's see the effect:
It can be seen that there is no problem. Every time a new connection arrives, we will add the file descriptor of the new connection to the array, and finally the array will put these legal file descriptors into select for monitoring.
Let's continue to modify the code to make our select server support normal IO communication:
Because we need to handle all file descriptors, we encapsulate the accept part in the hander function, and then implement corresponding functions according to different file descriptors:
void HanderEvent(fd_set &rfds)
{
for (int i = 0;i<fd_num;i++)
{
//过滤掉非法的文件描述符
if (fdarray[i] == defaultfd)
continue;
//如果是listensock事件就绪，就去监听新连接获取文件描述符，如果不是listensock事件，那么就是普通的IO事件就绪了
if (FD_ISSET(fdarray[i], &rfds) && fdarray[i] == _listensock)
{
Accepter(_listensock);
}
else if (FD_ISSET(fdarray[i], &rfds))
{
Recver(fdarray[i],i);
}
else
{
}
}
}When the listensock file descriptor is ready, we call the accept function to handle listening to new connections. If the ordinary file descriptor is ready, then execute the read data function:
void Accepter(int listensock)
{
// listensock必然就绪
std::string clientip;
uint16_t clientport = 0;
int sock = Sock::Accept(listensock, &clientip, clientport);
if (sock < 0)
{
return;
}
logMessage(NORMAL, "accept success [%s:%d]", clientip.c_str(), clientport);
// 开始进行服务器的处理逻辑
// 将accept返回的文件描述符放到自己管理的数组中，本质就是放到了select管理的位图中
int i = 0;
for (i = 0; i < fd_num; i++)
{
if (fdarray[i] != defaultfd)
{
continue;
}
else
{
break;
}
}
if (i == fd_num)
{
logMessage(WARNING, "server is full ,please wait");
close(sock);
}
else
{
fdarray[i] = sock;
}
print();
}accept is the code in the hander function just now, we will directly explain how to process the data:
void Recver(int sock,int pos)
{
//注意：这样的读取有问题，由于没有定协议所以我们不能确定是否能读取一个完整的报文，并且还有序列化反序列化操作...
//由于我们只做演示所以不再定协议，在TCP服务器定制的协议大家可以看看
char buffer[1024];
ssize_t s = recv(sock,buffer,sizeof(buffer)-1,0);
if (s>0)
{
buffer[s] = 0;
logMessage(NORMAL,"client# %s",buffer);
}
else if (s == 0)
{
//对方关闭文件描述符，我们也要关闭并且下次不让select关心这个文件描述符了
close(sock);
fdarray[pos] = defaultfd;
logMessage(NORMAL,"client quit");
}
else
{
//读取失败，关闭文件描述符
close(sock);
fdarray[pos] = defaultfd;
logMessage(ERROR,"client quit: %s",strerror(errno));
}
//2.处理 request
std::string response = func(buffer);
//3.返回response
write(sock,response.c_str(),response.size());
}First of all, there is a problem with our data processing, because under normal circumstances, a custom protocol is required to ensure that a complete message is read, and serialization and deserialization are required. Today, we will not do these tasks for demonstration purposes. After reading the data, we perform an echo print on the server. If the reading fails or the client closes the file descriptor, our server should also close the corresponding file descriptor at this time, and we want to describe the file in the array The character is set to an illegal state, so that the next select will no longer monitor this file descriptor. After getting the message from the client, we directly call a func function to process it. func is a newly added function for demonstration, as shown in the figure below:
It can be seen that we simply return the client's message, but in fact the function of this function is to process the client request and send a response to the client after serialization and deserialization.
After getting the response, we directly write it back to the file descriptor used for communication. In this way, we have modified the code, let's run it and see:
It can be seen that the program runs without problems.
Summarize
Let's summarize the characteristics of the select server:
1. There is an upper limit on the file descriptors that select can wait for at the same time. Changing the kernel can only increase the upper limit a little, but it cannot completely solve it.
2. The select server must use a third-party array to maintain legal file descriptors.
3. Most of the parameters of select are input and output types. Before calling select, all file descriptors must be reset. After calling, we also need to check and update all file descriptors, which brings the cost of traversal.
4. Why is the first parameter of select the largest file descriptor +1? This is because file descriptors also need to be traversed at the kernel level
5. Select uses bitmaps, so it will frequently switch from kernel mode to user mode, and then switch from user mode to kernel mode to copy data back and forth, which has the problem of copy cost.
So how to solve the above problem? The following poll and epoll servers will solve this problem.