foreword
In this article, we will introduce the development of the server, and explore how to develop a high-performance and high-concurrency server program from multiple aspects. It should be noted that in general large-scale servers, the complexity lies in its business, not in the basic framework of its code engineering.
Large servers generally consist of multiple services, may support CDN, or support so-called "distributed", etc. This article will not introduce these things, because no matter how complex the structure of the server is, it is composed of a single server. So the focus of this article is to discuss the structure of a single service program, and the structure here also refers to the network communication layer structure of a single server. If you can really understand what I am talking about, then start on this basic structure. Any business is possible, and this structure can also be extended into complex groups of multiple servers, such as "distributed" services.
Although the code example in this article is based on C++, it is also suitable for Java (I am also a Java developer), and the principle is the same, except that Java may be wrapped with a virtual machine on the basis of the basic operating system network communication API. It's just a layer of interfaces (Java may even provide some ready-made APIs based on some common network communication framework ideas, such as NIO). In view of this, this article does not discuss those big, empty, general technical terms, but talks about real coding schemes or methods for optimizing existing coding that can guide readers in practical work. In addition, the technology discussed here involves both windows and linux platforms.
The so-called high performance means that the server can smoothly handle the connection of each client and respond to the client's request with low latency as much as possible; the so-called high concurrency means not only that the server can support multiple client connections at the same time, but also that these clients will be connected during the connection period. There is constant data communication with the server. There are often various network libraries on the Internet that claim that a single service can support millions or even tens of millions of concurrency at the same time. Then I actually checked it and found that it can only support many connections at the same time.
If a server can simply accept n connections (n may be very large), but it can't process the data to and from these connections in an orderly manner, it doesn't make any sense. Application doesn't make any sense.
This article will introduce two aspects, one is the basic network communication components in the server; the other is how to use these basic communication components to integrate into a complete and efficient server framework. Note: The client in the following content of this article is a relative concept, referring to the terminal connected to the service program currently discussed, so the client here may be either a client program in our traditional sense, or a connection to the service. other server programs.
1. Network communication components
According to the ideas introduced above, we start with the network communication components of the service program.
issues that need resolving
Since it is a server program that will definitely involve network communication, what problems should the network communication module of the server program solve? At present, there are many network communication frameworks on the network, such as libevent, boost asio, ACE, but the common technical means of network communication are similar, at least the following problems must be solved:
How can I detect a new client connection?
How to accept client connections?
How to detect whether the client has data sent?
How to receive the data sent by the client?
How to detect connection anomalies? How to deal with the abnormal connection?
How to send data to the client?
How to close the connection after sending data to the client?
People with a little bit of network knowledge can answer several of the above questions, such as receiving the client connection with the accept function of the socket API, receiving the client data with the recv function, sending data to the client with the send function, and detecting the client Whether there is a new connection and whether the client has new data can use the select, poll, epoll and other socket APIs of IO multiplexing technology (IO multiplexing). This is indeed the case. These basic socket APIs constitute the foundation of server network communication. No matter how cleverly designed the network communication framework is, they are all built on the basis of these basic socket APIs. But how to organize these basic socket APIs skillfully is the crux of the problem. We say that the server is very efficient and supports high concurrency. In fact, it is only a technical implementation method. In any case, from the perspective of software development, it is nothing more than a program. Therefore, as long as the program can meet the requirements of "minimize waiting or not waiting" "This principle is efficient, that is to say, high efficiency is not "too busy and busy to die", but that everyone can be idle, but if there is work to be done, everyone should try to do it together instead of part of it. Do things in turn 123456789, and the other part is idle and doing nothing. It may be a bit abstract, so let's take some examples to illustrate.
E.g:
By default, if the recv function has no data, the thread will block there;
By default, the send function, if the tcp window is not large enough, the data will be blocked there;
When the connect function connects to the other end by default, it will also block there;
Or to send a piece of data to the peer, you need to wait for the peer to answer, if the peer does not respond, the current thread is blocked here.
None of the above is the way of thinking about efficient server development, because none of the above examples meet the principle of "minimize waiting", so why do you have to wait? Is there a way that these processes do not need to wait, preferably not only do they not need to wait, but also notify me when these things are done. This way I can do some other things during these cpu time slices that would have been used for waiting. Yes, that is, the IO Multiplexing technology (IO multiplexing technology) that we will discuss below.
Comparison of Several IO Multiplexing Mechanisms
At present, the windows system supports select, WSAAsyncSelect, WSAEventSelect, and completion port (IOCP), and the linux system supports select, poll, and epoll. Here we do not specifically introduce the usage of each specific function, let's discuss a little deeper things. The API functions listed above can be divided into two levels:
Level 1: select and poll
Level 2: WSAAsyncSelect, WSAEventSelect, Completion Port (IOCP), epoll
Why is it so divided? Let’s introduce the first level first. In essence, the select and poll functions actively query whether there are events on the socket handle (may be one or more) within a certain period of time, such as readable events, writable events or error events. That is to say, we still need to take the initiative to do these detections at regular intervals. If some events are detected during this time, our time will not be wasted, but what if there are no events during this time? We can only do useless work. To put it bluntly, it is still a waste of time, because if a server has multiple connections, in the case of limited CPU time slices, we spend a certain amount of time to detect some socket connections, but find what they are. There are no events, and we have something to deal with during this time, so why do we take the time to do this test? Wouldn't it be nice to spend this time doing what we need to do? So for the server program, in order to be efficient, we should try to avoid spending time actively querying whether some sockets have events, but tell us to deal with these sockets when there are events. This is what the functions of level 2 do. They are actually equivalent to changing the active query to see if there is an event. When there is an event, the system will tell us that we will deal with it at this time, that is, "good steel is used on the blade". . It's just that the functions of level 2 notify us in different ways. For example, WSAAsyncSelect uses the event mechanism of the windows window message queue to notify us of the window procedure function we set. IOCP uses GetQueuedCompletionStatus to return the correct status, and epoll is the return of the epoll_wait function. That's it.
For example, the connect function connects to the other end. If the socket used to connect to the socket is non-blocking, then although the connection cannot be completed immediately, connect will return immediately without waiting. After the connection is completed, WSAAsyncSelect will return FD_CONNECTan event to tell us that the connection is successful, and epoll will The EPOLLOUT event is generated, and we can also know that the connection is complete. Even when the socket has data to read, WSAAsyncSelect generates the FD_READ event, epoll generates the EPOLLIN event, and so on. So with the above discussion, we can get the correct posture for network communication to detect readable, writable or error events. This is my second principle here: Minimize the time spent doing useless work. This may not show any advantages when the server resources are sufficient, but if there are a large number of tasks to be processed, this becomes a performance bottleneck.
Correct posture for detecting network events
According to the above introduction, first, in order to avoid meaningless waiting time, second, instead of actively querying the events of each socket, we adopt the strategy of waiting for the operating system to notify us of the status of the event. Our sockets must be set to non-blocking. On this basis, we return to the seven questions mentioned in column (1):
How can I detect a new client connection?
How to accept client connections?
By default, the accept function will block there. If epoll detects an EPOLLIN event on the listening socket, or WSAAsyncSelect detects an FD_ACCEPT event, it means that a new connection arrives at this time. At this time, the accept function is called, and it will not block. Of course, you should also set the new socket to be non-blocking. This way we can send and receive data on the new socket.
How to detect whether the client has data sent?
How to receive the data sent by the client?
Similarly, we should also collect data when there is a readable event on the socket, so that we do not have to wait when calling the recv or read function. As for how much data is collected at one time? We can decide according to our own needs, even you can repeat recv or read in a loop. For sockets in non-blocking mode, if there is no data, recv or read will return immediately, and the error code EWOULDBLOCK will indicate that there is no data. . Example:
bool CIUSocket::Recv()
{
int nRet = 0;
while(true)
{
char buff[512];
nRet = ::recv(m_hSocket, buff, 512, 0);
if(nRet == SOCKET_ERROR) //Close the Socket as soon as an error occurs
{
if (::WSAGetLastError() == WSAEWOULDBLOCK)
break;
else
return false;
}
else if(nRet < 1)
return false;
m_strRecvBuf.append(buff, nRet);
::Sleep(1);
}
return true;
}
5. How to detect connection abnormality? How to deal with the abnormal connection?
Similarly, when we receive an abnormal event such as EPOLLERR or close event FD_CLOSE, we know that there is an exception, and our handling of the exception is generally to close the corresponding socket. In addition, if the send/recv or read/write function operates on a socket, if it returns 0, it means that the peer has closed the socket, and this connection does not need to exist at this time, we can also close the corresponding socket.
6. How to send data to the client?
This is also a common network communication interview question. In a certain year, Tencent's background development position was asked such a question. Sending data to the client is a little more troublesome than receiving data, and it also requires some skills. First of all, we cannot register the detection data writable event from the beginning like registering the detection data readable event, because if the detection is writable, in general, as long as the peer receives data normally, our sockets are writable. If we Setting to monitor writable events will cause writable events to be triggered frequently, but we do not necessarily have data to send at this time. So the correct way is: if there is data to be sent, try to send it first. If it can't be sent or only the part is sent, we need to cache the rest, and then set to detect writable events on the socket, and then download When the next writable event occurs, continue to send, if it still cannot be sent completely, continue to set and listen to the writable event, and so on, until all data is sent. Once all the data has been sent, we need to remove listening for writable events to avoid useless writable event notifications. I don’t know if you noticed that if only part of the data is sent out at a time, the rest of the data should be temporarily stored. At this time, we need a buffer to store this part of the data. We call this buffer the “send buffer” . The sending buffer not only stores the data that has not been sent this time, but also stores the new data that needs to be sent from the upper layer during the sending process. To ensure order, the new data should be appended to the current remaining data, starting from the head of the send buffer when sending. That is to say, the first comes first, and the later is sent later.
7. How to close the connection after sending data to the client?
This problem is more difficult to deal with, because the "sent" here is not necessarily the real transmission. Even if we call the send or write function successfully, it will only successfully write data into the protocol stack of the operating system. As for whether it can be sent out , It is difficult to judge when it is sent, and it is even more difficult to judge whether the other party has received it. Therefore, we can only simply think that send or write returns the number of bytes of the data we sent, and we think that "the data has been sent". Then call the socket API such as close to close the connection. Of course, you can also call the shutdown function to achieve a so-called "half shutdown". On the topic of closing the connection, we will open a separate topic to discuss it.
Passively close connections and actively close connections
In practical applications, passively closing the connection is because we detected an abnormal event of the connection, such as EPOLLERR, or the peer closes the connection, and send or recv returns 0. At this time, this connection has no necessary meaning, and we are forced to Close the connection.
And to actively close the connection, we actively call close/closesocket to close the connection. For example, the client sends us illegal data, such as some attempted data packets for network attacks. At this time, for security reasons, we close the socket connection.
send buffer and receive buffer
The send buffer has been introduced above, and the meaning of its existence has been explained. The same is true for the receive buffer. When the data is received, we can unpack it directly, but this is not good. Reason 1: Unless some commonly known protocol formats are agreed, such as the http protocol, most server business protocols are It is different, that is to say, the interpretation of the data format in a data packet should be a matter of the business layer, and should be decoupled from the network communication layer. In order to make the network layer more general, we cannot know what the upper layer protocol looks like, because different The protocol formats are different, they are related to the specific business. Reason 2: Even if we know the protocol format, we unpack and process the corresponding business at the network layer. If the business processing is time-consuming, such as requiring complex operations, or connecting to the database for account password verification, then our network thread will need to A lot of time is spent on these tasks, so other network events may not be processed in a timely manner. In view of the above two points, we really need a receive buffer, put the received data into the buffer, and use a special business thread or business logic to take out the data from the receive buffer, and unpack and process the business.
Having said so much, how big should the send buffer and receive buffer be? This is a commonplace question, because we often encounter this problem: the pre-allocated memory is too small and not enough, and if it is too large, it may be wasteful. How to do it? The answer is to design a buffer that can grow dynamically like string and vector, allocate on demand, and expand if it is not enough.
It should be noted that the send buffer and receive buffer mentioned here are one for each socket connection. This is our most common design solution.
protocol design
Except for some common protocols, such as http and ftp protocols, most server protocols are formulated according to the business. The protocol is designed, and the format of the data packet is set according to the protocol. We know that the tcp/ip protocol is streaming data, so streaming data is like streaming water, and there is no obvious boundary between packets. For example, end A sends three consecutive data packets to end B, each data packet is 50 bytes, end B may receive 10 bytes first, then 140 bytes; or receive 20 bytes first Bytes, 20 bytes are received, 110 bytes are received; 150 bytes may also be received at one time. These 150 bytes can be received by B in any combination and number of bytes. So when we discuss the design of the protocol, the first problem is how to define the boundaries of the packets, that is, how the receiver knows the size of each packet data. There are currently three commonly used methods:
Fixed size, this method assumes that the size of each packet is a fixed number of bytes. For example, the size of each packet discussed above is 50 bytes, and every 50 bytes received by the receiving end is regarded as a packet.
Specify the package terminator, for example, end with a \r\n (line feed and carriage return), so that as long as the peer receives such a terminator, it can consider that a package has been received, and the next data is the next package. content.
Specify the size of the packet. This method combines the above two methods. Generally, the packet header is a fixed size. There is a field in the packet header to specify the packet
body or the entire size of the packet. After receiving the data, the peer parses the fields in the packet header to obtain the packet body. Or the size of the entire package, and then define the boundaries of the data according to this size.
The second problem to be discussed in the protocol is that when designing the protocol, it is necessary to make unpacking as easy as possible, that is to say, the format field of the protocol should be as clear as possible.
The third issue to be discussed in the protocol is that a single data packet assembled according to the protocol should be as small as possible. Note that this refers to a single data packet, which has the following advantages: First, for some mobile devices, their data processing capabilities and limited bandwidth capacity, small data can not only speed up the processing speed, but also save a lot of traffic costs; second, if a single data packet is small enough, it can greatly reduce the bandwidth pressure on the server side that frequently communicates with the network, The system it is on can also use less memory. Just think: If a stock server is a stock server, if the data packet of a stock is 100 bytes or 1000 bytes, what is the difference between 10,000 stocks?
协议要讨论的第四个问题是，对于数值类型，我们应该显式地指定数值的长度，比如long型，在32位机器上是32位4个字节，但是如果在64位机器上，就变成了64位8个字节了。这样同样是一个long型，发送方和接收方可能因为机器位数的不同会用不同的长度去解码。所以建议最好，在涉及到跨平台使用的协议最好显式地指定协议中整型字段的长度，比如int32、int64等等。下面是一个协议的接口的例子，当然java程序员应该很熟悉这样的接口：
class BinaryReadStream
{
private:
const char* const ptr;
const size_t len;
const char* cur;
BinaryReadStream(const BinaryReadStream&);
BinaryReadStream& operator=(const BinaryReadStream&);
public:
BinaryReadStream(const char* ptr, size_t len);
virtual const char* GetData() const;
virtual size_t GetSize() const;
bool IsEmpty() const;
bool ReadString(string* str, size_t maxlen, size_t& outlen);
bool ReadCString(char* str, size_t strlen, size_t& len);
bool ReadCCString(const char** str, size_t maxlen, size_t& outlen);
bool ReadInt32(int32_t& i);
bool ReadInt64(int64_t& i);
bool ReadShort(short& i);
bool ReadChar(char& c);
size_t ReadAll(char* szBuffer, size_t iLen) const;
bool IsEnd() const;
const char* GetCurrent() const{ return cur; }
public:
bool ReadLength(size_t & len);
bool ReadLengthWithoutOffset(size_t &headlen, size_t & outlen);
};
class BinaryWriteStream
{
public:
BinaryWriteStream(string* data);
virtual const char* GetData() const;
virtual size_t GetSize() const;
bool WriteCString(const char* str, size_t len);
bool WriteString(const string& str);
bool WriteDouble(double value, bool isNULL = false);
bool WriteInt64(int64_t value, bool isNULL = false);
bool WriteInt32(int32_t i, bool isNULL = false);
bool WriteShort(short i, bool isNULL = false);
bool WriteChar(char c, bool isNULL = false);
size_t GetCurrentPos() const{ return m_data->length(); }
void Flush();
void Clear();
private:
string* m_data;
};
其中BinaryWriteStream是编码协议的类，BinaryReadStream是解码协议的类。可以按下面这种方式来编码和解码。
编码：
std::string outbuf; BinaryWriteStream writeStream(&outbuf); writeStream.WriteInt32(msg_type_register); writeStream.WriteInt32(m_seq); writeStream.WriteString(retData); writeStream.Flush();
解码：
BinaryReadStream readStream(strMsg.c_str(), strMsg.length());
int32_t cmd;
if (!readStream.ReadInt32(cmd))
{
return false;
}
//int seq;
if (!readStream.ReadInt32(m_seq))
{
return false;
}
std::string data;
size_t datalength;
if (!readStream.ReadString(&data, 0, datalength))
{
return false;
}
二、服务器程序结构的组织
上面的六个标题，我们讨论了很多具体的细节问题，现在是时候讨论将这些细节组织起来了。根据我的个人经验，目前主流的思想是one thread one loop+reactor模式（也有proactor模式）的策略。通俗点说就是一个线程一个循环，即在一个线程的函数里面不断地循环依次做一些事情，这些事情包括检测网络事件、解包数据产生业务逻辑。我们先从最简单地来说，设定一些线程在一个循环里面做网络通信相关的事情，伪码如下：
while(退出标志)
{
//IO复用技术检测socket可读事件、出错事件
//（如果有数据要发送，则也检测可写事件）
//如果有可读事件，对于侦听socket则接收新连接；
//对于普通socket则收取该socket上的数据，收取的数据存入对应的接收缓冲区，如果出错则关闭连接；
//如果有数据要发送，有可写事件，则发送数据
//如果有出错事件，关闭该连接
}
另外设定一些线程去处理接收到的数据，并解包处理业务逻辑，这些线程可以认为是业务线程了，伪码如下：
//从接收缓冲区中取出数据解包，分解成不同的业务来处理上面的结构是目前最通用的服务器逻辑结构，但是能不能再简化一下或者说再综合一下呢？我们试试，你想过这样的问题没有：假如现在的机器有两个cpu（准确的来说应该是两个核），我们的网络线程数量是2个，业务逻辑线程也是2个，这样可能存在的情况就是：业务线程运行的时候，网络线程并没有运行，它们必须等待，如果是这样的话，干嘛要多建两个线程呢？除了程序结构上可能稍微清楚一点，对程序性能没有任何实质性提高，而且白白浪费cpu时间片在线程上下文切换上。所以，我们可以将网络线程与业务逻辑线程合并，合并后的伪码看起来是这样子的：
while(退出标志)
{
//IO复用技术检测socket可读事件、出错事件
//（如果有数据要发送，则也检测可写事件）
//如果有可读事件，对于侦听socket则接收新连接；
//对于普通socket则收取该socket上的数据，收取的数据存入对应的接收缓冲区，如果出错则关闭连接；
//如果有数据要发送，有可写事件，则发送数据
//如果有出错事件，关闭该连接
//从接收缓冲区中取出数据解包，分解成不同的业务来处理
}
你没看错，其实就是简单的合并，合并之后和不仅可以达到原来合并前的效果，而且在没有网络IO事件的时候，可以及时处理我们想处理的一些业务逻辑，并且减少了不必要的线程上下文切换时间。
我们再更进一步，甚至我们可以在这个while循环增加其它的一些任务的处理，比如程序的逻辑任务队列、定时器事件等等，伪码如下：
while(退出标志)
{
//定时器事件处理
//IO复用技术检测socket可读事件、出错事件
//（如果有数据要发送，则也检测可写事件）
//如果有可读事件，对于侦听socket则接收新连接；
//对于普通socket则收取该socket上的数据，收取的数据存入对应的接收缓冲区，如果出错则关闭连接；
//如果有数据要发送，有可写事件，则发送数据
//如果有出错事件，关闭该连接
//从接收缓冲区中取出数据解包，分解成不同的业务来处理
//程序自定义任务1
//程序自定义任务2
}
注意：之所以将定时器事件的处理放在网络IO事件的检测之前，是因为避免定时器事件过期时间太长。假如放在后面的话，可能前面的处理耗费了一点时间，等到处理定时器事件时，时间间隔已经过去了不少时间。虽然这样处理，也没法保证定时器事件百分百精确，但是能尽量保证。当然linux系统下提供eventfd这样的定时器对象，所有的定时器对象就能像处理socket这样的fd一样统一成处理。这也是网络库libevent的思想很像，libevent将socket、定时器、信号封装成统一的对象进行处理。
说了这么多理论性的东西，我们来一款流行的开源网络库muduo来说明吧（作者：陈硕），原库是基于boost的，我改成了C++11的版本，并修改了一些bug，在此感谢原作者陈硕。
上文介绍的核心线程函数的while循环位于eventloop.cpp中：