The content of the "Preface" article is about the agreement. The general content is to re-understand the agreement and the customization of the simple agreement. The purpose is to help understand the agreement. Let's start to explain!
"Belonging column" network programming
"Author" Mr. Maple Leaf (fy)
"Motto" Cultivate myself on the way forward
"Mr. Maple Leaf is a little literary" "Sentence Sharing"
I have dealt with me for a long time, and I would rather be me.
——Princes of Fenghuo Opera "Sword Come"
Table of contents
1.2 Serialization and deserialization
Second, the network version of the calculator
3. Serialization and deserialization
1. Renegotiate the agreement
An agreement is an "agreement" that both parties need to abide by.
In a computer network, a protocol is used to specify a series of rules and conventions for data transmission, communication, and interaction. The network protocol defines details such as the way computers communicate, data format, transmission rate, error detection and correction, etc., so as to ensure that the devices in the network can understand each other and exchange data correctly.
1.1 Structured data
The api interface of socket, when reading and writing data, is sent and received in the form of "string", what if we want to transmit some "structured data"?
What is structured data? ?
For example, in QQ chat, you can’t simply send a message, but also pack the avatar url, time nickname, etc. into a message, and send the data of this message to the other party together. The packaged message is a structured data
1.2 Serialization and deserialization
Serialization and deserialization:
- Serialization is the process of converting a data structure or object into a stream of bytes. During serialization, an object's state information is converted into a sequence of bytes, which can be stored in a file or transmitted over a network
- Deserialization is the process of converting a stream of bytes or other stored form back into a data structure or object. During deserialization, the sequence of bytes is reconverted into an object's state information so that the object can be recreated and its data used
Purpose of serialization and deserialization
- Data persistence: Through serialization, the state of the object can be saved to a file or database, so that the state of the object can be restored from it when the program is restarted or reloaded
- Data transmission: Through serialization, objects can be converted into byte streams for delivery in network transmission
- Cross-platform and cross-language interaction: Through serialization, objects can be converted into a common byte stream format, enabling data exchange and sharing between different platforms and different programming languages. Whether it is Java, Python, C++ or other programming languages, as long as serialization and deserialization can be performed, cross-platform and cross-language data interaction can be realized
When sending a message to the network, the message first needs to be serialized and then sent. After the message is sent to the other party through the protocol stack, the party receiving the message also needs to deserialize the message in order to use the message normally.
Second, the network version of the calculator
The following implements a network version of the calculator, the main purpose is to feel what a protocol is, and to understand the process of simple business protocol customization, serialization and deserialization , the focus is not on the calculator
2.1 Server
The code directly uses the socket socket TCP multi-threaded version, which has been explained before, so I won’t explain it anymore
The steps to initialize the server initServer function are roughly as follows:
- Call the socket function to create a socket.
- Call the bind function to bind a port number for the server
- Call the listen function to set the socket to the listening state
The steps to start the server start function are roughly as follows:
- Call the accept function to get a new link
- Serve clients
In addition to providing services for the client, we need to rewrite, other codes are all before
tcpServer.hpp
Note: There are too many codes, only a small part is posted
static const int gbacklog = 5;
typedef std::function<void(const Request &req, Response &resp)> func_t;
// 错误类型枚举
enum
{
UAGE_ERR = 1,
SOCKET_ERR,
BIND_ERR,
LISTEN_ERR
};
// 业务处理 -- 解耦
void handlerEntery(int sockefd, func_t func)
{
}
class tcpServer; // 声明
class ThreadDate
{
public:
ThreadDate(int sockfd, func_t func)
: _sockfd(sockfd), _func(func)
{}
public:
int _sockfd;
func_t _func;
};
class calServer
{
public:
calServer(const uint16_t &port)
: _listensock(-1), _port(port)
{}
// 初始化服务器
void initServer()
{}
// 启动服务器
void start(func_t func)
{
for (;;)
{
// 5. 为sockfd提供服务，即为客户端提供服务
// 多线程版
pthread_t tid;
ThreadDate *td = new ThreadDate(sockfd, func);
pthread_create(&tid, nullptr, threadRoutine, td);
}
}
static void *threadRoutine(void *args)
{
pthread_detach(pthread_self()); // 线程分离
ThreadDate *td = static_cast<ThreadDate *>(args);
handlerEntery(td->_sockfd, td->_func); // 业务处理
close(td->_sockfd); // 必须关闭，由新线程关闭
delete td;
return nullptr;
}
~calServer()
{}
private:
int _listensock; // listen套接字，不是用来数据通信的，是用来监听链接到来
uint16_t _port; // 端口号
};The only function we need to rewrite is the business processing handlerEntery
// 业务处理 -- 解耦
void handlerEntery(int sockefd, func_t func)
{}Transmission Control (TCP)
Suppose the host on the left is the server and the host on the right is the client
The server sends data to the client or vice versa:
- The send function (write, etc.) has its own application layer buffer in its own application layer. Calling the send function actually copies the data to the buffer of the transport layer. Whether the data is sent to the network is determined by the TCP protocol itself, so The TCP protocol is called the Transmission Control Protocol, keyword: Transmission Control
- The receiving function (read, etc.) also has its own application layer buffer in its own application layer. Calling the receiving function actually copies the data in the transport layer buffer to its own application layer buffer.
So the sending and receiving functions we call are essentially copy functions
So one party is sending, and the other is also sending, and the two parties will not affect at all, because they have paired buffers, one is responsible for sending and one is responsible for receiving, so TCP is full duplex
So TCP will have problems when reading data (byte stream-oriented)
For example, the other party sends multiple messages at once, and these messages are accumulated in the receiving buffer of TCP. When the application layer reads the messages, how can it judge that it is a complete message? Or what to do if only half of the message is read? Or how to deal with reading half a message? Or read two messages? ?
Therefore, it is necessary to clarify the size of the message and the boundary of the message
Solution:
- Fixed length of message
- Distinguish with special symbols
- self-describing
2.2 Custom protocol
For a customized protocol, it is necessary to ensure that both communication parties (client and server) can abide by the agreement.
We can design a simple protocol. The data can be divided into request data and response data. Therefore, we need to make an agreement on the request data and response data respectively. Two structures can be used to encapsulate the data request and response.
The completed business processing function handlerEntery
This function is roughly divided into five steps:
- Read and receive a complete message, unpack the message
- Deserialize the message
- Really carry out business processing, calculate and process data
- Serialize the result of data processing
- Add a header to the message, and finally send the message of the response result
// 业务处理 -- 解耦
void handlerEntery(int sockfd, func_t func)
{
std::string inbuffer; // 读取的报文全部放在inbuffer里面
while (true)
{
// 1.读取 -- 收取到一个完整报文
std::string req_text; // 用于接收一个完整报文
if (!recvPackage(sockfd, inbuffer, &req_text)) // 接收一个完整报文
return;
std::cout << "带报头的报文：" << req_text << std::endl;
std::string req_str; // 获取解包之后的结果
deLength(req_text, &req_str); // 解包
std::cout << "解包后的报文：" << req_str << std::endl;
// 2. 请求request -- 反序列化
Request req;
if (!req.deserialize(req_str)) // 请求反序列化
return;
// 3. 业务逻辑 -- 处理数据
Response resp; // 拿取计算结果
func(req, resp); // 计算，回调函数
// 4.响应Response -- 序列化
std::string resp_str; // 拿取响应序列化结果
resp.serialize(&resp_str); // 序列化
std::cout << "计算完成，响应序列化结果：" << resp_str << std::endl;
// 5. 发送响应的结果
std::string send_str = enLength(resp_str); // 构建成为一个完整的报文
std::cout << "构建成为一个完整的报文：" << send_str << std::endl;
send(sockfd, send_str.c_str(), send_str.size(), 0); // 发送也有bug，暂时不用理会
}
}Make sure you can read a complete message
// 接收一个报文
bool recvPackage(int sockfd, std::string &inbuffer, std::string *text)
{
char buffer[1024];
while (true)
{
ssize_t n = recv(sockfd, buffer, sizeof(buffer) - 1, 0);
if (n > 0)
{
buffer[n] = 0;
inbuffer += buffer;
auto pos = inbuffer.find(LINE__SEP);
if (pos == std::string::npos)
continue; // 报文不完整，继续读取
std::string text_len_str = inbuffer.substr(0, pos);
int text_len = std::stoi(text_len_str); // 报文有效载荷大小
// "content_len" + "\r\n" + "exitcode result" + "\r\n"
int total_len = text_len_str.size() + 2 * LINE__SEP_LEN + text_len; // 一个完整报文的长度
if (inbuffer.size() < total_len)
continue; // 报文不完整，继续读取
std::cout << "处理前的inbuffer: " << inbuffer << std::endl;
// 走到这里，至少有一个完整的报文
*text = inbuffer.substr(0, total_len); // 拿走报文
inbuffer.erase(0, total_len); // 删除已拿走的报文
std::cout << "处理后的inbuffer: " << inbuffer << std::endl;
return true;
}
else
{
return false;
}
}
}custom protocol
- The member variables in the request structure need to include two operands and corresponding operators
- The response structure needs to include a calculation result. In addition, the response structure also needs to include a status field, indicating the status of this calculation
Note: The process of serialization and deserialization is not part of the agreement
class Request
{
public:
Request() : x(0), y(0), op(0){};
Request(int x_, int y_, char op_) : x(x_), y(y_), op(op_)
{}
// 请求序列化 -- 暂时自己写
bool serialize(std::string *out)
{
}
// 请求反序列化 -- 暂时自己写
bool deserialize(std::string &in)
{
}
public:
int x;
int y;
char op;
};
class Response
{
public:
Response() : exitcode(0), result(0)
{}
Response(int exitcode_, int result_) : exitcode(exitcode_), result(result_)
{}
// 响应结果序列化 -- 暂时自己写
bool serialize(std::string *out)
{
}
// 响应结果反序列化 -- 暂时自己写
bool deserialize(std::string &in)
{
}
public:
int exitcode;
int result;
};Note: After the agreement is customized, it must be seen by the client and the server at the same time, so that they can abide by this agreement
Function introduction
send function, used for TCP to send data
The function prototype of the send function is as follows:
ssize_t send(int sockfd, const void *buf, size_t len, int flags);Parameter Description:
- sockfd: A specific file descriptor, indicating that data is written to the socket corresponding to the file descriptor.
- buf: the data to be sent.
- len: the number of bytes of data to be sent.
- flags: The way of sending, generally set to 0, which means blocking sending.
Return value description:
- The number of bytes actually written is returned if the write is successful, and -1 is returned if the write fails, and the error code will be set at the same time.
This function is consistent with the write function
The recv function is used for TCP to receive data
The function prototype of the ecv function is as follows:
ssize_t recv(int sockfd, void *buf, size_t len, int flags);
Parameter Description:
- sockfd: A specific file descriptor that reads data from the file descriptor.
- buf: The storage location of the data, indicating that the read data is stored in this location.
- len: the number of data, indicating the number of bytes of data read from the file descriptor.
- flags: The way of reading, generally set to 0, which means blocking reading.
Return value description:
- If the return value is greater than 0, it means the number of bytes actually read this time.
- If the return value is equal to 0, it means that the peer has closed the connection.
- If the return value is less than 0, it means that an error was encountered while reading.
The function of this function is consistent with read
2.3 Client
The code of the client is similar to the previous one, just modify the read and write data of the start function, and abide by our customized protocol
// 启动客户端
void start()
{
// 客户端需要发起链接，链接服务端
struct sockaddr_in server;
memset(&server, 0, sizeof(server));
server.sin_family = AF_INET;
server.sin_port = htons(_serverport); // 主机转网络序列
server.sin_addr.s_addr = inet_addr(_serverip.c_str()); // 1.std::string类型转int类型 2.把int类型转换成网络字节序 （这两个工作inet_addr已完成）
if (connect(_sockfd, (struct sockaddr *)&server, sizeof(server)) != 0)
{
std::cerr << "socket connect error" << std::endl;
}
else // 连接成功
{
std::string line;
std::string inbuffer;
while (true)
{
// 发送请求
std::cout << "Input Cal>> ";
std::getline(std::cin, line);
Request req = parseLine(line); // 对输入的字符串做解析
std::string content; // 获取序列化的结果
req.serialize(&content); // 对输入的内容进行序列化
std::string send_str = enLength(content); // 添加报头
send(_sockfd, send_str.c_str(), send_str.size(), 0); // 发送，bug,不理会
std::string package, text;
if (!recvPackage(_sockfd, inbuffer, &package)) continue; // 获取一个完整的报文
if (!deLength(package, &text)) continue; // 对报文进行解包
Response resp;
resp.deserialize(text); // 对报文反序列化
std::cout << "exitcode: " << resp.exitcode << ", result: " << resp.result << std::endl;
}
}
}2.4 All codes
The code is all in gitee
code_linux/code_202306_27/protocol Maple_fylqh/code - Code Cloud - Open Source China (gitee.com)
2.5 Code Testing
Compile without problems
Create a new window, run the server first, then start the client, the client first uses the local loopback to test
client, normal
The input needs to be in accordance with the protocol requirements, and the test result is normal
The above code is just to understand what is the protocol, the process of serialization and deserialization
3. Serialization and deserialization
For serialization and deserialization, we will not write it ourselves. The above is just to experience the process. Serialization and deserialization have corresponding library support. They are all ready-made solutions. We can use the library directly. The protocol is We may need to write our own
Common serialization and deserialization libraries:
- json
- protobuf
- xml
Among them, json is easy to use, supported by C++, Java, Python, etc., protobuf and json are commonly used in C++, and xml is commonly used in Java
Install the json library
install -y jsoncpp-develNote: Ordinary users need sudo privilege escalation
The installation is complete
Generally installed in this path
Using json needs to include header files
#include <jsoncpp/json/json.h>Compilation requires the name of the library
The article is not very easy to write, a bit watery...
--------------------- END ----------------------
「 作者 」 枫叶先生
「 更新 」 2023.6.28
「 声明 」 余之才疏学浅，故所撰文疏漏难免，
或有谬误或不准确之处，敬请读者批评指正。