foreword
I believe that without the Internet, there would be no rich world today. This note records my learning of the basic knowledge of the network under the Linux system, starting with various concepts and relationships about the network, and gradually building up the understanding of the network and the knowledge related to network programming.
My last Linux article~
[Linux] Network socket programming - qihaila's blog - CSDN blog
Let's get started~
Table of contents
3. Basic process of network transmission
Communication within the same LAN
Communication not within the same LAN
1. Re-understand the protocol && network version calculator
Serialization and deserialization & the use of Json library
Quickly construct the message format of http request and response
1. Concepts and Relationships
1. LAN
In the beginning, computers operated independently, and if they were operated in concert, the efficiency would be very low. For example, a calculator completes part of the processing, and then copies the processed data to the next computer...
So a server was born, and several computers are linked together for network interconnection.
LAN :
When the number of computers increases, LANs are connected through routers.
Wide area network WAN :
Connect computers thousands of miles apart.
For LAN and WAN , it is actually a relative concept, which is distinguished by the size of the scale. In China, the continuous growth of the network is done by communication companies (such as the three major operators, Huawei, etc.).
2. Network protocol
First, an agreement is an agreement.
Why do conventions exist?
Two computers communicate and send data to each other, and the data exists in categories. But only 01 exists in the computer, so it needs to be distinguished, and it is necessary to agree on a protocol to distinguish categories and reduce costs (increase effective actions and reduce invalid actions).
Therefore, it is necessary to do a good job in the communication protocol of software and hardware, and unify it to become an industry standard (hardware standard).
For protocols, let's not forget that in Linux everything is a file, and that first describes the organization .
1. The operating system needs to carry out protocol management -- describe first, then organize
2. The essence of the protocol is software, and software can be layered.
3. When the protocol is designed, it is divided into layers . -> Why divide into layered structure? a. The scene is complex and b. the functions are decoupled, which is convenient for people to perform various maintenance .
After understanding the above conditions, we can use a practical problem to introduce the layered structure of the protocol: the complexity of communication is in essence positively related to the distance .
Because there is a loss during transmission, complex designs are required to reduce the loss, which is also the problem that these protocol stacks need to solve. For example, in the field of communication: the problem of packet loss needs to be solved by the transport layer ; the positioning of the target position in the transmission process needs to be solved by the network layer ; when a LAN is transmitted to another LAN to solve the next-hop host problem, a data link is required. layer ; hardware and the like have a physical layer . How data is handled within the context of the application requires the application layer . In fact, the above description is an entire network transmission ecology, which is controlled based on the TCP/IP protocol (5-layer protocol).
OSI：
Before understanding the specific TCP/IP protocol, the first thing that exists is OSI (Open Systems Interconnection Open System Interconnection), as shown in the figure below:
OSI is similar to a draft-blueprint, which means to use it as a theoretical guide to help different hosts perform data transmission. Logical definitions and specifications.
TCP/IP protocol:
After having the theoretical guidance of OSI, according to actual needs, the following TCP/IP protocol is specified , with a total of five layers, but in fact, only the upper four layers are concerned for software.
3. Basic process of network transmission
Packaging & Unpacking
First of all, if two hosts are in the same local area network, they can communicate. (such as screen projection)
Logically speaking, according to the above TCP/IP protocol, because host A sends directly to host B at the application layer (AB is two hosts in the same LAN), but actually (physically speaking) it is top-down , the link layer is transmitted to the target host, and then sent to the corresponding application layer from bottom to top at the physical location of the target host.
Each layer actually has its own protocol customization scheme, so each layer needs its own protocol header .
Add headers from top to bottom. From bottom to top is to remove the header. The header is similar to the delivery order (seller (downward delivery - fill out the form (for the courier)) - buyer (received the courier, will not ask for the courier order)) After the same layer protocol receives the data packet, the extra Part of is the header.
Layer-by-layer encapsulation into a data frame-the process of adding headers. After the LAN communication, it is delivered upwards - layer by layer through the header to identify the corresponding part (except the header, then it is the payload) to which protocol is delivered upwards, and the payload is delivered upwards. Repeat the above steps until the application layer.
Therefore, we can understand some of the essence from the above knowledge:
The essence of encapsulation: adding headers
The essence of unpacking: remove the header & expand the analysis
It can be imagined as a stack structure, encapsulation is to continuously push the stack, and unpacking is to continuously pop the stack.
Communication within the same LAN
So how do we understand the propagation between link layers in the same LAN?
We use the example of students and teachers attending classes in the same classroom as an example of communication in a LAN.
When two hosts communicate in the same local area network, it is similar to the roll call when the teacher is in class. The whole class should have heard the teacher's roll call, but only the student who was called stood up. Correspondingly, all hosts in the same LAN should receive the sent data frame, but only the target host will unpack the data packet, and the rest will be discarded if it is not for itself.
Similarly, if it is noisy in the classroom, two students who communicate normally will be disturbed. Therefore, if two hosts communicating on the same LAN are interfered by other hosts, they will not be able to communicate and send collision problems. The mitigation measures are: wait for a while, and retransmit again (note that all hosts in the same LAN must comply).
In a local area network, the only thing that represents a host is the MAC address. The network card is determined.
Under the cloud server (the version I use is centos), use the command: ifconfig eth0 to view the mac address of the current server and other intranet-related addresses.
For example, ether means the mac address. Each bit represents a hexadecimal number, a total of 12 hexadecimal numbers, and each hexadecimal number represents 4 bits, so the mac address is 48 bits. Others, such as inet, represent the ip address of the intranet.
So if the protocols followed by the link layer are different, that is, they are not in the same LAN, how should the two hosts communicate?
Communication not within the same LAN
Let's first have a preliminary understanding through the following picture:
Because it is no longer a LAN, it needs to be handed over to the router , and the Ethernet protocol header is removed through the Ethernet driver and handed over to the router. If the target ip router is in the same local area network, then the driver of its corresponding protocol is delivered downwards to add headers, and then unpacking starts from bottom to top. -- Same layer protocol. At this time, the protocols of the upper three layers are consistent, but the link layer is inconsistent.
A router is a device that connects different networks and has multiple interfaces, each of which connects to a different network. In addition, if it needs to span multiple networks, the data will be forwarded through multiple routers until it reaches the target host. During this process, each router will maintain a routing table according to the routing protocol to select the optimal path for forwarding.
Then if the above-mentioned spanning multiple networks, if A sends to B, then the MAC address has no effect, and the ip address is needed at this time. Let's use the following example to briefly understand the ip address:
For example, we use navigation to locate a certain place. At this time, our starting point to a certain place is actually two ip addresses, one is the starting ip, which is the source ip, and the destination is the target ip . But this route will pass through many places, and these many places will form many jump paths, all of which can reach the target ip, but the length of the path exists, which is up to us to choose.
So many of these places are actually the MAC address, and the mac address will continue to change. But our source ip and destination ip will not change. And the change of the mac address is what the router does. And the mac address of the next stop is affected by the impact from ip.
Therefore, in the use of the TCP/IP protocol, the protocol seen above the intermediate route, that is, ip, is consistent . That is, the messages are also the same.
When you see this, you should have a certain understanding of the network. At this time, we can learn some content about network sockets to deepen our understanding. You can use my blog or other materials to learn~ The content after that is We need to master socket programming, so I won't go into details in this article.
[Linux] Network socket programming - qihaila's blog - CSDN blog
2. Application layer
Next, let's talk about the usage and understanding of application layer related protocols in detail. Note that the basic programming of TCP and UDP sockets is required here , and we will use the code to understand the relevant knowledge more deeply.
1. Re-understand the protocol && network version calculator
Let's first understand the protocol by implementing a network version of the calculator.
This network version of the calculator adopts the TCP protocol, and the server calculates the data sent by the client and returns it to the corresponding client.
Then let's think about it: when transmitting between hosts, TCP is oriented to byte streams . Generally speaking, it is the transmission between strings. So if we want the client to send data to the server, we can parse the string to do so.
But parsing strings is inevitably too coupled, that is to say, the scalability will become very poor. Once the server is changed, the client will not be able to use it. Then we can think about it, can we pack the data that needs to be transmitted into a structure ? Can't we just send the structure?
The idea is fine, but don’t forget that under different platforms, there are problems such as size endianness and memory alignment. If you send the data of the structure directly, the read data is likely to be wrong. Therefore, here we send the structure before , first perform a serialization , convert it into a string and send it, and then deserialize it after sending it to the target host , and convert it into a structure. There will be no errors at this time.
Serialization and deserialization & the use of Json library
In fact, the operation of serialization and deserialization is a protocol-customized operation. For serialization, we specify a format so that data can be represented in a string, and then deserialization is to read the correct data from this string according to the format. Of course, serialization has already been implemented by others. We can verify it in two steps in the following code implementation. One is to use our own defined protocol, and the other is to use the third-party library Json to implement.
In the following code, we define Request as the request class and Response as the response class . We believe that: when the client communicates with the server, the client first sends a request to the server. This request is the calculation expression this time. We can use three variables to store this expression, and two variables to store the data. , a variable holds the symbol. When we need to send, we only need to serialize this request class object into a string. (The code is defined as length\r\n_x _op _y\r\n) There are 2 reasons for this definition: 1. length can confirm whether the follow-up is complete for us, if it is not complete (that is, when sending based on the byte stream, there is no completely) then read on . 2. \r\n is a relatively common format. If you only want to keep the same space, the space in the middle is to ensure that different data can be obtained during subsequent deserialization. When the server receives the request from the client, it needs to deserialize the string, and deserialization means parsing the above string. The same is true for the Response response, except that we define a return code and a return value inside. You can judge whether the result is valid according to the return code.
If the Json library is used, then we don't need to define the form of the string ourselves, we only need to bring the library and the corresponding key-value form.
First, you need to install the Json library, the installation command: sudo yum install jsoncpp-devel. Then you need to add -ljsoncpp to the compilation option to find this library. When quoting in the file #include<jsoncpp/json/json.h> is enough.
Before transferring data, we need to save the data, use Json::Value root ; an object can be created, similar to map or hash map, which supports key-value, so we can use root[key] = to add data value can be added. When serializing, you only need to create a Writer object. There are two options for this object, a StyledWriter and a FastWriter . Among them, the format converted by FastWriter to a string is more streamlined, depending on personal preference. Generally, the first one is used for debugging, and the second is used for daily use. Secondly, after creating the Json::FastWriter writers object, just writers.writer(root); will return a string for sending. Use the writer method to serialize the Value object . For deserialization, Json::Reader read; object is enough, read.parse(string, Value object); use parse method to deserialize string into Value object . When extracting data, you need to specify the data type , such as int _x = root["x"].asInt();
Next, we use conditional compilation in the ComputerProtocol.hpp file to decide whether to use a custom solution or a json library solution. The code is as follows:
#pragma once
#include <string>
#include <iostream>
#define My_Protocol
#ifndef My_Protocol
#include <jsoncpp/json/json.h>
#endif
// 计算器业务 - 自定义协议
namespace ComputerData
{
#define SPACER "\r\n"
#define SPACERLEN (sizeof(SPACER) - 1)
// 请求
class Request
{
public:
Request() = default;
Request(int x, int y, char op)
:_x(x), _y(y), _op(op)
{}
// 序列化
// 序列化格式: length\r\n_x _op _y\r\n
std::string Serialize()
{
#ifdef My_Protocol
std::string str = std::to_string(_x);
str += ' ';
str += _op;
str += ' ';
str += std::to_string(_y);
std::string header = std::to_string(str.size());
return header + SPACER + str + SPACER;
#else
// 此处使用Json库
Json::Value root;
root["x"] = _x;
root["op"] = _op;
root["y"] = _y;
Json::FastWriter writer;
return writer.write(root); // 一步就序列化
#endif
}
// 反序列化
// 提取到对应的元素中去
// length\r\n_x _op _y\r\n
// 5\r\n1 + 1\r\n
bool Deserialization(const std::string& str)
{
#ifdef My_Protocol
// 首先，到达这里的str必须是上述样例的模样，不能多一个字符或者少一个字符
// 下面进行字符串解析，如果一个存在不遵守，那么就无法提取成功
size_t index1 = str.find(SPACER, 0);
if (index1 == std::string::npos) return false;
int length = atoi(str.substr(0, index1).c_str());
size_t begin1 = index1 + SPACERLEN;
size_t index2 = str.find(' ', begin1);
if (index2 == std::string::npos) return false;
_x = atoi(str.substr(begin1, index2 - begin1).c_str());
size_t begin2 = index2 + 1;
size_t index3 = str.find(' ', begin2);
if (index3 == std::string::npos) return false;
_op = str[index3 - 1];
size_t begin3 = index3 + 1;
size_t index4 = str.find(SPACER, begin3);
if (index4 == std::string::npos) return false;
_y = atoi(str.substr(begin3, index4 - begin3).c_str());
// std::cout << length << " x:" << _x << " op:" << _op << " y:" << _y << std::endl;
return true;
#else
// 使用Json库进行反序列化
Json::Value root;
Json::Reader read;
read.parse(str, root); // 反序列化
_x = root["x"].asInt();
_op = root["op"].asInt();
_y = root["y"].asInt();
return true;
#endif
}
int _x;
int _y;
char _op;
};
// 响应
class Response
{
public:
Response() = default;
Response(int code, int result)
:_code(code), _result(result)
{}
// 序列化
// 序列化格式: length\r\ncode result\r\n
std::string Serialize()
{
#ifdef My_Protocol
std::string str = std::to_string(_code);
str += ' ';
str += std::to_string(_result);
std::string header = std::to_string(str.size());
return header + SPACER + str + SPACER;
#else
Json::Value root;
root["code"] = _code;
root["result"] = _result;
Json::FastWriter writes;
return writes.write(root);
#endif
}
bool Deserialization(const std::string& str)
{
#ifdef My_Protocol
// 首先，到达这里的str必须是上述样例的模样，不能多一个字符或者少一个字符
// 下面进行字符串解析，如果一个存在不遵守，那么就无法提取成功
size_t index1 = str.find(SPACER, 0);
if (index1 == std::string::npos) return false;
int length = atoi(str.substr(0, index1).c_str());
size_t begin1 = index1 + SPACERLEN;
size_t index2 = str.find(' ', begin1);
if (index2 == std::string::npos) return false;
_code = atoi(str.substr(begin1, index2 - begin1).c_str());
size_t begin2 = index2 + 1;
size_t index3 = str.find(SPACER, begin2);
if (index3 == std::string::npos) return false;
_result = atoi(str.substr(begin2, index3 - begin2).c_str());
// std::cout << length << " code:" << _code << " result:" << _result << std::endl;
return true;
#else
Json::Value root;
Json::Reader read;
read.parse(str, root);
_code = root["code"].asInt();
_result = root["result"].asInt();
return true;
#endif
}
int _code; // 状态码
int _result; // 结果
};
}In addition, for our custom, it is necessary to determine whether it is complete when deserializing . Because TCP sends data based on the byte stream, it is not like the datagram is sent at one time, so it is possible that we only read a part, so we need a function to check each string we accept, if If it is correct, save this string and return it. Reset the buffer for acceptance and continue to accept. If it is not correct, you need to return an empty string to tell the upper layer that the acceptance is not complete, and then the buffer cannot be moved and continue to accept. (So here it means that the buffer accepted by recv must be accepted in the form of +=)
// 11\r\n121342 + 12\r\n 17 - 2*2 - 2 = 17-6 = 11
std::string Decode(std::string& str)
{
// 解码，即必须检查当前读取的是否是完整的数据，如果不是就继续读取
// 首先，开头必须是数字
size_t index1 = str.find(SPACER, 0);
if (index1 == std::string::npos) return "";
int length = atoi(str.substr(0, index1).c_str());
int pos = str.size() - 2 * SPACERLEN - index1;
if (pos >= length)
{
// 说明此时长度已经可以提取了，修改返回即可
std::string s = str.substr(0, length + 2 * SPACERLEN + index1);
str.erase(0, length + 2 * SPACERLEN + index1); // 删掉，此时外面的str为+=
return s;
}
else return ""; // 否则返回空串
}Of course, Json is the same, you can study it by yourself~
Before serialization, we first prepare the prerequisites for socket programming. The server first initializes the listening socket, and then returns to the service socket during the connection establishment process to read data, process business, and send data. The client initializes the client socket, sends and receives messages after the connection is established. code show as below:
In the MySock.hpp file, we encapsulate some common uses of TCP sockets, such as creating a socket sock , binding the socket object to ip and port , setting the listening socket listen on the server side , and accepting connections on the server side Return the service socket acceot , the client connects with the server . In addition, according to the above deserialization requirements, customize the acceptance of recv . Of course, send is free.
#ifndef __MY_SOCK__
#define __MY_SOCK__
// 封装TCPsock相关操作接口
#include <iostream>
#include <string>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <cstring>
#include "log.hpp"
namespace QiHai
{
struct Sock
{
// 创建套接字 错误返回-1，这里会输出错误日志，注意佩戴上log.hpp文件，编译加-DMYFILE选项日志输入到当前目录的文件中，否则就是标准输出
static int socket()
{
int sock = ::socket(AF_INET, SOCK_STREAM, 0);
if (0 > sock) // ::表示使用全局命名空间的，不属于任何一个类或者命名空间 SOCK_STREAM是字节流-即TCP协议
{
logMessage(FATAL, "socket error-%d: %s[file:%s|line:%d]", errno, strerror(errno), __FILE__, __LINE__);
}
else logMessage(NORMAL, "socket success, sock is %d", sock);
return sock;
}
// 绑定ip和端口
static bool bind(int sock, int16_t port, std::string ip = "0.0.0.0")
{
struct sockaddr_in address;
socklen_t n;
memset(&address, 0, sizeof address); // 清零
address.sin_family = AF_INET;
address.sin_port = htons(port); // 主机转网络
inet_aton(ip.c_str(), &address.sin_addr); // 点分十进制转化为数，然后转化为网络字节序
if (0 > ::bind(sock, (sockaddr*)&address, sizeof(address)))
{
// 差错处理
logMessage(FATAL, "bind error-%d: %s[file:%s|line:%d]", errno, strerror(errno), __FILE__, __LINE__);
return false;
}
return true;
}
// 服务端设置监听
static bool listen(int sock, int backlog = 20) // 挂起队列最大长度默认设置为20
{
if (0 < ::listen(sock, backlog))
{
logMessage(FATAL, "listen error-%d: %s[file:%s|line:%d]", errno, strerror(errno), __FILE__, __LINE__);
return false;
}
logMessage(NORMAL, "listen success, init......");
return true; // 设置此套接字为监听状态成功
}
// 服务器进行连接
static int accept(int listenSock, std::string* clientIp = nullptr, int16_t* clientPort = nullptr)
{
// 想要客户端的ip和port为可选项
struct sockaddr_in client;
memset(&client, 0, sizeof client);
socklen_t addrlen = sizeof client;
int clientSock = ::accept(listenSock, (sockaddr*)&client, &addrlen);
if (0 > clientSock)
{
logMessage(ERROR, "accept error-%d: %s[file:%s|line:%d]", errno, strerror(errno), __FILE__, __LINE__);
// 错误但是不致命，可以进行重连哦
return clientSock;
}
std::string ip = inet_ntoa(client.sin_addr);
int16_t port = ntohs(client.sin_port);
logMessage(NORMAL, "[%s: %d] accept suncess...", ip.c_str(), port);
// 如果指针不为空就设置客户端返回的数据
if (clientIp)
{
*clientIp = ip; // 网络字节序-本机-数字-点分十进制
}
if (clientPort)
{
*clientPort = port; // 网络-本机
}
return clientSock;
}
// 客户端进行连接
static bool connect(int sock, const std::string& ip, int16_t port)
{
struct sockaddr_in server;
memset(&server, 0, sizeof server);
server.sin_family = AF_INET;
server.sin_port = htons(port);
inet_aton(ip.c_str(), &server.sin_addr);
if (0 > ::connect(sock, (sockaddr*)&server, sizeof server))
{
logMessage(FATAL, "connect error-%d: %s[file:%s|line:%d]", errno, strerror(errno), __FILE__, __LINE__);
return false;
}
return true;
}
// 发送消息
static bool send(int sock, const std::string& buffer)
{
if (0 > ::send(sock, buffer.c_str(), buffer.size(), 0))
{
logMessage(FATAL, "send error-%d: %s[file:%s|line:%d]", errno, strerror(errno), __FILE__, __LINE__);
return false;
}
return true;
}
// 接收消息
static bool recv(int sock, std::string& buffer)
{
char _buffer[1024];
// printf("------------1-------------\n");
ssize_t n = ::recv(sock, _buffer, sizeof(_buffer) - 1, 0);
// printf("------------2-------------\n");
if (n > 0)
{
_buffer[n] = '\0';
buffer += _buffer; // 注意必须是+= 此时返回的数据最后才能保证是完整的
}
else if (n == 0)
{
logMessage(NORMAL, "对方主机关闭......");
return false;
}
else{
logMessage(FATAL, "recv error-%d: %s[file:%s line:%d]", errno, strerror(errno), __FILE__, __LINE__);
return false;
}
return true;
}
};
}
#endifThe log.hpp file can be found in my previous Linux blog~ Of course, you can also change it to standard output, the same, anyway, it is just a specific output of some text.
Then write out our server-side and client-side codes. The client side is the loop creation request after the connection is successful. After the sending is successful, it waits for the response from the server side, and the result is returned after deserialization. After the server gets the request, it deserializes the obtained data for processing, and then makes a response and serializes it for sending. Note that the acceptance is different from the previous socket programming. We need to judge whether the accepted string meets the requirements, otherwise we need to continue reading.
//UserManual.hpp - 帮助文件
#pragma once
#include <iostream>
#include <string>
static void User(const std::string& argv)
{
if (argv == "./TCPServer")
std::cout << "User: " << argv << " port" << std::endl;
else if (argv == "./TCPClient")
std::cout << "User: " << argv << " ip port" << std::endl;
else
std::cout << "?" << std::endl;
}
//TCPClient.cpp - 客户端代码
#include "MySock.hpp"
#include "UserManual.hpp"
#include <unistd.h>
#include "ComputerProtocol.hpp"
int main(int argc, char* argv[])
{
if (argc != 3)
{
User(argv[0]);
exit(-1);
}
int sock = QiHai::Sock::socket();
if (sock < 0)
exit(1);
// 注意客户端不需要进行绑定
// 与服务器连接
if (!QiHai::Sock::connect(sock, argv[1], atoi(argv[2]))) exit(2);
// 连接完毕后处理业务即可
// std::string message;
// while (true)
// {
// std::cout << "请输入# ";
// std::getline(std::cin, message);
// QiHai::Sock::send(sock, message);
// if (QiHai::Sock::recv(sock, message))
// {
// std::cout << "server# ";
// std::cout << message << std::endl; // 注意类似于endl和'\n'等需要刷新缓冲区的！务必记住
// }
// else break;
// }
std::string res;
bool quit = false;
while (!quit)
{
std::cout << "请输入(x+y):";
ComputerData::Request r;
std::cin >> r._x >> r._op >> r._y;
QiHai::Sock::send(sock, r.Serialize());
while (true)
{
if (QiHai::Sock::recv(sock, res))
{
std::string tmp = ComputerData::Decode(res);
if (tmp.empty()) continue;
ComputerData::Response p;
p.Deserialization(tmp);
if (p._code == 1) std::cout << "除零错误" << std::endl;
else if (p._code == 2) std::cout << "未识别操作符" << std::endl;
else if (p._code == 0) std::cout << r._x << r._op << r._y << "=" << p._result << std::endl; // 注意类似于endl和'\n'等需要刷新缓冲区的！务必记住
break;
}
else
{
quit = true;
break;
}
}
}
close(sock);
return 0;
}
//TCPServer.cpp - 服务器端代码
#include "TCPServer.hpp"
#include <memory>
#include "UserManual.hpp"
#include "ComputerProtocol.hpp"
void testComputerProtocol()
{
// 测试序列化是否正确
ComputerData::Request r(121342, 12, '+');
// std::cout << "序列化：";
std::string str = r.Serialize();
std::cout << str << std::endl;
// std::cout << "反序列化：";
ComputerData::Request q;
q.Deserialization(str);
// std::cout << "------------------\n";
ComputerData::Response p(1, q._x + q._y);
// std::cout << "序列化：";
std::string str2 = p.Serialize();
std::cout << str2 << std::endl;
// std::cout << "反序列化：";
ComputerData::Response q2;
q2.Deserialization(str2);
std::string tmp = ComputerData::Decode(str); //
if (tmp.empty()) printf("失败\n");
else std::cout << tmp << std::endl;
std::string tmp2 = ComputerData::Decode(str2);
if (tmp.empty()) printf("失败\n");
else std::cout << tmp2 << std::endl;
std::cout << "-------------------------------\n";
std::cout << str << "\n\n" << str2 << std::endl;
}
void test(void* argc)
{
QiHai::TCPServerData* tmp = (QiHai::TCPServerData*)argc;
std::string buffer;
while (true)
{
if (QiHai::Sock::recv(tmp->_sock, buffer))
{
std::cout << "[" << tmp->_ip << ": " << tmp->_port << "]# ";
std::cout << buffer << std::endl; // 注意类似于endl和'\n'等需要刷新缓冲区的！务必记住
}
else break;
QiHai::Sock::send(tmp->_sock, buffer);
}
exit(0);
}
void computer(void* argc)
{
// 实现网络计算器的业务
QiHai::TCPServerData* tmp = (QiHai::TCPServerData*)argc;
std::string buffer;
ComputerData::Request q;
while (true)
{
if (QiHai::Sock::recv(tmp->_sock, buffer))
{
// printf("+++++++++1++++++++++++++\n");
// 检查读取的是否完整
std::string res = ComputerData::Decode(buffer);
if (res.empty()) continue; // 继续读
std::cout << "[" << tmp->_ip << ": " << tmp->_port << "]# ";
q.Deserialization(res);
std::cout << q._x << q._op << q._y << std::endl;
}
else break;
int result, code = 0;
switch (q._op)
{
case '+':
result = q._x + q._y;
break;
case '-':
result = q._x - q._y;
break;
case '*':
result = q._x * q._y;
break;
case '/':
if (q._y != 0) result = q._x / q._y;
else code = 1;
break;
default:
code = 2;
break;
}
ComputerData::Response p(code, result);
QiHai::Sock::send(tmp->_sock, p.Serialize());
}
}
int main(int argc, char *argv[])
{
if (argc != 2)
{
User(argv[0]);
exit(-1);
}
std::unique_ptr<QiHai::TCPServer> server(new QiHai::TCPServer(atoi(argv[1]), computer));
server->init();
server->start();
// testComputerProtocol();
return 0;
}At this point, we can do a test and we will be successful~
So, can we introduce a daemon process to let its process background complete this task?
daemon process
A daemon is a background process. How do we create a background program?
Before creating daemons, we need to understand the session system in Linux.
When a user logs in to a Linux system, a new session is created. A session is a period of time during which a user interacts with the system. During this time period, users can open and close terminal windows, execute commands, processes and other operations.
All servers written so far run in the foreground.
1. Foreground process : The process associated with the terminal -- that is, whether the process can normally obtain your input is the foreground process. - Allows me to enter and operate things
2. Any xshell login only allows one foreground process and multiple background processes.
3. In addition to its own pid and ppid, the process also has a group id.
4. In the command line, use pipelines to start multiple processes at the same time. Multiple processes are siblings, and the parent process is bash. Anonymous pipeline communication can be used. 5.
Multiple processes created at the same time can be a process group. The group leader is the first created process PGID
6. For any login, the logged-in user needs to have multiple processes (groups) to provide services to this user (bash). Users themselves can start many processes or process groups. The process of providing services to users or the process started by itself must belong to a session mechanism as a whole. SIDUser -> login : session (bash (bash itself forms a group) + terminal --- process group)
user <- logout : In theory, any process that was ever started must be released. --There may be differences in the processing of different operating systems.
So if we want a resident process, we need to avoid the process in the session where bash is located. We need this process to form a session group by itself.
7. How to talk about self-editing and self-contained conversations? pid_t setsid(void) ; -- Create a session and call it the group leader. If the return value succeeds, it means who called me, and if it fails, it means -1.
8. But if setsid is to be called successfully, it must be ensured that the current process is not the leader of the process group . How can I be sure that I am not the team leader? become a child process. -fork() is guaranteed.
9. The daemon cannot print messages directly to the monitor. Once printing will be suspended or even terminated.
We want our process to become a daemon process. First, we need to convert all the server's output to the display screen into input to the file (set it yourself, if you use my log file here, add a -DMYFILE macro to point the output to the current file. ). And ignore the signals that affect the process hang: such as SIGPIPE signal and SIGCHLD signal . The SIGPIPE signal means that sending a message to a closed client in network programming will be killed by the system sending this signal. Naturally, we don't want this to happen, and we also ignore SIGCHLD to prevent zombie processes from appearing.
Then it becomes the group leader of the new session itself - first of all, it is a child process. Secondly, the three file descriptors opened by default - standard input and output errors are redirected to the /dev/null file to prevent it from affecting other files. (In Linux systems, /dev/null is a special file, also known as a null device. It is a virtual device file, and any content written to it will be discarded, and any read from it operation will return an empty sequence of bytes. Simply put, it is a black hole, and all input will be swallowed.)
In this way, one of our daemon processes is created:
//Daemon.hpp
#pragma once
#include <iostream>
#include <signal.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
// 将此进程变为守护进程 - 利用接口setsid成为新的会话（注意自己不可是组长 - 利用子进程即可）
// 将此进程默认打开的三个文件描述符重定向到黑洞中：/dev/null
// 忽视信号SIGPIPE, SIGCHLD
void MyDaemon()
{
signal(SIGPIPE, SIG_IGN);
signal(SIGCHLD, SIG_IGN);
if (fork() > 0) exit(0);
// 孤儿进程 - 不是组长了
setsid();
int nullfile = open("/dev/null", O_RDONLY | O_WRONLY); // 读和写的方式打开
if (nullfile > 0)
{
dup2(0, nullfile);
dup2(1, nullfile);
dup2(2, nullfile);
close(nullfile);
}
}
At this point, you can see that it has become a background process when you run it. You can see this process by using the netstat -ntap command, and you can kill this process by using kill -9 pid.
Through the above experiments, we understand that in real network communication, since it is transmitted through strings, we generally use a regulation to specify how to convert data into byte streams and convert byte streams for the corresponding data. Such is the agreement.
2. http protocol
In fact, the application layer is nothing more than the specific logic written by programmers based on the socket interface, and a lot of work is related to text processing! --Protocol analysis and processing.
Then, after development, there are some very useful and extensive application layer protocols that can be directly provided for our reference. Among them, http (Hypertext Transfer Protocol) is one of them~
Before starting to learn the http protocol, let's focus on understanding the prerequisite knowledge such as URL and special character escaping.
URL
Obviously, if you are currently viewing my blog on a PC, there should be a URL on your browser. A URL is actually what we usually call a web address. Under the http protocol, the URL is as follows:
http://domain name[:port]file path...
The first http indicates the protocol name . Of course, the encrypted https is now the mainstream (https follow-up modules will explain). A domain name is actually a company or individual associated with the domain name based on the IP address , which is unique on the entire network. The port is optional. In the http protocol, the default access is port 80 , and in the https protocol, the default access is port 443 . Of course, you can also specify the port yourself. Don't forget that the ip and port in the socket represent the only process in the entire network. The file path is the path to access the target process - the web server , but do not look at \ is the root directory of the access, one of the directories may be specified during the encoding process (we can see it in the subsequent encoding implementation), if not specified, the default that is\ .
We can think about it in reverse, what are the routine behaviors we usually use when surfing the Internet?
1. What do we want to get . A picture, video, text, etc. -- when the resource is not obtained, the resource is on the corresponding server. (Linux) There may be many resources on a server. -File--Request resources to get to your local--The service process opens the file you want to access, reads the file, and sends the file to the client through the network. ->Open this file, find this file first. In Linux, a file is identified by a path. At this time, the web root directory: / is the file separator, which is exactly the file path under Linux.
2. What do we want to upload .
Therefore, the URL is actually the only resource that locates the Internet. Uniform Resource Locator. All resources in the world can be accessed as long as you find their URL. So www is the World Wide Web.
urlencode and urldecode
However, sometimes characters such as / ? have been understood by the URL as special meanings, so these characters cannot appear at will, but when some parameters need to carry these special characters, they need to be escaped.
The rules of escaping are as follows: convert the characters to be transcoded into hexadecimal, and then from right to left, take 4 digits (less than 4 digits and process them directly), make one digit for every 2 digits, add % in front, and encode it as % XY format . The character is determined by the ASCII code value, such as '?' How do we transcode? It is known that the encoding of ?utf-8 is 0x3F, (note that it is an English question mark), then taking 4 digits from right to left is exactly one hexadecimal, which is %3F, such as the following URL:
Then, analogous to the network calculator we wrote before, the client should send a request to the server, and the server will parse it after receiving it and send back a response. It's just that this time it has become an application layer protocol developed by http.
Then in the transmission process, in order to extract the transmission format and the needs of serialization and deserialization, it naturally has its own set of messages.
Quickly construct the message format of http request and response
Purely from the message point of view, http can be a line-based text protocol.
Request request format:
Request line : method url (if there is no destination path, it will visit/default path) request protocol version (http/1.1)\r\n (separated by spaces)
Request header (relevant attributes of this request, followed by \r\n)
key: value\r\n
......
6~8 request linesEmpty line \r\n (only this content)
request body (can be absent)
Response response:
Status line : protocol version (server version) status code status code description\r\n(the most common status and status code is 404 NotFound)
Response header (relevant attributes of this request, followed by \r\n)
key: value\r\n
......
6~8 request linesEmpty line \r\n (only this content)
Response body (video, audio, image, html...)
Of course, for a computer, looking at http requests and responses is a linear structure-a string structure. The protocol version is that the two hosts check whether the protocol version of the other party is consistent with their own during the exchange process, and the inconsistent server needs to provide different services. The url is the resource path you want to access, if not, then the default is \.
Now, we can establish a consensus: how does the protocol encapsulate headers-encapsulation, unpacking? That is, how does http distinguish between headers and payloads?
\r\ nDistinguish header and payload by blank line. You must be able to read the header -> the next step is to read the text. -> How do we know the size of the text? (There is an attribute in the header——Content-Length: 123) Then you can know how many bytes the text has.
So can we use the small tool telnet to quickly build an http request and check the results returned by the server?
Enter the command: telnet www.baidu.com 80 Enter the input state and enter a line of header: host: www.baidu.com, and then press Enter twice, we can get the following results:
The html file is the body returned by the response. If it is a browser, it will render it to form the web page we see.
So can we now simply implement a web server based on TCP socket programming, accept connections from browser clients on the server, and return a simple web page? Of course you can.
First of all, we need to use the mysock file we wrote above, that is, the interface related to the TCP socket is encapsulated by ourselves, and we can call it directly when creating the server, omitting a lot of steps.
When creating a server, a function object is passed in, and the parameter of the function is the service socket, which is convenient for us to send and receive information to the corresponding client, and then this function can be written in the cpp source file. The code for server creation is as follows:
#include "MySock.hpp"
#include <functional>
#include <signal.h>
#include <unistd.h>
using func_t = std::function<void(int)>;
class HttpServer
{
public:
HttpServer(int16_t port, func_t func)
:_listenSock(-1),_serveSock(-1), _port(port), _func(func)
{}
void init()
{
_listenSock = QiHai::Sock::socket();
if (_listenSock < 0) exit(1);
if (!QiHai::Sock::bind(_listenSock, _port)) exit(2);
if (!QiHai::Sock::listen(_listenSock)) exit(3);
}
void start()
{
signal(SIGCHLD, SIG_IGN); // 应用层设置为忽视，这样子进程退出就会自动被系统回收，不会产生僵尸进程了
while (true)
{
_serveSock = QiHai::Sock::accept(_listenSock);
if(_serveSock < 0) continue;
// 创建子进程对其处理
if(fork() == 0)
{
// 子进程
close(_listenSock);
_func(_serveSock); // 传给上一层传入此函数去处理
close(_serveSock);
exit(0);
}
close(_serveSock);
}
}
private:
int _listenSock;
int _serveSock; // 服务套接字
int16_t _port;
func_t _func;
};Now, we can focus on writing the web server code. Since the communication is carried out through the http protocol, the browser client is similarly sent to the request we sent to Baidu's web server through telnet before. We can simply print it out, and then follow the format of the response, the response line: http/1.1 200 ok\r\n Then add a blank line without a response header, followed by a simple body: html: <html>< h3>hello!</h3></html> . Of course, the above is just a demonstration. In fact, we do not recommend doing so (directly write the html file into the source file). As for the status code and status code description, we will explain it later.
//httpServer.cpp
#include "HttpServer.hpp"
#include <memory>
#include <iostream>
#include <string>
#include "UserManual.hpp"
void httpFunc(int socket)
{
char buffer[10240];
ssize_t n = recv(socket, buffer, sizeof(buffer) - 1, 0);
if (n > 0)
{
buffer[n] = '\0';
std::cout << buffer << "---------------------------------" << std::endl;
// web服务器收到的是一个http的请求
// 方法 路径(没有默认\) 协议版本\r\n
// key-value....
// \r\n
// 请求正文(可以没有)
// 1试着自己手动构建一个http响应
std::string httpresponse = "HTTP/1.1 200 ok\r\n"; // 协议版本 状态码(比如404) 状态码描述\r\n
httpresponse += "\r\n"; // 空行
// 正文
httpresponse += "<html><h3>hello!</h3></html>"; // 一般不这么写，一般卸载html文件内，这里只是作为测试
send(socket, httpresponse.c_str(), httpresponse.size(), 0); // 向客户端(比如浏览器访问此web服务器)
}
}
int main(int argc, char* argv[])
{
if (argc != 2)
{
User(argv[0]);
exit(-1);
}
std::unique_ptr<HttpServer> server(new HttpServer(atoi(argv[1]), httpFunc));
server->init();
server->start();
return 0;
}After we run the compilation, we can observe the following results:
It can be found that at this time, both the browser on the PC and the browser on Android sent requests to our server, and our server also sent a simple response.
Since the response can be easily constructed, can we decompose the returned http string (according to the line), and decompose the first line according to the space, so that we can get the relevant data of the request line sent by the http request sent by the other client up. We do an analysis of the directory it wants to visit, and put the html file in the wwwroot directory. If there is a path, access the corresponding resource (use the file to read the content of the html file and return it as the text). By default, it will return the index.html file in the wwwroot directory - the homepage file. If there is no path, we will return the file in the error folder. error.html file.
First, let's implement the function of splitting a string and save it in the header file of the util tool:
// util.hpp
#ifndef _WWW_UTIL_
#define _WWW_UTIL_
#include <vector>
#include <string>
// 工具类
// 提供字符串切割，根据传入指定的字符串和字符对其进行切分，传入vector数组中。字符串数组 分隔符 待处理字符串
static void cutString(std::vector<std::string>& v, const std::string& sep, const std::string& processstr)
{
// abc d e\0 sep=' '
size_t index = 0, tmp = 0;
size_t len = processstr.size();
while (index < len)
{
index = processstr.find(sep, tmp);
v.push_back(processstr.substr(tmp, index - tmp));
tmp = index + sep.size();
}
}
#endifThen we simply achieve our above goal in the function called by our server:
static const char* ROOTPATH = "wwwroot";
void httpFunc(int socket)
{
char buffer[10240];
ssize_t n = recv(socket, buffer, sizeof(buffer) - 1, 0);
if (n > 0)
{
buffer[n] = '\0';
std::cout << buffer << "---------------------------------" << std::endl;
// web服务器收到的是一个http的请求
// 方法 路径(没有默认\) 协议版本\r\n
// key-value....
// \r\n
// 请求正文(可以没有)
std::vector<std::string> v; // 整体数据
cutString(v, "\r\n", buffer); // 对传回来的请求做解析，提取每一行数据
std::vector<std::string> requestline; // 针对于请求行
cutString(requestline, " ", v[0]);
std::string httpresponse; // 要发送的响应
std::string path; // 打开资源的路径
if (requestline[1] == "/")
{
// 默认不添加任何资源位置，就是/-默认的
path = "wwwroot/index.html";
}
else path = ROOTPATH + requestline[1];
std::string text;
std::ifstream in(path); // 默认in 读的方式打开本地文件
httpresponse += "HTTP/1.1 200 ok\r\n"; // 2xx 为正确响应
if (in.is_open())
{
// httpresponse += "HTTP/1.1 200 ok\r\n"; // 2xx 为正确响应
// 成功打开
std::string line;
while (std::getline(in, line))
{
text += line;
}
in.close();
}
else
{
// httpresponse += "HTTP/1.1 404 NotFound\r\n"; // 4xx 为服务器无法处理请求
// 打开文件失败，返回错误界面
std::ifstream err("wwwroot/error/error.html");
std::string line;
while (std::getline(err, line))
{
text += line;
}
err.close();
}
httpresponse += "\r\n"; // 空行，属性没有带
httpresponse += text; // 响应正文
send(socket, httpresponse.c_str(), httpresponse.size(), 0);
}
}
The html file simply applies the template:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>QiHai</title>
</head>
<body>
<h3>欢迎来到我的第一个界面！</h3>
</body>
</html>
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>error</title>
</head>
<body>
<h1>抱歉！你访问的资源并不存在！</h1>
</body>
</html>Take a look at the directory structure at this time:
You can simply test the effect:
It can be seen that our current effect has been achieved. So, after the demonstration is over, let's start to refine the content in http step by step.
http details:
Observing the above example, we can find that the essence of http is text analysis.
For the request methods in the request line, there are roughly the following classifications:
But in fact, the most commonly used methods are GET and POST.
We describe these two methods in detail below:
GET and POST
For online behavior, our two request methods have different scenarios.
1. Get the resource data from the server.
GET method.
2. Submit the client data to the server.
GET method, POST method.
But for submission, the GET method and the POST method have different behaviors. GET will echo the transmitted content in the url column, but the POST method will not echo it, but write it into the body, which is relatively private.
Below, we use the html form to verify the difference between the two methods in the second case.
You can take a look at the basic format of the html form:
<form name="input" action="/a/b/notexist.html" method="GET">
Username: <input type="text" name="user"> <br/>
Password: <input type="password" name="pwd"> <br/>
<input type="submit" value="登陆">
</form>We don't need to worry about the html form, and we can simply look it up if we want to know. In short, the html form is actually to collect user data, and push the user data to the server according to a certain method. The data in the form will be converted into a part of the http request. Action is sent to the target file, and method is a different method. Let's test GET first. You don't need to touch any part of the main program, just add the above html table to the body of the index.html webpage.
It can be seen that after we submit, we can find that it is displayed in the url column. This is because the information will be displayed in the request line if the GET method is used for submission:
But this will be very insecure, let's try the POST method.
It can be found that it is not displayed in the url address bar at this time, we check the results of server monitoring:
As you can see, it is written in the text. This is the difference between GET and POST.
response status code
Then, in the response sent from the server to the client, there are status codes and status code descriptions in the first line, that is, the response line. As shown in the table below:
| status code | Status code description | example |
|---|---|---|
| 1XX | informational status code | |
| 2XX | The request was processed normally. | 200 OK (The request is successful. Generally used for GET and POST requests) |
| 3XX | Additional action is required to complete the request | 301 Moved Permanently (permanent redirection) 302 Found (temporarily moved. Similar to 301. But the resource is only temporarily moved. The client should continue to use the original URI) |
| 4XX | The server was unable to process the request | 404 Not Found (cannot find the corresponding resource) |
| 5XX | The server processed the request incorrectly |
The general information is as above, and the examples are only examples of common ones for understanding. If you want more details, you can check this URL: http status code .
We can use our code demo to implement the 3XX request. Because it is a redirection, in fact, 301 is ok, and the URL in the label will also change, while 302 is just a redirection. 302 is more similar to the interface that we usually jump to after logging in to the website.
{
//.....
if (in.is_open())
{
httpresponse += "HTTP/1.1 200 ok\r\n"; // 2xx 为正确响应
// 成功打开
std::string line;
while (std::getline(in, line))
{
text += line;
}
in.close();
}
if (text.empty())
{
// 为真说明文件没有找到，则没有打开，我们向重定向到err.html界面 使用302状态码进行
httpresponse += "http/1.1 302 Found\t\n";
httpresponse += "Location: http://43.143.4.250:8080/error/error.html\r\n"; // 加上此报头属性转向对应的资源 url
}
//......
}
After testing, we can see that when we access resources that do not exist, an error.html interface is returned to us. Let's observe the messages received by the server:
It can be found that for accessing resources that do not exist, the browser client sends two consecutive requests. The first time is our own request, but the second time is a request for our redirected file, so the understanding of redirection can be as follows Figure to understand:
HTTP header attributes
When we talked about the status code 302 before, we mentioned a header attribute Location, indicating the url resource that the client will access later. In addition, there are some common header attributes as shown in the following figure:
Content-Type : Data type (text/html, etc.)
Content-Length : The length of the Body
Host : The client informs the server that the requested resource is on which port of the host;
User-Agent : Declare the user's operating system and browser server version information;
referer : which page the current page is redirected from;
location : used with the 3xx status code to tell the client where to visit next;
Cookie : used to store a small amount of information on the client. Usually used to implement Session (session) function;Connection: keep-alive long connection
Connection: close short connection(A complete web page is composed of a lot of resources. Short connection-meaning that each request needs to be connected once, and the cost is a bit high. Long connection achieves the purpose of improving efficiency.)
Here for Cookie, let's talk about session management in http:
session management
Characteristics of an http session:
1. Simple and fast
2. No connection (TCP maintains the connection)
3. Stateless (I don't know the historical state - that is, it will not record any behavior of the user) For example, if the state is not recorded, then the page will be redirected after login How do I know that I am logged in? But in fact, when we use it, the general website will record my status.According to the above characteristics, it can be concluded that http only needs to maintain the network function .
In view of the above, let's elaborate on the role of the two header attributes Cookie and Set-Cookie:
Assuming that the cookie is a pass to save user data (such as account password), we use the following figure to make a demonstration:
If the cookie file is cleared, you need to log in again.
But above-mentioned existence very big potential safety hazard. Leaving aside the danger of http plaintext transmission, if the client is implanted with a Trojan horse program, if the cookie file is stolen - personal information will be seriously leaked.
So the improvement plan is to let the user's private information we save be managed under the server, and the server will generate a session id after accepting the first login - using an algorithm to help us form a unique ID. The client can continue to log in using this unique id. If a leak occurs at this time, the hacker gets the session id, and can access it again if he visits again. However, personal information will not be disclosed. cookie files - can't stop. However, there is a detection on the server side, such as a sudden change in the ip, which will invalidate the original session id. You need to log in again or set a timer for this id, and it will be invalid after a certain period of time.
However, it can be found that it is limited by the plaintext transmission of http. Before using the html form, regardless of the POST or GET method, the user's private information is in plaintext in the http message. If criminals intercept it at this time, the same There will be a privacy breach. Therefore, what we need is a more secure application layer solution to protect user privacy.
https is an application layer protocol based on http plus encrypted transmission. Because it comes with encryption, the implementation will become very complicated.