[Network] Protocol customization and Json serialization and deserialization

application layer

The network programs written by our programmers to solve our practical problems and meet our daily needs are all in the application layer

First understanding of TCP protocol communication process

• Make and break links

Based on the TCP protocol, we need to know what kind of time point role the corresponding interface belongs to during the TCP communication process when writing code, and discuss it in detail in the TCP protocol. Three handshakes, four waves

listen state: ready, you can connect, accept: get the link, not create the link, the link has already been created at the bottom layer, call accept at the application layer to get the link up

connect: 1. Initiate a connection request 2. Bind the socket; establish a connection and establish a connection request to the server at the bottom layer. In TCP, the connection scheme is a three-way handshake scheme. connect will initiate a three-way handshake and initiate a connection request It is different from the actual establishment of a link. The establishment of a link is automatically completed by the OS of both parties. Why is it automatically completed? In the network layering, the lower three layers are internal to the OS and cannot be perceived by users. Call connect through the client to let the OS help us complete the three-way handshake.

And accept is to obtain the link, the link has already been established, so accept does not participate in any details of the three-way handshake, accept must be completed before the link is obtained, that is, the link is established. The three-way handshake is completed by the OS itself, connect is only initiated, and accept is only the end. Even if the upper layer does not call accept, the three-way handshake can be established.

TCP guarantees reliability is not related to write and read, it is done by the OS of both parties, and will be discussed in detail later.

After the link is established, the link must be disconnected, so since UDP does not need to establish a link, naturally there is no need to talk about disconnecting the link

The work of waving four times is done by the OS of both parties, and when we decide to break up, once the system call close is called, the user layer does not need to worry about it.

• understand link

When talking about male and female friends, they will express their love, and there must be one party who initiates the link. No matter how you express it, the probability of seeing each other is extremely low. And how to initiate the link actively? First of all, the man confesses first, and then the woman makes a statement. When will they be together? The man answered right now. This is the successful three-way handshake between the two parties. (Although rejection is the norm in real life)

What Link Building Is Actually Doing: Note a few things

• what is link building

The so-called establishment of a connection, the three-way handshake is simply a means, not an end. In order to make both parties remember this set, one server connects to the client, and many clients come to connect, which means that many clients come. OS It should be clearly distinguished that links need to be managed and described in the organization first, and corresponding link data structures need to be created to describe all links and manage them. The so-called link is the link structure created inside the OS, which contains the corresponding attribute information when the link is established. When a new link comes in, every time a link comes, the server will build a link object, and manage all link objects internally with a specific data structure. This is the link modeling process. There is a cost to maintaining links. Occupies memory resources and must be managed with objects.

Breaking the link requires four waves of hands. The ultimate purpose of breaking the link is undoubtedly to release the established link information. Four waves to understand:

The boyfriend and girlfriend get along very well, and they have reached the palace of marriage, but they are defeated by reality and can't make it through. Then one party proposes a divorce, but what you say doesn’t count, and the other party says it’s okay. After a while, the partner says to leave, so I want to leave too. Then you see, I’m OK too. Therefore, breaking the link is a matter for both parties, and the opinions of both parties must be sought. The two sides are negotiating, TCP must ensure reliability, and you must ensure that you hear what you say, and I know it, and vice versa. This is the legendary four waves

Comparison of TCP and UDP

Reliable transmission VS unreliable transmission

Connection VS Connectionless

Byte stream vs datagram

custom agreement

Customization of Application Layer Protocols

renegotiate the agreement

The protocol is a kind of agreement. When reading and writing data, the interface of socket api is received in the form of strings. What if you want to transmit some "structured data"?

Structured data: When talking in the group, in addition to the message itself, there are also information such as avatars, nicknames, and time. But instead of individual individuals, what you need to do is to form these messages into a message—package them into a string.

From the changeable one this process is serialization. After being transmitted through the network, what is received is a message. What is the purpose of receiving a message? Turn a string into multiple strings, this process is deserialization

When business data is sent to the network, it is serialized and sent first, and the received sequence byte stream must be deserialized before it can be used

A business agreement is a structure. It is not enough to say that, so we have to write an agreement by hand.

Application scenario: Form a string and the other party receives it. After receiving it, the upper layer has no time to receive it, and the other party sends another one. It is possible to read it all at once. How can the upper layer ensure that it receives a message?

How does tcp guarantee to receive a complete message here?

Understand business protocols, understand serialization and deserialization.

web version calculator

For example, we need to implement a server version of the adder. We need the client to send the two additions, subtractions, multiplications and divisions to be calculated, and then the server will perform the calculation, and finally return the result to the client

TCP is byte-oriented, so the boundary between packets and packets is clear:

TCP is full-duplex. If the receiver has no time to read, there will be a lot of data in the receiving buffer. How to ensure that a complete message is read when reading:

1. Fixed length 2. Special symbols 3. Self-description

Serialization, deserialization and custom protocol are two different things, they are things at different stages, custom protocol: header + payload

Protocal.hpp

Custom protocol:

#define SEP " "
#define SEP_LEN strlen(SEP)
#define LINE_SEP "\r\n"
#define LINE_SEP_LEN strlen(LINE_SEP)

Request and response: Request, Response

Request: x, y, op ("x op y") x and y are data, op is an operator, such as 1+2

Response: Set the exit code exitcode and the result result()

Add a header to the request and response. The header set here is the length, enLength (that is, adding the size and converting it into a string), that is, encapsulating the enLength function:

//"x op y"->"content_len"\r\n"x op y"\r\n
//"exitcode result"->"cotent_len"\r\n"exitcode result"\r\n
const std::string enLength(const std::string &text)
{
    std::string send_string = std::to_string(text.size());
    send_string += LINE_SEP;
    send_string += text;
    send_string += LINE_SEP;
    return send_string;
}

To extract the message from the request and response, as long as the message is not required, the deLength function is encapsulated:

//"cotent_len"\r\n"exitcode result"\r\n
bool deLength(const std::string &package, std::string *text)
{
    auto pos = package.find(LINE_SEP);
    if (pos == std::string::npos)
        return false;
    std::string text_len_string = package.substr(0, pos);
    int text_len = std::stoi(text_len_string);
    *text = package.substr(pos + LINE_SEP_LEN, text_len);
    return true;
}

Serialize and deserialize requests and responses: for serialization and deserialization, we can use Json to implement

Serialization process: structured data -> "x op y"

Deserialization process: "x op y" -> structured data

Protocol.hpp also provides the recvPackage function

#define SEP " "
#define SEP_LEN strlen(SEP)
#define LINE_SEP "\r\n"
#define LINE_SEP_LEN strlen(LINE_SEP)
enum
{
    OK = 0,
    DIV_ZERO,
    MOD_ZERO,
    OP_ERROR
};

//"x op y" --->"content_len"\r\n"x op y"\r\n,添加报头
std::string enLength(const std::string &text)
{
    std::string send_string = std::to_string(text.size());
    send_string += LINE_SEP;
    send_string += text;
    send_string += LINE_SEP;
    return send_string;
}

//"content_len"\r\n"exitcode result"\r\n
// 去掉报头，得到"exitcode result"
bool deLength(const std::string &package, std::string *text)
{
    auto pos = package.find(LINE_SEP);
    if (pos == std::string::npos)
        return false;
    std::string text_len_string = package.substr(0, pos); // content_len:如“14”
    int text_len = std::stoi(text_len_string);
    *text = package.substr(pos + LINE_SEP_LEN, text_len);
    return true;
}

class Request
{
public:
    Request() : x(0), y(0), op(0)
    {
    }

    Request(int x_, int y_, char op_) : x(x_), y(y_), op(op_)
    {
    }

    // 序列化：
    // 结构化-> "x op y"
    bool serialize(std::string *out)
    {
#ifdef MYSELF
        *out = "";
        std::string x_string = std::to_string(x);
        std::string y_string = std::to_string(y);

        *out = x_string;
        *out += SEP;
        *out += op;
        *out += SEP;
        *out += y_string;
#else
        Json::Value root;
        root["first"] = x;
        root["second"] = y;
        root["oper"] = op;

    Json::FastWriter writer;
    *out = writer.write(root);
#endif
        return true;
    }

    // 反序列化化：
    //"x op y"->结构化
    bool deserialize(const std::string &in)
    {
#ifdef MYSELF
        auto left = in.find(SEP);
        auto right = in.rfind(SEP);
        if (left == std::string::npos || right == std::string::npos)
            return false;
        if (left == right)
            return false;
        if (right - (left + SEP_LEN) != 1)
            return false;
        std::string x_string = in.substr(0, left);
        std::string y_string = in.substr(right + SEP_LEN);

        if (x_string.empty())
            return false;
        if (y_string.empty())
            return false;
        x = std::stoi(x_string);
        y = std::stoi(y_string);
        op = in[left + SEP_LEN];
#else
        Json::Value root;
        Json::Reader reader;
        reader.parse(in, root);

        x = root["first"].asInt();
        y = root["second"].asInt();
        op = root["oper"].asInt();
#endif
        return true;
    }

public:
    int x;
    int y;
    char op;
};

class Response
{
public:
    Response() : exitcode(0), result(0)
    {
    }

    Response(int exitcode_, int result_) : exitcode(exitcode_), result(result_)
    {
    }

    bool serialize(std::string *out)
    {
#ifdef MYSELF
        *out = "";
        std::string ec_string = std::to_string(exitcode);
        std::string res_string = std::to_string(result);

        *out = ec_string;
        *out += SEP;
        *out += res_string;
#else
        Json::Value root;
        root["exitcode"] = exitcode;
        root["result"] = result;

        Json::FastWriter writer;
        *out = writer.write(root);
#endif
        return true;
    }

    bool deserialize(const std::string &in)
    {
#ifdef MYSELF
        auto mid = in.find(SEP);
        if (mid == std::string::npos)
            return false;
        std::string ec_string = in.substr(0, mid);
        std::string res_string = in.substr(mid + SEP_LEN);

        if (ec_string.empty() || res_string.empty())
            return false;
        exitcode = std::stoi(ec_string);
        result = std::stoi(res_string);
#else
        Json::Value root;
        Json::Reader reader;
        reader.parse(in, root);

        exitcode = root["exitcode"].asInt();
        result = root["result"].asInt();
#endif
        return true;
    }

public:
    int exitcode;
    int result;
};

// 读取报文,保证读取的是一个完整的报文 ,inbuffer由外部传入
// "content_len"\r\n"x op y"\r\n"content_len"\r\n"x op y"\r\n"content_len"\r\n"x op
bool recvPackage(int sock, std::string &inbuffer, std::string *text)
{
    char buffer[1024];
    while(true)
    {
        ssize_t n = recv(sock, buffer, sizeof(buffer) - 1, 0);
        if(n>0)
        {
            buffer[n] = 0;
            inbuffer+=buffer;
            
            auto pos = inbuffer.find(LINE_SEP);
            if(pos == std::string::npos) continue;
            std::string text_len_string = inbuffer.substr(0,pos);
            int text_len  =std::stoi(text_len_string);
            int total_len = text_len_string.size()+2*LINE_SEP_LEN+text_len;

            std::cout<<"处理前#inbuffer:\n"<<inbuffer<<std::endl;
            if(inbuffer.size()< total_len)
            {
                std::cout<<"你输入的消息,没有遵守所定制的协议,正在等待后续的内容,continue"<<std::endl;
                continue;
            }

            //至少是一个完整的报文
            *text = inbuffer.substr(0,total_len);
            inbuffer.erase(0,total_len);

            std::cout<<"处理后#inbuffer:\n"<<inbuffer<<std::endl;
            break;
        }
        else
            return false;
    }
    return true;
}

For the recvPackage function, we must ensure that at least one complete message is read

CalServer

server code

//CalServer.hpp
namespace server
{
    enum
    {
        USAGE_ERR = 1,
        SOCKET_ERR,
        BIND_ERR,
        LISTEN_ERR
    };

    static const uint16_t gport = 8080;
    static const int gbacklog = 5;
    typedef std::function<bool(const Request &req, Response &resp)> func_t;


    void handlerEntery(int sock,func_t func)
    {
        std::string inbuffer;
        while(true)
        {
            //1.读取:"content_len"\r\n"x op y"\r\n
            //保证读到的消息是一个完整的请求
            std::string req_text,req_str;
            if(!recvPackage(sock,inbuffer,&req_text)) return;
            std::cout<<"带报头的请求:\n"<<req_text<<std::endl;
            //去掉报头
            if(!deLength(req_text,&req_str)) return;
            std::cout<<"去掉报头的正文:\n"<<req_str<<std::endl;

            //2.对请求Request,反序列化
            //2.1得到一个结构化的请求对象
            Request req;
            if(!req.deserialize(req_str)) return;

            //3.计算机处理————业务逻辑
            Response resp;
            func(req,resp);

            //4.对响应Response,进行序列化
            //4.1得到一个"字符串"
            std::string resp_str;
            resp.serialize(&resp_str);

            std::cout<<"计算完成，序列化响应:"<<resp_str<<std::endl;

            //5.发送响应
            std::string send_string = enLength(resp_str);
            std::cout<<"构建完成完整的响应\n"<<send_string<<std::endl;
            send(sock,send_string.c_str(),send_string.size(),0);


        }
    }
    class CalServer
    {
    public:
        CalServer(const uint16_t&port = gport):_listensock(-1),_port(port)
        {}

        void initServer()
        {
            _listensock = socket(AF_INET,SOCK_STREAM,0);
            if(_listensock<0)
            {
                logMessage(FATAL,"create socket error");
                exit(SOCKET_ERR);
            }

            logMessage(NORMAL,"create socket success:%d",_listensock);

            struct sockaddr_in local;
            memset(&local,0,sizeof(local));
            local.sin_family = AF_INET;
            local.sin_port = htons(_port);
            local.sin_addr.s_addr = INADDR_ANY;
            if(bind(_listensock,(struct sockaddr*)&local,sizeof(local))<0)
            {
                logMessage(FATAL,"bind socket error");
                exit(BIND_ERR);
            }
            logMessage(NORMAL,"bind socket success");

            if(listen(_listensock,gbacklog)<0)
            {
                logMessage(FATAL,"listen socker error");
                exit(LISTEN_ERR);
            }
            logMessage(NORMAL,"listen socket success");
        }

        void start(func_t func)
        {
            for(;;)
            {
                struct sockaddr_in peer;
                socklen_t len = sizeof(peer);
                int sock = accept(_listensock,(struct sockaddr*)&peer,&len);
                if(sock<0)
                {
                    logMessage(ERROR,"accept error,next");
                    continue;
                }
                logMessage(NORMAL,"accept a new link success,get new sock:%d",sock);

                pid_t id = fork();
                if(id == 0)
                {
                    close(_listensock);
                    handlerEntery(sock,func);
                    close(sock);
                    exit(0);
                }
                close(sock);

                pid_t ret = waitpid(id,nullptr,0);
                if(ret>0)
                {
                    logMessage(NORMAL,"wait child success");

                }
            }
        }
         ~CalServer() {}
    public:
        int _listensock;
        uint16_t _port;
    };
}

//CalServer.cc
#include "calServer.hpp"
#include <memory>
using namespace server;
using namespace std;
static void Usage(string proc)
{
    cout << "\nUsage:\n\t" << proc << " local_port\n\n";
}

// req是处理好的完整的请求对象
// resp:根据req进行业务处理，填充resp，不需要管理任何IO，序列化和反序列化
bool cal(const Request &req, Response &resp)
{
    // req已经有结构化的数据
    resp.exitcode = OK;
    resp.result = 0;
    switch (req.op)
    {
    case '+':
        resp.result = req.x+req.y;
        break;
    case '-':
        resp.result = req.x-req.y;
        break;
    case '*':
        resp.result = req.x*req.y;
        break;
    case '/':
    {
        if(req.y==0) resp.exitcode = DIV_ZERO;
        else resp.result = req.x/req.y;
    }
    break;
    case '%':
    {
        if(req.y == 0)
            resp.exitcode = MOD_ZERO;
        else resp.result = req.x%req.y;
    }
    break;
    default:
        resp.exitcode = OP_ERROR;
        break;
    }
    return true;
}

// tcp服务器，启动上和udp server一模一样
// ./tcpserver local_port
int main(int argc, char *argv[])
{
    if (argc != 2)
    {
        Usage(argv[0]);
        exit(USAGE_ERR);
    }

    uint16_t port = atoi(argv[1]);
    unique_ptr<CalServer> tsvr(new CalServer(port));
    tsvr->initServer();
    tsvr->start(cal);
    return 0;
}

CalClient

ParseLine: parse, build a request: "1+1"

#pragma once
#include <iostream>
#include <string>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <unistd.h>
#include "Protocol.hpp"
#define NUM 1024

class CalClient
{
public:
    CalClient(const std::string &serverip, const uint16_t &serverport)
        : _sock(-1), _serverip(serverip), _serverport(serverport)
    {
    }
    void initClient()
    {
        // 1. 创建socket
        _sock = socket(AF_INET, SOCK_STREAM, 0);
        if (_sock < 0)
        {
            std::cerr << "socket create error" << std::endl;
            exit(2);
        }
    }
    void start()
    {
        struct sockaddr_in server;
        memset(&server, 0, sizeof(server));
        server.sin_family = AF_INET;
        server.sin_port = htons(_serverport);
        server.sin_addr.s_addr = inet_addr(_serverip.c_str());

        if (connect(_sock, (struct sockaddr *)&server, sizeof(server)) != 0)
        {
            std::cerr << "socket connect error" << std::endl;
        }
        else
        {
            std::string line;
            std::string inbuffer;
            while (true)
            {
                std::cout << "mycal>>> ";
                std::getline(std::cin, line);  // 1+1
                Request req = ParseLine(line); // "1+1"
                std::string content;
                req.serialize(&content);
                std::string send_string = enLength(content);
                send(_sock, send_string.c_str(), send_string.size(), 0); // bug?? 不管

                std::string package, text;
                //  "content_len"\r\n"exitcode result"\r\n
                if (!recvPackage(_sock, inbuffer, &package))
                    continue;
                if (!deLength(package, &text))
                    continue;
                // "exitcode result"
                Response resp;
                resp.deserialize(text);
                std::cout << "exitCode: " << resp.exitcode << std::endl;
                std::cout << "result: " << resp.result << std::endl;
            }
        }
    }
    Request ParseLine(const std::string &line)
    {
        // 建议版本的状态机！
        //"1+1" "123*456" "12/0"
        int status = 0; // 0:操作符之前，1:碰到了操作符 2:操作符之后
        int i = 0;
        int cnt = line.size();
        std::string left, right;
        char op;
        while (i < cnt)
        {
            switch (status)
            {
            case 0:
            {
                if(!isdigit(line[i]))
                {
                    op = line[i];
                    status = 1;
                }
                else left.push_back(line[i++]);
            }
            break;
            case 1:
                i++;
                status = 2;
                break;
            case 2:
                right.push_back(line[i++]);
                break;
            }
        }
        std::cout << std::stoi(left)<<" " << std::stoi(right) << " " << op << std::endl;
        return Request(std::stoi(left), std::stoi(right), op);
    }
    ~CalClient()
    {
        if (_sock >= 0)
            close(_sock);
    }

private:
    int _sock;
    std::string _serverip;
    uint16_t _serverport;
};

#include "calClient.hpp"
#include <memory>

using namespace std;
static void Usage(string proc)
{
    cout<<"\nUasge:\n\t"<<proc<<" serverip serverport\n\n";
}
int main(int argc,char*argv[])
{
    if(argc!=3)
    {
        Usage(argv[0]);
        exit(1);
    }
    string serverip = argv[1];
    uint16_t serverport = atoi(argv[2]);

    unique_ptr<CalClient> tcli(new CalClient(serverip,serverport));
    tcli->initClient();
    tcli->start();
    return 0;
}

Json installation

sudo yum install -y jsoncpp-devel