content
The relationship between websocket and http
The disadvantages of http lead to why websocket is needed
After the TCP connection is completed, the meaning of a handshake
Detailed analysis: How to distinguish between handshake data and normal interaction data?
Handshake details core: Sec-WebSocket-Key ---> Sec-WebSocket-Accept
The details of transform data push --- data packing and unpacking.
Online Test Tool + My Test Results
Preface.
- Since the implementation of websocket in this article is based on reactor, I need to use part of the code I wrote before for reactor implementation. If you are not familiar with reactor, you can go to Kangkang
The relationship between websocket and http
The disadvantages of http lead to why websocket is needed
- HTTP is a stateless , connectionless, non-persistent one - way half-duplex application layer protocol
- What is stateless, there is no memory for historical connections, each connection is a new connection
- Connectionless and non-persistent actually mean the same thing, a request, a response, and will not last.
- One-way half-duplex means that the communication request can only be initiated by the client, the server can only respond to the request , and the server cannot actively send data to the client
Causes the problem --- the server cannot actively send data to the client , if there is a certain state change on the server, it cannot actively push the data to the client in real time
- In response to the above problems, http is still used at the beginning, but the problem that the server needs to actively send data to the client is solved by means of timed polling and long polling .
- Timed polling, constantly and regularly ask the server, do you need to send a message, keep requesting, regularly send out inquiries, if the server needs to send data to the client, you can send the status change and the data to the client when the inquiry comes
- Disadvantages of regular polling: there may be a serious delay, and continuous polling and wasteful occupation of server-side resources increase the pressure on the server-side, and constantly establish connections and waste resources . There are many invalid requests . The establishment of a connection consumes a lot of my bandwidth and resources. I need to process the connection quickly, and many times I have no data to update, and there are a lot of invalid requests ( the server said that I am passive )
Learn from a brother's life case, easy to understand: [WebSocket Protocol] The next evolution of Web communication I want to illustrate this with some simple and obvious examples. As part of the HTML5 plan, the developed WebSocket specification introduced the WebSocket https://blog.csdn.net/qq_41103843/article/details/124116838?utm_source=app&app_version=5.3.0&code=app_1562916241&uLinkId=usr1mkqgl919blen The front-end websocket written by the above-mentioned boss Implement a web chat room, and the explanation of websocket is also quite good. My takeaway examples are all based on it.
We usually order takeaway (polling example)
0 seconds: Has the food arrived? (Customer)
0 seconds: Shipping is in progress. (Takeaway brother)
1 second: Has the food arrived? (Customer)
1 second: Shipping is in progress. (Takeaway brother)
2 seconds: Has the food arrived? (Customer)
2 seconds: Delivering. (Takeaway brother) 2 seconds and so many inquiries are invalid. Inquiry
3 seconds: Has the food arrived? (Customer)
3 seconds: Yes, sir, this is your takeaway. (takeaway brother)
- Upgrading to long polling can only solve the problem of delay, and can achieve the purpose of pushing the status data of the server to the client in real time, but the http connection is always open, long connection, waste of system resources , the client needs Waiting for the server to respond, the server is always occupied by the client. ( The server is always occupied, invalid occupation )
Long polling life example
0 seconds: Has the food arrived? (client)
. . . The middle call keeps on hanging up until the takeaway is delivered in
3 seconds: yes, sir, this is your takeaway. (takeaway brother)
Solve the problem --- The websocket full-duplex communication protocol was born, and the server can actively send data to the client
- The stage is divided into a handshake stage + data exchange stage
- Main core: full duplex, the server can actively send data to the client
Features of websocket
- Based on the TCP protocol, the server-side implementation is relatively easy
- It has good compatibility with the HTTP protocol, the default ports are also 80 and 443, and the handshake phase is based on the HTTP protocol
- The data format is relatively lightweight, the performance overhead is small, and the communication is efficient
- Can send text or binary data
Packet Analysis
GET /chat HTTP/1.1 #请求行
Host: server.example.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Origin: http://example.com
Sec-WebSocket-Protocol: chat, superchat
Sec-WebSocket-Version: 13
#请求头部
- Note: The upgrade to websocket in the above box actually informs the server that the client wants to establish a websocket connection. Do you support it? If the server supports it, there must be two header fields returned in the response message.
- response
HTTP/1.1 101 Switching Protocols #响应行,状态行
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=
Sec-WebSocket-Protocol: chat
# 响应报头
- Sec-WebSocket-Accept is actually the result returned after hashing based on the client's Key
-
The example scenario of websocket in our life (server ( backend ) refreshes data in real time to web client ( frontend ))
- Real-time refresh of the barrage
- Page jump after scanning WeChat QR code
- Real-time refresh of stock data
Analysis of the implementation of the websocket protocol in blocks, how to encapsulate the websocket application layer protocol on the basis of reactor (which protocols are encapsulated and implemented exactly)
-
Process analysis
-
Handshake Details:
-
After the TCP connection is completed, the meaning of a handshake
- Handshake: Make sure that the server supports the websocket protocol, that is, the server responds to the client, I have received your upgrade request, and I support the websocket upgrade
Detailed analysis: How to distinguish between handshake data and normal interaction data?
- In fact, the core is to distinguish between different stages, states --- state machines, to distinguish each state
- State machine --- the bottom layer of the http protocol also exists, and the protocol encapsulation is an essential part, because it is necessary to distinguish different stages, whether it is the stage of handshake establishment of connection, or the stage of data interaction... These http protocols must be distinguished at the bottom layer . of
- Each connection has the following three states
enum WEBSOCKET_STATUS {
WS_HANDSHARK,//握手状态
WS_DATATRANSFORM,//数据交互状态
WS_DATAEND,//断开状态
};Handshake details core: Sec-WebSocket-Key ---> Sec-WebSocket-Accept
- Why do you need to go through layers of encryption key -> Accept key ?
- To be on the safe side, to prove they can handle websocket requests. Key authentication.
- The pseudo code of the encryption process is as follows: Get accept.
//伪代码如下
str = Sec-WebSocket-Key;
//拿出Sec-WebSocket-Key客户端序列串
str += GUID;
sha = SHA-1(str); //SHA-1一种hash
accept = base64_encode(sha);
-
The details of transform data push --- data packing and unpacking.
- The following is the unique data frame format of websocket. The data transmission after the handshake needs to process the data to be sent according to the format of the data frame.
-
There are three necessary parts to make a custom protocol. A custom application layer protocol based on tcp
- Opcode. eg: FIN RSV
- package length
- mask-key
- Data encapsulation function + data unpacking function (I haven't fully understood the specific encapsulation and unpacking process. If there is a chance later, Xiaojie hopes to re-analyze it. The following is the code of the predecessors for reference)
void umask(char *data,int len,char *mask) {
int i;
for (i = 0;i < len;i ++)
*(data+i) ^= *(mask+(i%4));
}
char* decode_packet(char *stream, char *mask, int length, int *ret) {
nty_ophdr *hdr = (nty_ophdr*)stream;
unsigned char *data = stream + sizeof(nty_ophdr);
int size = 0;
int start = 0;
//char mask[4] = {0};
int i = 0;
//if (hdr->fin == 1) return NULL;
if ((hdr->mask & 0x7F) == 126) {
nty_websocket_head_126 *hdr126 = (nty_websocket_head_126*)data;
size = hdr126->payload_length;
for (i = 0;i < 4;i ++) {
mask[i] = hdr126->mask_key[i];
}
start = 8;
} else if ((hdr->mask & 0x7F) == 127) {
nty_websocket_head_127 *hdr127 = (nty_websocket_head_127*)data;
size = hdr127->payload_length;
for (i = 0;i < 4;i ++) {
mask[i] = hdr127->mask_key[i];
}
start = 14;
} else {
size = hdr->payload_length;
memcpy(mask, data, 4);
start = 6;
}
*ret = size;
umask(stream+start, size, mask);
return stream + start;
}
int encode_packet(char *buffer,char *mask, char *stream, int length) {
nty_ophdr head = {0};
head.fin = 1;
head.opcode = 1;
int size = 0;
if (length < 126) {
head.payload_length = length;
memcpy(buffer, &head, sizeof(nty_ophdr));
size = 2;
} else if (length < 0xffff) {
nty_websocket_head_126 hdr = {0};
hdr.payload_length = length;
memcpy(hdr.mask_key, mask, 4);
memcpy(buffer, &head, sizeof(nty_ophdr));
memcpy(buffer+sizeof(nty_ophdr), &hdr, sizeof(nty_websocket_head_126));
size = sizeof(nty_websocket_head_126);
} else {
nty_websocket_head_127 hdr = {0};
hdr.payload_length = length;
memcpy(hdr.mask_key, mask, 4);
memcpy(buffer, &head, sizeof(nty_ophdr));
memcpy(buffer+sizeof(nty_ophdr), &hdr, sizeof(nty_websocket_head_127));
size = sizeof(nty_websocket_head_127);
}
memcpy(buffer+2, stream, length);
return length + 2;
}
- Note again that after recv data, you need to call different functions according to different states to process the data
- Overall code:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <netinet/tcp.h>
#include <arpa/inet.h>
#include <pthread.h>
#include <fcntl.h>
#include <errno.h>
#include <sys/epoll.h>
#include <openssl/sha.h>
#include <openssl/pem.h>
#include <openssl/bio.h>
#include <openssl/evp.h>
typedef struct sockaddr SA;
#define BUFFSIZE 1024
#define GUID "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"
enum WEBSOCKET_STATUS {
WS_HANDSHARK,//握手状态
WS_DATATRANSFORM,//数据交互状态
WS_DATAEND,//断开状态
};
struct sockitem {
int sockfd;
int (*callback)(int fd, int events, void* arg);
//arg 传入 sockitem*
char recvbuffer[BUFFSIZE];
char sendbuffer[BUFFSIZE];
int rlen;//recvlen
int slen;//sendlen
int status;//存储状态
};
//mainloop / eventloop
struct reactor {
int epfd;
struct epoll_event events[512];
};
struct reactor* eventloop = NULL; //事件循环
int recv_cb(int fd, int events, void* arg);
int send_cb(int fd, int events, void* arg);
// websocket
char* decode_packet(char *stream, char *mask, int length, int *ret);
int encode_packet(char *buffer,char *mask, char *stream, int length);
struct _nty_ophdr {
unsigned char opcode:4,
rsv3:1,
rsv2:1,
rsv1:1,
fin:1;
unsigned char payload_length:7,
mask:1;
} __attribute__ ((packed));
struct _nty_websocket_head_126 {
unsigned short payload_length;
char mask_key[4];
unsigned char data[8];
} __attribute__ ((packed));
struct _nty_websocket_head_127 {
unsigned long long payload_length;
char mask_key[4];
unsigned char data[8];
} __attribute__ ((packed));
typedef struct _nty_websocket_head_127 nty_websocket_head_127;
typedef struct _nty_websocket_head_126 nty_websocket_head_126;
typedef struct _nty_ophdr nty_ophdr;
int base64_encode(char *in_str, int in_len, char *out_str) {
BIO *b64, *bio;
BUF_MEM *bptr = NULL;
size_t size = 0;
if (in_str == NULL || out_str == NULL)
return -1;
b64 = BIO_new(BIO_f_base64());
bio = BIO_new(BIO_s_mem());
bio = BIO_push(b64, bio);
BIO_write(bio, in_str, in_len);
BIO_flush(bio);
BIO_get_mem_ptr(bio, &bptr);
memcpy(out_str, bptr->data, bptr->length);
out_str[bptr->length-1] = '\0';
size = bptr->length;
BIO_free_all(bio);
return size;
}
//读取一行, allbuff整个缓冲区, level 当前ind linebuff 存储一行
int readline(char* allbuff, int level, char* linebuff ) {
int n = strlen(allbuff);
for (; level < n; ++level) {
//\r\n 回车换行, 表示行末
if (allbuff[level] == '\r' && allbuff[level + 1] == '\n') {
return level + 2;
} else {
*(linebuff++) = allbuff[level]; //存储行数据
}
}
return -1;
}
//握手,
int handshark(struct sockitem* si, struct reactor* mainloop) {
char linebuff[256];//存储一行
char sec_accept[32];//存储进行处理之后的子序列
unsigned char sha1_data[SHA_DIGEST_LENGTH + 1] = {0};
char head[BUFFSIZE] = {0};//存储整个头部信息
int level = 0;
//读取Sec-WebSocket-Key并且处理获取accept-key返回密匙
do {
memset(linebuff, 0, sizeof(linebuff));//清空
level = readline(si->recvbuffer, level, linebuff);
if (strstr(linebuff, "Sec-WebSocket-Key") != NULL) {
//说明是key 值, 需要进行加密处理
strcat(linebuff, GUID);//str += GDID
SHA1((unsigned char*)&linebuff + 19, strlen(linebuff + 19), (unsigned char*)&sha1_data);
//SHA1(str);
base64_encode(sha1_data, strlen(sha1_data), sec_accept);
//将数据全部放入到head中, 写响应信息
sprintf(head, "HTTP/1.1 101 Switching Protocols\r\n"\
"Upgrade: websocket\r\n" \
"Connection: Upgrade\r\n" \
"Sec-WebSocket-Accept: %s\r\n" \
"\r\n", sec_accept);
printf("response\n");
printf("%s\n\n\n", head);
//然后进行将其加入到reactor中
memset(si->recvbuffer, 0, BUFFSIZE);
memcpy(si->sendbuffer, head, strlen(head));//to send
si->slen = strlen(head);
//to set epollout events;
struct epoll_event ev;
ev.events = EPOLLOUT | EPOLLET;
//si->sockfd = si->sockfd;
si->callback = send_cb;
//握手完成 --》 状态数据交互
si->status = WS_DATATRANSFORM;
ev.data.ptr = si;
epoll_ctl(mainloop->epfd, EPOLL_CTL_MOD, si->sockfd, &ev);
//握手之后接下来server 需要关注send数据
break;
}
} while ((si->recvbuffer[level] != '\r' || si->recvbuffer[level + 1] != '\n') && level != -1);
return 0;
}
//数据交互函数
int transform(struct sockitem *si, struct reactor *mainloop) {
int ret = 0;
char mask[4] = {0};
char *data = decode_packet(si->recvbuffer, mask, si->rlen, &ret);
printf("data : %s , length : %d\n", data, ret);
ret = encode_packet(si->sendbuffer, mask, data, ret);
si->slen = ret;
memset(si->recvbuffer, 0, BUFFSIZE);
struct epoll_event ev;
ev.events = EPOLLOUT | EPOLLET;
//ev.data.fd = clientfd;
si->sockfd = si->sockfd;
si->callback = send_cb;
si->status = WS_DATATRANSFORM;//标识IO事件处于数据交互状态.
ev.data.ptr = si;
epoll_ctl(mainloop->epfd, EPOLL_CTL_MOD, si->sockfd, &ev);
return 0;
}
void umask(char *data,int len,char *mask) {
int i;
for (i = 0;i < len;i ++)
*(data+i) ^= *(mask+(i%4));
}
char* decode_packet(char *stream, char *mask, int length, int *ret) {
nty_ophdr *hdr = (nty_ophdr*)stream;
unsigned char *data = stream + sizeof(nty_ophdr);
int size = 0;
int start = 0;
//char mask[4] = {0};
int i = 0;
//if (hdr->fin == 1) return NULL;
if ((hdr->mask & 0x7F) == 126) {
nty_websocket_head_126 *hdr126 = (nty_websocket_head_126*)data;
size = hdr126->payload_length;
for (i = 0;i < 4;i ++) {
mask[i] = hdr126->mask_key[i];
}
start = 8;
} else if ((hdr->mask & 0x7F) == 127) {
nty_websocket_head_127 *hdr127 = (nty_websocket_head_127*)data;
size = hdr127->payload_length;
for (i = 0;i < 4;i ++) {
mask[i] = hdr127->mask_key[i];
}
start = 14;
} else {
size = hdr->payload_length;
memcpy(mask, data, 4);
start = 6;
}
*ret = size;
umask(stream+start, size, mask);
return stream + start;
}
int encode_packet(char *buffer,char *mask, char *stream, int length) {
nty_ophdr head = {0};
head.fin = 1;
head.opcode = 1;
int size = 0;
if (length < 126) {
head.payload_length = length;
memcpy(buffer, &head, sizeof(nty_ophdr));
size = 2;
} else if (length < 0xffff) {
nty_websocket_head_126 hdr = {0};
hdr.payload_length = length;
memcpy(hdr.mask_key, mask, 4);
memcpy(buffer, &head, sizeof(nty_ophdr));
memcpy(buffer+sizeof(nty_ophdr), &hdr, sizeof(nty_websocket_head_126));
size = sizeof(nty_websocket_head_126);
} else {
nty_websocket_head_127 hdr = {0};
hdr.payload_length = length;
memcpy(hdr.mask_key, mask, 4);
memcpy(buffer, &head, sizeof(nty_ophdr));
memcpy(buffer+sizeof(nty_ophdr), &hdr, sizeof(nty_websocket_head_127));
size = sizeof(nty_websocket_head_127);
}
memcpy(buffer+2, stream, length);
return length + 2;
}
//设置非阻塞
static int set_nonblock(int fd) {
int flags;
flags = fcntl(fd, F_GETFL, 0);
if (flags < 0) return -1;
if (fcntl(fd, F_SETFL, flags | O_NONBLOCK) < 0) return -1;
return 0;
}
int send_cb(int fd, int events, void* arg) {
//发送sendbuffer中的数据
struct sockitem* si = (struct sockitem*)arg;
send(fd, si->sendbuffer, si->slen, 0);
//设置关注读事件, 写完这批交互数据，接下来该继续读了
struct epoll_event ev;
ev.events = EPOLLIN | EPOLLET;
si->sockfd = fd;//从新设置sockfd
si->callback = recv_cb;
ev.data.ptr = si;
memset(si->sendbuffer, 0, BUFFSIZE);
//发送完数据从新将缓冲区置为0
epoll_ctl(eventloop->epfd, EPOLL_CTL_MOD, fd, &ev);
}
//关闭连接
int close_connection(struct sockitem* si, unsigned int event) {
struct epoll_event ev;
close(si->sockfd);//关闭连接
//将关注IO事件结点从监视红黑树中删除
ev.events = event;
epoll_ctl(eventloop->epfd, EPOLL_CTL_DEL, si->sockfd, &ev);
free(si);
return 0;
}
//处理读取数据
int recv_cb(int fd, int events, void* arg) {
struct sockitem* si = (struct sockitem*)arg;
struct epoll_event ev;
int ret = recv(fd, si->recvbuffer, BUFFSIZE, 0);
if (ret < 0) {
if (errno == EAGAIN || errno == EWOULDBLOCK) {
return -1;//非阻塞表示缓冲区中没有数据
} else {
}
close_connection(si, EPOLLIN);
} else if (ret == 0) {
printf("disconnect %d\n", fd);
close_connection(si, EPOLLIN);
} else {
si->rlen = 0;//重置rlen
if (si->status == WS_HANDSHARK) {
//说明是请求握手数据
printf("request\n");
printf("%s\n", si->recvbuffer);
handshark(si, eventloop);//完成握手
} else if (si->status == WS_DATATRANSFORM) {
transform(si, eventloop);
} else if (si->status == WS_DATAEND) {
close_connection(si, EPOLLOUT | EPOLLET);
}
}
}
int accept_cb(int fd, int events, void* arg) {
//处理新的连接。 连接IO事件处理流程
struct sockaddr_in cli_addr;
memset(&cli_addr, 0, sizeof(cli_addr));
socklen_t cli_len = sizeof(cli_addr);
int cli_fd = accept(fd, (SA*)&cli_addr, &cli_len);
if (cli_fd <= 0) return -1;
char cli_ip[INET_ADDRSTRLEN] = {0}; //存储cli_ip
printf("recv from ip %s at port %d\n", inet_ntop(AF_INET, &cli_addr.sin_addr, cli_ip, sizeof(cli_ip)),
ntohs(cli_addr.sin_port));
//注册接下来的读事件处理器
struct epoll_event ev;
ev.events = EPOLLIN | EPOLLET;
struct sockitem* si = (struct sockitem*)malloc(sizeof(struct sockitem));
si->sockfd = cli_fd;
si->status = WS_HANDSHARK;//等待握手的状态
si->callback = recv_cb;//设置事件处理器
ev.data.ptr = si;
epoll_ctl(eventloop->epfd, EPOLL_CTL_ADD, cli_fd, &ev);
return cli_fd;
}
int main(int argc, char* argv[]) {
if (argc != 2) {
fprintf(stderr, "usage %s <port>", argv[0]);
return -1;
}
int port = atoi(argv[1]);
int sockfd = socket(AF_INET, SOCK_STREAM, 0);
if (sockfd == -1) {
fprintf(stderr, "socket error");
return -2;
}
set_nonblock(sockfd);
struct sockaddr_in serv_addr;
memset(&serv_addr, 0, sizeof(serv_addr));
serv_addr.sin_family = AF_INET;
serv_addr.sin_addr.s_addr = INADDR_ANY;
serv_addr.sin_port = htons(port);
if (bind(sockfd, (SA*)&serv_addr, sizeof(serv_addr)) == -1) {
fprintf(stderr, "bind error");
return -3;
}
if (listen(sockfd, 5) == -1) {
fprintf(stderr, "listen error");
return -4;
}
//init eventloop
eventloop = (struct reactor*)malloc(sizeof(struct reactor));
//创建监视事件红黑树的根部
eventloop->epfd = epoll_create(1);
//注册处理连接IO处理函数
struct epoll_event ev;
ev.events = EPOLLIN;
struct sockitem* si = (struct sockitem*)malloc(sizeof(struct sockitem));
si->sockfd = sockfd;
si->callback = accept_cb;
ev.data.ptr = si;
epoll_ctl(eventloop->epfd, EPOLL_CTL_ADD, sockfd, &ev);
while (1) {
int nready = epoll_wait(eventloop->epfd, eventloop->events, 512, -1);
if (nready < -1) {
break;
}
int i = 0;
for (; i < nready; ++i) {
if (eventloop->events[i].events & EPOLLIN) {
struct sockitem *si = (struct sockitem*)eventloop->events[i].data.ptr;
si->callback(si->sockfd, eventloop->events[i].events, si);
}
if (eventloop->events[i].events & EPOLLOUT) {
struct sockitem *si = (struct sockitem*)eventloop->events[i].data.ptr;
si->callback(si->sockfd, eventloop->events[i].events, si);
}
}
}
return 0;
}-
Online Test Tool + My Test Results
websocket online test WebSocket online test tool Internet of Things 
http://www.websocket-test.com/
Summarize this article
- This article starts with the disadvantages of http and analyzes why websocket, a new application layer protocol, is needed.
- In order to solve the problem that the server needs to actively push data to the client. The back-end server actively pushes data to the front-end web page.
- Although http polling for long connections is also possible, the polling delay is long, and invalid connections are continuously established. As a result, the server does not need to push data at all, which is a waste of resources. Although long polling solves the delay problem, it continues to Occupying the server is also a waste of server resources. After all, you occupy the server but only need to push data once for a long event
- So the new server can actively push data, and the tcp-based full-duplex websocket was born
- Websocket is divided into two stages: handshake and data interaction. The handshake phase is based on http upgrades.
- In order to distinguish the data phase during recv, the state machine was born
- The core of the handshake phase is that the key confirms whether the server supports websocket. key----> accept
- After str += GUID SHA-1(str) hash then base64_encode (str);
- Then we encapsulate our own application layer protocol based on tcp if we need to:
Unique data frame format: 1. Opcode 2. Packet length 3, mask-key