Article directory
Study link
Code in the example package in the netty project on GitHub
Ruan Yifeng WebSocket Tutorial
WebSocket protocol: from beginner to proficient in 5 minutes
http
Server
NettyHttpServer
You can refer to:The code in the example package in the netty project on GitHub
import io.netty.bootstrap.ServerBootstrap;
import io.netty.channel.ChannelFuture;
import io.netty.channel.ChannelInitializer;
import io.netty.channel.EventLoopGroup;
import io.netty.channel.nio.NioEventLoopGroup;
import io.netty.channel.socket.SocketChannel;
import io.netty.channel.socket.nio.NioServerSocketChannel;
import io.netty.handler.codec.http.HttpObjectAggregator;
import io.netty.handler.codec.http.HttpRequestDecoder;
import io.netty.handler.codec.http.HttpResponseEncoder;
public class NettyHttpServer {
public static void main(String[] args) throws InterruptedException {
EventLoopGroup bossGroup = new NioEventLoopGroup();
EventLoopGroup workerGroup = new NioEventLoopGroup(16);
try {
ServerBootstrap serverBootstrap = new ServerBootstrap();
serverBootstrap.group(bossGroup, workerGroup)
.channel(NioServerSocketChannel.class)
.childHandler(new ChannelInitializer<SocketChannel>() {
@Override
protected void initChannel(SocketChannel ch) throws Exception {
ch.pipeline().addLast("http-decoder", new HttpRequestDecoder());
ch.pipeline().addLast("http-encoder", new HttpResponseEncoder());
ch.pipeline().addLast("aggregator", new HttpObjectAggregator(65536));
ch.pipeline().addLast("serverHandler", new HelloWorldServerHandler());
}
});
ChannelFuture channelFuture = serverBootstrap.bind(8080).sync();
channelFuture.channel().closeFuture().sync();
} finally {
workerGroup.shutdownGracefully();
bossGroup.shutdownGracefully();
}
}
}
HelloWorldServerHandler
import io.netty.buffer.Unpooled;
import io.netty.channel.ChannelFuture;
import io.netty.channel.ChannelFutureListener;
import io.netty.channel.ChannelHandlerContext;
import io.netty.channel.SimpleChannelInboundHandler;
import io.netty.handler.codec.http.*;
import lombok.extern.slf4j.Slf4j;
import static io.netty.handler.codec.http.HttpHeaderNames.CONNECTION;
import static io.netty.handler.codec.http.HttpHeaderNames.CONTENT_LENGTH;
import static io.netty.handler.codec.http.HttpHeaderNames.CONTENT_TYPE;
import static io.netty.handler.codec.http.HttpHeaderValues.CLOSE;
import static io.netty.handler.codec.http.HttpHeaderValues.KEEP_ALIVE;
import static io.netty.handler.codec.http.HttpHeaderValues.TEXT_PLAIN;
import static io.netty.handler.codec.http.HttpResponseStatus.OK;
@Slf4j
public class HelloWorldServerHandler extends SimpleChannelInboundHandler<HttpObject> {
private static final byte[] CONTENT = {
'H', 'e', 'l', 'l', 'o', ' ', 'W', 'o', 'r', 'l', 'd' };
@Override
protected void channelRead0(ChannelHandlerContext ctx, HttpObject msg) throws Exception {
log.info("来了Http消息了");
if (msg instanceof HttpRequest) {
HttpRequest req = (FullHttpRequest) msg;
boolean keepAlive = HttpUtil.isKeepAlive(req);
FullHttpResponse response = new DefaultFullHttpResponse(req.protocolVersion(), OK,
Unpooled.wrappedBuffer(CONTENT));
response.headers()
.set(CONTENT_TYPE, TEXT_PLAIN)
.setInt(CONTENT_LENGTH, response.content().readableBytes());
if (keepAlive) {
if (!req.protocolVersion().isKeepAliveDefault()) {
response.headers().set(CONNECTION, KEEP_ALIVE);
}
} else {
// Tell the client we're going to close the connection.
response.headers().set(CONNECTION, CLOSE);
}
ChannelFuture f = ctx.writeAndFlush(response);
if (!keepAlive) {
f.addListener(ChannelFutureListener.CLOSE);
}
}
}
@Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
log.info("active===>");
}
@Override
public void channelRegistered(ChannelHandlerContext ctx) throws Exception {
log.info("register===>");
}
@Override
public void channelInactive(ChannelHandlerContext ctx) throws Exception {
log.info("断开连接===>");
}
@Override
public void channelUnregistered(ChannelHandlerContext ctx) throws Exception {
log.info("取消注册===>");
}
}
client
ApiPost
Use the ApiPost interface test tool to send the request, and the test is as follows
Server log output
register===>
active===>
来了Http消息了
断开连接===>
取消注册===>
websocket
Preliminary understanding
Why you need WebSocket
In the http protocol, the client sends a request to the server, and the server returns the query results. HTTP 协议做不到服务器主动向客户端推送信息.
The characteristics of this one-way request are destined to make it very troublesome for the client to know if the server has continuous status changes. We can only use "polling": every once in a while, a query is issued to find out whether the server has new information. The most typical scenario is a chat room.
Polling is inefficient and wastes resources (because you must keep connecting, or the HTTP connection is always open). Therefore, engineers have been thinking about whether there is a better way. This is how WebSocket was invented.
Introduction
The WebSocket protocol was born in 2008 and became an international standard in 2011. All browsers already support it.
Its biggest feature is that,服务器可以主动向客户端推送信息，客户端也可以主动向服务器发送信息, it is a true two-way equal dialogue, which is a type of server push technology.
WebSocket与http协议一样都是基于TCP的, so they are all reliable protocols. The called send function of WebSocket is ultimately transmitted through the TCP system interface in the implementation.
Other features include:
(1) Built on the TCP protocol, server-side implementation is relatively easy.
(2) It has good compatibility with HTTP protocol. The default ports are also 80 and 443, and the HTTP protocol is used in the handshake phase, so it is not easy to block during the handshake and can pass various HTTP proxy servers.
(3) The data format is relatively lightweight, has low performance overhead, and efficient communication.
(4) You can send text or binary data.
(5) There is no origin restriction and the client can communicate with any server.
(6) The protocol identifier is ws (or wss if encrypted), and the server address is the URL.
WebSocket client for browsers
Simple example of client
Usage of WebSocket is quite simple.
var ws = new WebSocket("wss://echo.websocket.org");
ws.onopen = function(evt) {
console.log("Connection open ...");
ws.send("Hello WebSockets!");
};
ws.onmessage = function(evt) {
console.log( "Received Message: " + evt.data);
ws.close();
};
ws.onclose = function(evt) {
console.log("Connection closed.");
};
Client API
WebSocket constructor
The WebSocket object serves as a constructor for creating new WebSocket instances.
var ws = new WebSocket('ws://localhost:8080');
After executing the above statement, the client will connect to the server.
For a list of all properties and methods of instance objects, see mozilla-WebSocket Introduction.
webSocket.readyState
The readyState property returns the current state of the instance object (read-only), and there are four types.
- CONNECTING: The value is 0, indicating that it is connecting.
- OPEN: The value is 1, indicating that the connection is successful and communication is possible.
- CLOSING: A value of 2 indicates that the connection is closing.
- CLOSED: A value of 3 indicates that the connection has been closed or failed to open the connection.
Here's an example.
switch (ws.readyState) {
case WebSocket.CONNECTING:
// do something
break;
case WebSocket.OPEN:
// do something
break;
case WebSocket.CLOSING:
// do something
break;
case WebSocket.CLOSED:
// do something
break;
default:
// this never happens
break;
}
webSocket.onopen
The onopen attribute of the instance object is used to specify the callback function after the connection is successful.
ws.onopen = function () {
ws.send('Hello Server!');
}
If you want to specify multiple callback functions, you can use the addEventListener method.
ws.addEventListener('open', function (event) {
ws.send('Hello Server!');
});
webSocket.onclose
The onclose attribute of the instance object is used to specify the callback function after the connection is closed.
ws.onclose = function(event) {
var code = event.code;
var reason = event.reason;
var wasClean = event.wasClean;
// handle close event
};
ws.addEventListener("close", function(event) {
var code = event.code;
var reason = event.reason;
var wasClean = event.wasClean;
// handle close event
});
webSocket.onerror
The onerror attribute of the instance object is used to specify the callback function when an error is reported.
socket.onerror = function(event) {
// handle error event
};
socket.addEventListener("error", function(event) {
// handle error event
});
webSocket.onmessage
The onmessage attribute of the instance object is used to specify the callback function after receiving server data.
ws.onmessage = function(event) {
var data = event.data;
// 处理数据
};
ws.addEventListener("message", function(event) {
var data = event.data;
// 处理数据
});
Note that the server data may be text,也可能是二进制数据 (blob object or Arraybuffer object).
ws.onmessage = function(event){
if(typeof event.data === String) {
console.log("Received data string");
}
if(event.data instanceof ArrayBuffer){
var buffer = event.data;
console.log("Received arraybuffer");
}
}
In addition to dynamically determining the received data type, also可以使用binaryType属性，显式指定收到的二进制数据类型.
// 收到的是 blob 数据
ws.binaryType = "blob";
ws.onmessage = function(e) {
console.log(e.data.size);
};
// 收到的是 ArrayBuffer 数据
ws.binaryType = "arraybuffer";
ws.onmessage = function(e) {
console.log(e.data.byteLength);
};
webSocket.send()
The send() method of the instance object is used to send data to the server.
Example of sending a text.
ws.send('your message');
Example of sending a Blob object.
var file = document.querySelector('input[type="file"]').files[0];
ws.send(file);
Example of sending an ArrayBuffer object.
// Sending canvas ImageData as ArrayBuffer
var img = canvas_context.getImageData(0, 0, 400, 320);
var binary = new Uint8Array(img.data.length);
for (var i = 0; i < img.data.length; i++) {
binary[i] = img.data[i];
}
ws.send(binary.buffer);
webSocket.bufferedAmount
The bufferedAmount property of the instance object indicates how many bytes of binary data are yet to be sent. It can be used to determine whether the sending is completed.
var data = new ArrayBuffer(10000000);
socket.send(data);
if (socket.bufferedAmount === 0) {
// 发送完毕
} else {
// 发送还没结束
}
interactive process
Build environment
NettyWsServer
@Slf4j
public class NettyWsServer {
public static void main(String[] args) throws InterruptedException {
EventLoopGroup bossGroup = new NioEventLoopGroup();
EventLoopGroup workerGroup = new NioEventLoopGroup(16);
try {
ServerBootstrap serverBootstrap = new ServerBootstrap();
serverBootstrap.group(bossGroup, workerGroup)
.channel(NioServerSocketChannel.class)
.childHandler(new ChannelInitializer<SocketChannel>() {
@Override
protected void initChannel(SocketChannel ch) throws Exception {
ch.pipeline().addLast("http-decoder", new HttpRequestDecoder());
ch.pipeline().addLast("http-encoder", new HttpResponseEncoder());
ch.pipeline().addLast("aggregator", new HttpObjectAggregator(655360));
WebSocketServerProtocolConfig wsServerConfig = WebSocketServerProtocolConfig
.newBuilder()
.websocketPath("/websocket")
.maxFramePayloadLength(Integer.MAX_VALUE)
.checkStartsWith(true).build();
ch.pipeline().addLast("websocketHandler", new WebSocketServerProtocolHandler(wsServerConfig));
ch.pipeline().addLast("wsTextHandler", new WsTextHandler());
}
});
ChannelFuture channelFuture = serverBootstrap.bind(8080).sync();
log.info("=========ws服务器启动成功==========");
channelFuture.channel().closeFuture().sync();
} finally {
workerGroup.shutdownGracefully();
bossGroup.shutdownGracefully();
}
}
}
WsTextHandler
@Slf4j
public class WsTextHandler extends SimpleChannelInboundHandler<TextWebSocketFrame> {
@Override
protected void channelRead0(ChannelHandlerContext ctx, TextWebSocketFrame msg) throws Exception {
log.info("收到Ws客户端消息: {}", msg.text());
}
}
index.html
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Document</title>
</head>
<body>
发送内容: <input type="text" id="content">
<button id="sendBtn">发送</button>
</body>
<script>
var ws = new WebSocket('ws://127.0.0.1:8080/websocket')
ws.onopen = function(evt) {
console.log('ws连接建立');
}
ws.onclose = function(evt) {
console.log('ws连接断开');
}
ws.onerror = function(evt) {
console.log('ws连接发生错误');
}
ws.onmessage = function(msg) {
console.log('收到消息: ' + JSON.stringify(msg));
}
const contentIpt = document.querySelector('#content')
const sendBtn = document.querySelector('#sendBtn')
sendBtn.addEventListener('click', function() {
console.log(contentIpt.value);
ws.send(contentIpt.value)
})
</script>
</html>
Postman tests websocket connection
You can also use vscocde to use live server to directly start index.html or use postman to test as follows
Connection establishment process (handshake)
As mentioned earlier, WebSocket reuses the HTTP handshake channel. Specifically, it means that the client negotiates the upgrade protocol with the WebSocket server through an HTTP request. After the protocol upgrade is completed, subsequent data exchange will follow the WebSocket protocol.
Both the Http protocol and the WebSocket protocol are built on the Tcp connection. The Tcp connection itself supports two-way communication, but the WebSocket handshake process requires the help of Http at this stage. Once the connection is established, the data is processed according to the data frame defined by the WebSocket protocol. Interaction.
1. Client: Apply for protocol upgrade
First, the client initiates a protocol upgrade request. As you can see, the standard HTTP message format is used and only the GET method is supported.
GET / HTTP/1.1
Host: localhost:8080
Origin: http://127.0.0.1:3000
Connection: Upgrade
Upgrade: websocket
Sec-WebSocket-Version: 13
Sec-WebSocket-Key: w4v7O6xFTi36lq3RNcgctw==
The key request headers have the following meanings:
Connection: Upgrade: Indicates that the protocol needs to be upgradedUpgrade: websocket: Indicates that you want to upgrade to the websocket protocol.Sec-WebSocket-Version: 13: Indicates the websocket version. If the server does not support this version, a Sec-WebSocket-Version header needs to be returned, which contains the version number supported by the server.Sec-WebSocket-Key: It is matched with the Sec-WebSocket-Accept in the subsequent server response header to provide basic protection, such as malicious connections or unintentional connections.
Note that the above request omits some non-key request headers. Since it is a standard HTTP request, request headers such as Host, Origin, Cookie, etc. will be sent as usual. During the handshake phase, security restrictions, permission verification, etc. can be carried out through relevant request headers.
2. Server: respond to protocol upgrade
The server returns the following content. Status code 101 indicates protocol switching. The protocol upgrade is now completed, and subsequent data interactions will follow the new protocol.
HTTP/1.1 101 Switching Protocols
Connection:Upgrade
Upgrade: websocket
Sec-WebSocket-Accept: Oy4NRAQ13jhfONC7bP8dTKb4PTU=
Note: Each header ends with \r\n, and an extra blank line \r\n is added to the last line. In addition, the HTTP status code returned by the server can only be used during the handshake phase. After the handshake phase, only specific error codes can be used.
3. Calculation of Sec-WebSocket-Accept
The main function of Sec-WebSocket-Key/Sec-WebSocket-Accept is to provide basic protection and reduce malicious connections and accidental connections. The main purpose of Sec-WebSocket-Key is not to ensure data security, because the conversion calculation formulas of Sec-WebSocket-Key and Sec-WebSocket-Accept are public and very simple. The main function is to prevent some common accidents. (not on purpose)
Sec-WebSocket-Accept is calculated based on the Sec-WebSocket-Key in the client request header.
The calculation formula is:
- Splice Sec-WebSocket-Key with 258EAFA5-E914-47DA-95CA-C5AB0DC85B11.
- The digest is calculated via SHA1 and converted into a base64 string.
The pseudocode is as follows:
>toBase64( sha1( Sec-WebSocket-Key + 258EAFA5-E914-47DA-95CA-C5AB0DC85B11 ) )
Verify the previous return results:
const crypto = require('crypto');
const magic = '258EAFA5-E914-47DA-95CA-C5AB0DC85B11';
const secWebSocketKey = 'w4v7O6xFTi36lq3RNcgctw==';
let secWebSocketAccept = crypto.createHash('sha1')
.update(secWebSocketKey + magic)
.digest('base64');
console.log(secWebSocketAccept);
// Oy4NRAQ13jhfONC7bP8dTKb4PTU=
WireShark packet capture icon
ws protocol data interaction
The exchange of client and server data is inseparable from the definition of data frame format. Therefore, before actually explaining data exchange, let's first look at the data frame format of WebSocket.
The smallest unit of communication between WebSocket client and server is a frame, which consists of one or more frames to form a complete message.
- Sender: Cut the message into multiple frames and send them to the server;
- Receiving end: receives message frames and reassembles associated frames into complete messages;
The focus of this section is to explain the format of the data frame. For detailed definition, please refer to RFC6455 Section 5.2 .
1. Overview of data frame format
The unified format of WebSocket data frame is given below. Students familiar with the TCP/IP protocol should be familiar with such a diagram.
- From left to right, the units are bits. For example, FIN and RSV1 each occupy 1 bit, and opcode occupies 4 bits.
- The content includes identification, operation code, mask, data, data length, etc. (will be expanded on in the next section)
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-------+-+-------------+-------------------------------+
|F|R|R|R| opcode|M| Payload len | Extended payload length |
|I|S|S|S| (4) |A| (7) | (16/64) |
|N|V|V|V| |S| | (if payload len==126/127) |
| |1|2|3| |K| | |
+-+-+-+-+-------+-+-------------+ - - - - - - - - - - - - - - - +
| Extended payload length continued, if payload len == 127 |
+ - - - - - - - - - - - - - - - +-------------------------------+
| |Masking-key, if MASK set to 1 |
+-------------------------------+-------------------------------+
| Masking-key (continued) | Payload Data |
+-------------------------------- - - - - - - - - - - - - - - - +
: Payload Data continued ... :
+ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +
| Payload Data continued ... |
+---------------------------------------------------------------+
2. Detailed explanation of data frame format
Regarding the previous format overview diagram, here we will explain it field by field. If there is anything unclear, you can refer to the protocol specification or leave a message for communication.
FIN: 1 ratio special.
If it is 1, it means that this is the last fragment of the message (message). If it is 0, it means that it is not the last fragment of the message (message).
RSV1, RSV2, RSV3: Each occupies 1 bit.
Normally all are 0. When the client and server negotiate to use the WebSocket extension, these three flags can be non-zero, and the meaning of the values is defined by the extension. If a non-zero value is present and the WebSocket extension is not used, a connection error occurs.
Opcode: 4 ratio special.
Operation code, the value of Opcode determines how the subsequent data payload should be parsed. If the operation code is unknown, the receiving end SHOULD fail the connection. Optional operation codes are as follows:
- %x0: represents a continuation frame. When Opcode is 0, it means that data fragmentation is used for this data transmission, and the currently received data frame is one of the data fragments.
- %x1: Indicates that this is a text frame (frame)
- %x2: Indicates that this is a binary frame (frame)
- %x3-7: Reserved operation code for subsequent defined non-control frames.
- %x8: Indicates that the connection is disconnected.
- %x9: Indicates that this is a ping operation.
- %xA: Indicates that this is a pong operation.
- %xB-F: Reserved operation codes for subsequent defined control frames.
Mask: 1 ratio.
Indicates whether to mask the data payload. When sending data from the client to the server, the data needs to be masked; when sending data from the server to the client, the data does not need to be masked.
If the data received by the server has not been masked, the server needs to disconnect.
If Mask is 1, then a masking key will be defined in Masking-key, and this masking key will be used to unmask the data payload. All data frames sent from the client to the server have Mask 1.
The algorithm and purpose of masking are explained in the next section.
Payload length: The length of the data payload, in bytes. It is 7 bits, or 7+16 bits, or 1+64 bits.
Assume that the number Payload length === x, if
- x is 0~126: the length of the data is x bytes.
- x is 126: The following 2 bytes represent a 16-bit unsigned integer, and the value of the unsigned integer is the length of the data.
- x is 127: The following 8 bytes represent a 64-bit unsigned integer (the highest bit is 0), and the value of this unsigned integer is the length of the data.
In addition, if the payload length occupies multiple bytes, the binary representation of the payload length uses network order (big endian, important bits first).
Masking-key: 0 or 4 bytes (32 bits)
For all data frames transmitted from the client to the server, the data payload is masked, with a Mask of 1 and a 4-byte Masking-key. If Mask is 0, there is no Masking-key.
Note: The length of the payload data does not include the length of the mask key.
Payload data：(x+y) 字节
Payload data: includes extended data and application data. Among them, the extended data is x bytes and the application data is y bytes.
Extension data: If no extension is negotiated, the extension data is 0 bytes. All extensions must declare the length of the extension data, or how the length of the extension data can be calculated. Additionally, how the extension is used must be negotiated during the handshake phase. If extended data is present, the payload data length must include the length of the extended data.
Application data: Any application data, after extended data (if extended data exists), occupies the remaining position of the data frame. The length of the payload data minus the extension data length is the length of the application data.
3. Data transfer
Once the WebSocket client and server establish a connection, subsequent operations are based on the transmission of data frames.
WebSocket distinguishes the type of operation based on opcode. For example, 0x8 indicates disconnection, and 0x0-0x2 indicates data exchange.
1. Data fragmentation
Each WebSocket message may be split into multiple data frames. When the WebSocket receiver receives a data frame, it will determine whether the last data frame of the message has been received based on the value of FIN.
FIN=1 indicates that the current data frame is the last data frame of the message. At this time, the receiver has received the complete message and can process the message. FIN=0, then the receiver needs to continue to listen and receive other data frames.
In addition, opcode represents the type of data in the context of data exchange. 0x01 means text, 0x02 means binary. 0x00 is special and represents a continuation frame. As the name suggests, the data frame corresponding to the complete message has not yet been received.
2. Data fragmentation example
It’s more vivid to look at examples directly. The following example comes from MDN and can well demonstrate data sharding. The client sends messages to the server twice, and the server responds to the client after receiving the message. Here we mainly look at the messages sent by the client to the server.
first message
FIN=1, which means it is the last data frame of the current message. After the server receives the current data frame, it can process the message. opcode=0x1 means that the client sends text type.
second message
FIN=0, opcode=0x1, which means that the text type is sent, and the message has not been sent yet, and there are subsequent data frames.
FIN=0, opcode=0x0, which means that the message has not been sent yet, and there are subsequent data frames. The current data frame needs to be connected after the previous data frame.
FIN=1, opcode=0x0, indicating that the message has been sent and there is no subsequent data frame. The current data frame needs to be connected after the previous data frame. The server can assemble associated data frames into complete messages.
Client: FIN=1, opcode=0x1, msg="hello"
Server: (process complete message immediately) Hi.
Client: FIN=0, opcode=0x1, msg="and a"
Server: (listening, new message containing text started)
Client: FIN=0, opcode=0x0, msg="happy new"
Server: (listening, payload concatenated to previous message)
Client: FIN=1, opcode=0x0, msg="year!"
Server: (process complete message) Happy new year to you too!
4. Connection maintenance + heartbeat
In order to maintain real-time two-way communication between the client and the server, WebSocket needs to ensure that the TCP channel between the client and the server remains connected. However, for a connection that has no data exchange for a long time, if it is still maintained for a long time, the included connection resources may be wasted.
However, some scenarios cannot be ruled out. Although there is no data exchange between the client and the server for a long time, they still need to maintain the connection. At this time, heartbeat can be used to achieve this.
- Sender->Receiver: ping
- Receiver->Sender: pong
The operations of ping and pong correspond to the two control frames of WebSocket, and the opcodes are 0x9 and 0xA respectively.
For example, to send a ping from the WebSocket server to the client, only the following code is needed (using the ws module)
ws.ping('', false, true);