Netty codec and custom protocol
Encoding and decoding
basic introduction
- When writing a network application, because the data transmitted in the network is binary byte code data, it needs to be encoded when sending data, and it needs to be decoded when receiving data.
There are two components of codec (encoder): encoder (encoder) and decoder (encoder). When sending data, the data encoder must be converted into bytecode data, and when the data is received, the data decoder must be converted into business. data
Netty encoder
- Netty itself provides some encoders such as StringEncoder (encoding a string), ObjectEncoder (encoding a java object), StringDecoder (decoding a string), ObjectDecoder (decoding a java object)
- The ObjectEncoder and ObjectDecoder provided by Netty itself can be used to encode and decode POJO objects or various business objects. Even though the bottom layer uses java serial numbers, the java serialization technology itself is not efficient, and there are the following problems
- Cannot cross language
- After serialization, the volume is too large, which is more than 5 times of the binary code
- Serialization performance is too low
- The solution uses Google's Protobuf
Netty commonly used decoder
| decoder | Description |
|---|---|
| LineBasedFrameDecoder | The decoder will divide the received data according to (\n or \r\n) and send it to Handel for processing |
| DelimiterBasedFrameDecoder | Use custom special characters as message separators |
| HttpObjectDecoder | A decoder for HTTP data |
| LengthFileIdBasedFrameDecoder | By specifying the length to identify the entire packet message, so that the sticky packet and semi-packet message can be automatically processed |
Protobuf encoder
basic introduction
- Protobuf is an open source project released by Google. The full name is Google Protocol Buffers. It is a lightweight and efficient structured data storage format. It can be used for the serialization of structured data, or serial numbers. It is very suitable for data storage or RPC (remote procedure). Call) data exchange format
- Reference document: https://developers.google.com/protocol-buffers/docs/javatutorial
- Protobuf manages data in the form of message
- Protobuf supports cross-platform, server-side and can be written in different languages (C++, C#, Go, Java, Python, etc.)
how to use
-
To use the Protobuf encoder step, first we need to write a xxx.proto file and then use prptoc.exe to compile the xxx.proto we wrote. After compilation, a xxx.java file corresponding to xxx.proto will be generated. Use this The java file can encode (ProtobufEncoder) and decode (ProtobufDecoder) for this class
-
When writing .proto files in idea, you need to download the Protocol Buffers plug-in, so that when writing proto files in idea, there will be syntax highlighting reminders
Protobuf example
Introduce Maven
<!-- https://mvnrepository.com/artifact/com.google.protobuf/protobuf-java -->
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>3.6.1</version>
</dependency>
The proto type corresponds to the Java type table
Write a Student.proto
syntax = "proto3";//协议版本
option java_outer_classname = "StudentPOJO";//生成的外部类名，就是生成的文件名
//protobuf 使用message 管理数据
message Student{ //StudentPOJO中会生成一个Student内部类，它是真正发送的数据对象
int32 id = 1;//Student 类有一个属性 名字为 id 类型为int32 1表示属性的序号
string name = 2;
}
Download protoc.exe
Path: https://github.com/protocolbuffers/protobuf/releases
Execute the following command: protoc.exe --java_out=. Student.proto , a StudentPOJO.java file will be generated
Note: The protoc.exe downloaded from version 3.6.1 used in maven should also be 3.6.1
Copy into idea
We can see that there is an internal class in StudentPOJO. Student is the class that is really used to send data. We can see that there are many methods in it.
Client
The pipeline joins the ProtobufEncoder encoder
The client directly sets the data inside through **StudentPOJO.Student.newBuilder() and finally uses the build()** method to get the student object
Server
The pipeline joins the decoder, the decoding type is StudentPOJO.Student.getDefaultInstance()
The custom Handler receives directly, and the data can be obtained by using the get method
Protobuf multiple types
Through the Protobuf example, we basically know how to use Protobuf but we found a problem. Our type is very single. If we need to pass in different types, then the client and server need to create a lot of xxx.proto, client and The server side needs to configure various types of decoders in the pipeline, and the server side also needs to use different Handlers to receive
We rewrite a MyDataInfo
Here we store two message Student and Worker. We use a MyMessage to manage these two types. MyMessage uses enum to pass in 0/1 to obtain the corresponding type.
Client
On the client side, we randomly send different types of data in the past
Server
The server can decode MyMessage
Server Handler
The handler uses MyMessage to receive, and executes different types of printing by judging the type.
Custom encoder
basic introduction
We can customize the encoder and decoder, and decode and encode the data according to the rules we set. Before customizing the decoder and encoder, let's once again understand how the encoding and decoding handler chain of responsibility works
Our handler is divided into outboundhandl and inboundhandl, inboundhandl is an inbound event, outboundhandl is an outbound event, encoded as an outbound event, decoded as an inbound event, when to decode and when to encode, for example, the client is us, the service The end is the express station
- For example, if I want to send a courier to the courier station, then the first step we need to do is to pack our courier into the car, and then load the car to facilitate us to send to the courier station (encode here, our courier must be shipped It needs to be coded at the station), packed and sent to the courier station. The courier station sees that there is a courier entering the station, and it needs to unload all of our express (decoding here, when the data is received in the station) (Need to be decoded)
- For example, if the express station needs to send the express to us, then the express station needs to take a packing truck and pack it before it can be sent to us (encoding here, we need to encode when our express is out of the station), after the express arrives in our hands , Then we need to disassemble the express (decoding here, it needs to be decoded when data is received)
Encoding and decoding are relative to one party. Compared to the client, I want to send data to the server and need to be encoded (outboundhandl), and the data received from the server needs to be decoded (inboundhandl). Compared to the server, I want to send data to The client needs to be encoded (outboundhandl), and the data received from the client needs to be decoded (inboundhandl)
After the above explanation, I believe that I already know why we need to decode and encode and the process of decoding and encoding. Then why do we need to customize the encoder? For the example of me and the express station above, now I want to take the express to At the express station, do I need to load the car? I (the client) just load the car and a driver (Scoket) will pull it over, and the driver will pull it over. The question comes to the express station (server) in what order These goods are unloaded, if they are not in order, the goods will be heavy and the truck will fall down and become messy. This requires us to formulate a rule, how the client puts it, and how the server disassembles it. This is the custom decoder. usefulness
Custom encoder
To write a custom encoder, you need to send a MessageToByteEncoder<> and you need to pass in a generic, which indicates what type of encoding the class receives. For example, the current class can accept a Long type and encode it, use the writeLong() method to write
MessageToByteEncoder is inherited from OutboundHandler, it can be seen that encoding needs to be performed outbound
Custom decoder
To write a custom encoder, you need to send a ByteToMessageDecoder. The if here is to determine whether the received data is consistent (the TCP unpacking and dipping will explain the method later), so only if more than 8 characters can the message readLong operation, and then add to List out This will be added to the next handler processing
ByteToMessageDecoder is inherited and InboundHandler can see the inbound and needs to be decoded
Data transmission
Add our encoder and decoder to the pipeline channel, when the data enters the inbound and outbound channels, there will be corresponding channels for processing
In this way, when we call the ctx.writeAndFlush() method, we don’t need to use ctx.writeAndFlush(Unpooled.copyLong(1123456)); the writing method is written directly as ctx.writeAndFlush(123456L); when passing through the outbound handler, it will pass through the MessageToByteEncoder. Will use the writeLong() method to encode it
TCP sticking and unpacking
basic introduction
- TCP is connection-oriented, stream-oriented, and provides high-reliability services. There must be a pair of sockets at both ends of the transceiver (client and server). Sometimes, in order to improve the efficiency of transmission, the client will perform the transmission of data packets. Excellent (Nagle algorithm), combine multiple data with small intervals and small data volume into one large data packet, and then send it to the receiving end, but there is a problem with the receiving end, and the receiving end cannot distinguish the packet you sent Middle, is how many packets need to be split into
- Since TCP has no message protection boundary, the message boundary problem needs to be dealt with at the receiving end, which is what we call the sticking and unpacking problem
This is data transmission in an ideal state. The client sends two data packets to the server first, and the D1 and D2 servers accept the independent data packets of D1 and D2 in two times. There is no sticking and unpacking problems.
The server may receive two data packets at once, D1 and D2 are glued together, which is called TCP sticky packet
The server reads the data packet twice, but the first time it reads a complete D1 packet and a part of D2, the second time it reads the part of D2, which is called TCP unpacking
The server reads the data packet twice, but the first time it reads a part of D2, the second time it reads a complete D1 packet and the rest of D2. This is also called TCP unpacking.
Summary : Assuming you are the server, you receive the package simulation in the above situation. How to judge whether they are a complete package? How to split and merge them? It is possible that the incomplete package will cause problems in your entire business logic. TCP Sticking and unpacking are problems that must be solved in network programming
TCP sticky packet and unpacking case
Client
After the client establishes a connection, it will call a for loop, which will send a hello to the server, and the server will print out a message if it receives a message sent by the server.
Server
After the server receives the readable event, it will read the data and output it to the console. After the output is completed, it will send a piece of data back to the current channel
Run speed test
We can find that the data received by the server sends a sticky packet.
The problem that the client also sends sticky packets when receiving data from the server
solution
- Use custom protocol + codec to solve TCP sticking and unpacking problems
Custom protocol package
The len of the protocol packet is the key, the server will decode according to the length you sent when decoding
Custom encoder
Encode and send the MessageProtocol object in the channel
Custom decoder
The received data is decoded into a MessageProtocol object and processed by the next Handler
Add custom codecs to the client and server pipelines
Client and server receive and send data uniformly with MessageProtocol object
Test Results
Both the client and the server can decode according to the encapsulated MessageProtocol, and there will be no problems with packaging and unpacking.
