Six, Dubbo protocol module principle source code analysis

Course Outline:

Basic composition of RPC protocol
RPC protocol message encoding and implementation details
RPC protocol and usage supported in Dubbo

Basic composition of RPC protocol

Explanation of RPC protocol terms

In a typical RPC usage scenario, it includes components such as service discovery, load, fault tolerance, network transmission, and serialization. The RPC protocol specifies how the program performs network transmission and serialization. In other words, the implementation of an RPC protocol is equivalent to a non-transparent remote call implementation. How to do it?

The basic composition of the agreement:

Address: service provider address
Port: The protocol specifies the open port
Message encoding: Protocol message encoding, which is divided into two parts: request header and request body.
Serialization method: serialize the request body into an object
Hessian2Serialization、
DubboSerialization、
JavaSerialization
JsonSerialization
Running service: network transmission implementation
netty
mina
RMI service
servlet container (jetty, Tomcat, Jboss)

Use of RPC protocols supported in Dubbo

List of RPC protocols supported by dubbo

name	implementation description	connection description	Applicable scene
dubbo	Transmission service: mina, netty (default), grizzy Serialization: hessian2 (default), java, fastjson custom message	Single long connection NIO asynchronous transmission	1. Regular RPC calls 2. Small amount of data to be transferred 3. Providers are less than consumers
rmi	Transport: java rmi service serialization: java native binary serialization	Synchronous transmission of multiple short connection BIOs	1. Regular RPC call 2. Integration with the original RMI client 3. A small amount of files can be transferred 4. No firewall penetration
hessian	Transport service: servlet container Serialization: Hessian binary serialization	Based on Http protocol transmission, according to lazy servlet container configuration	1. There are more providers than consumers 2. Large fields and files can be transmitted ~~3. Cross-language calls~~
http	Transport service: servlet container serialization: java native binary serialization	According to lazy servlet container configuration	1. Mixed packet sizes
thrift	Integrate with thrift RPC and modify the packet header based on it	Long connection, NIO asynchronous transmission

Supplementary note about RMI not supporting firewall penetration:

原因在于RMI 底层实现中会有两个端口，一个是固定的用于服务发现的注册端口，另外会生成一个***随机***端口用于网络传输。因为这个随机端口就不能在防火墙中提前设置开放开。所以存在*防火墙穿透问题*

Protocol usage and configuration:

The Dubbo framework configuration protocol is very convenient, users only need to configure the *<* dubbo:protocol> element in the provider application .

 <!--
   name: 协议名称 dubbo|rmi|hessian|http|
   host:本机IP可不填，则系统自动获取
   port：端口、填-1表示系统自动选择
   server：运行服务  mina|netty|grizzy|servlet|jetty
   serialization：序列化方式 hessian2|java|compactedjava|fastjson
   详细配置参见dubbo 官网 dubbo.io
 -->
 <dubbo:protocol name="dubbo" host="192.168.0.11" port="20880" server="netty" 
  serialization=“hessian2” charset=“UTF-8” />

#TODO demonstrate using other protocols to configure Dubbo

The dubbo protocol uses json for serialization (see source code: com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol*)*
Adopt RMI protocol (see source code: com.alibaba.dubbo.rpc.protocol.rmi.RmiProtocol)
Adopt Http protocol (see source code: com.alibaba.dubbo.rpc.protocol.http.HttpProtocol.InternalHandler)
Adopt the Heason protocol (see source code: com.alibaba.dubbo.rpc.protocol.hessian.HessianProtocol.HessianHandler)

new PrintWriter(System.out)

netstat -aon|findstr "17732"

Serialization:

	features
fastjson	Text type: large size, slow performance, cross-language, high readability
fst	Binary type: Small size, compatible with JDK native serialization. Requires JDK 1.7 support.
hessian2	Binary type: cross-language, high fault tolerance, small size
java	Binary type: Null can be written on the basis of native JAVA
compactedjava	Binary type: similar to java, the content is compressed
nativejava	Binary type: native JAVA serialization
cryo	Binary type: the volume is smaller than hessian2, but the fault tolerance is not as good as hessian2

Hessian serialization:

Parameters and return values need to implement the Serializable interface
Parameters and return values cannot be customized to implement interfaces such as List, Map, Number, Date, Calendar, etc., and can only be implemented by JDK, because hessian will do special processing, and the attribute values in the custom implementation class will be lost.
Hessian serialization, only pass member attribute values and value types, not methods or static variables, compatibility [1] [2] :

data communication	Condition	result
A->B	Class A has one more attribute (or class B has one less attribute)	No exception is thrown, the value of the attribute with more in A, but not in B, the others are normal
A->B	Enumeration A has one more enumeration (or B has one less enumeration), and A uses the extra enumeration for transmission	throw exception
A->B	Enumeration A has one more enumeration (or B has one less enumeration), and A does not use the extra enumeration for transmission	No exception is thrown, B receives data normally
A->B	A and B have the same property name but different types	throw exception
A->B	serialId is not the same	normal transmission

The method added to the interface has no impact on the client. If the method is not required by the client, the client does not need to redeploy. The addition of attributes in input parameters and result sets has no effect on the client. If the client does not need new attributes, there is no need to redeploy.

Changes in input parameters and result set property names have no impact on client serialization, but if the client does not redeploy, regardless of input or output, property values with changed property names cannot be obtained.

Summary: The server side and the client side do not need to be completely consistent with the domain objects, but follow the principle of maximum matching.

3. Detailed explanation of RPC protocol message encoding and implementation

RPC transport implementation:

The transmission of the RPC protocol is based on TCP/IP, using Socket or Netty, mina and other network programming components. But there is a problem that TCP is a byte stream-oriented borderless protocol. It is only responsible for data transmission and does not distinguish the messages corresponding to each request. This will cause the problem of unpacking and sticking packets in TCP protocol transmission.

Reasons for unpacking and sticking:

We know that tcp transmits data in a flowing manner, and the smallest unit of transmission is a segment. There is an Options flag in the tcp Header. The common flag is mss (Maximum Segment Size), which means that the data transmitted by the connection layer each time has a maximum MTU (Maximum Transmission Unit), which is generally 1500 bits. Divided into multiple message segments, mss is the maximum limit minus the TCP header, just the size of the data to be transmitted, generally 1460 bits. Converted into bytes, that is more than 180 bytes.

In order to improve the performance of tcp, the sender will send the data to be sent to the buffer, wait for the buffer to be full, and then send the data in the buffer to the receiver. Similarly, the receiver also has a mechanism such as a buffer to receive data. Then the following will happen:

The data written by the application is larger than the MSS size, which will cause unpacking.
If the data written by the application is smaller than the MSS size, sticky packets will occur.
The receiving method does not read the socket buffer data in time, which will cause sticky packets.

Unpacking and sticking solutions:

Set a fixed-length message, and the server reads the content of the predetermined length each time as a complete message.
{“type”:“message”,“content”:“hello”}\n
Using a protocol with a message header, the message header stores the message start identifier and message length information. When the server obtains the message header, it parses out the message length, and then reads the content of the length backward.

**For example: **Content-Length in the Http protocol header indicates the size of the message body.

(Note ①: http message encoding)

Dubbo protocol message encoding:

Note ② Dubbo protocol message encoding:

magic : Similar to the magic number in the java bytecode file, it is used to judge whether it is a data packet of the dubbo protocol. The magic number is the constant 0xdabb, which is used to judge the beginning of the message.
flag : flag bit, a total of 8 address bits. The lower four bits are used to indicate the type of serialization tool used for the message body data (default hessian). Among the upper four bits, the first bit is 1, which means it is a request request, and the second bit is 1, which means two-way transmission (that is, there is a response returned) , and the third bit is 1, indicating that it is a heartbeat ping event.
status : status bit, set request response status, dubbo defines some response types. For specific types, see com.alibaba.dubbo.remoting.exchange.Response
invoke id: message id, long type. The unique identification id of each request (due to the use of asynchronous communication, it is used to correspond to the request request and the returned response)
body length: the length of the message body, int type, that is, how many bytes are recorded in the Body Content.

*（注：相关源码参见 **c**om.alibaba.dubbo.rpc.protocol.dubbo.DubboCodec**）*

The encoding and decoding process of Dubbo protocol:

Dubbo protocol encoding and decoding implementation process (source code comes from **dubbo2.5.8)

1、DubboCodec.encodeRequestData() 116L // 编码request
2、DecodeableRpcInvocation.decode()  89L   // 解码request
3、DubboCodec.encodeResponseData()   184L  // 编码response    
4、DecodeableRpcResult.decode()      73L   // 解码response