Ants gold recently served open-source research and development for many years to a basket of SOFA framework in which there is a very central RPC framework, it is called SOFA-BOLT. Today, it took nearly a day's time to carefully study its source code reading, reading encountered a number of problems, the gold suit ant relevant technical staff are very patient and timely answers to my difficult. Here I learned the knowledge together for everyone to share.
SOFA-BOLT Netty framework based on open source, while providing the server and the client. Its source is well worth reading, simple, thoughtful, is not an ordinary toy. It did not abuse design patterns, source code read up more directly, there is not much around the complex structure to go around.
A node can either be both RPC server and a client, as the client node of the other nodes need to provide services, as a server that provides services to other nodes. But this picture is not above a reasonable structure, because the two services coupled with each other, I need you, you need me, became egg problem. Reasonable general structure as shown below, do not form a ring between them.
Protocol
Protocol exchanges between client and server language, SOFA defines its own set of communication protocols, which codec is divided into two layers, a first layer is a binary sequence of message body object, which is part of the default by the open source Hession protocol library sequence is complete, the second layer is responsible for increasing the sequence number of the message body field of packaging, to form a complete message. Includes a request ID, length of the message body, the protocol version number and CRC32 checksum bits, etc.
If you want to further optimize network performance, SOFA Snappy compression protocol is also provided, the third layer may be increased in the conventional two-layer protocol basis, can significantly reduce the burden on network transmission. Time for space compression simultaneously, improve network performance, it will increase the CPU calculates, when it is necessary to appropriately use tradeoff.
connection pool
Generally it requires multiple connections between the client and the server, but can not establish a connection with each request. Typically maintains a connection pool by defining the maximum number of connections. Limited to the client and the server are connected via communication.
我们在使用Jedis客户端和Redis服务器进行通信时，也是通过连接池来获取连接的。Jedis的连接必须是线程独占的，因为它不是线程安全的。从连接池中获取连接时，其它线程就暂时拿不到这个连接了，待当前线程处理完毕后，要将连接归还给线程池，这样其它线程才可以继续使用这个连接。
Redis的客户端请求和应答是顺序性的，一问一答，所以请求和应答不需要唯一ID就可以建立起关联。
Bolt不一样，它的问答是乱序的，问和答之间是必须通过请求的唯一ID来建立起关联。Bolt的客户端是线程安全的，它可以同时传递多个请求，连接对象会维护一个正在处理的RPC请求对象字典。当客户端想要发起RPC请求时，它不是从连接池中摘出一个独占连接，而是随意选择一个连接来传递自己的请求，这个连接也可以被其它线程同时使用。
负载均衡
客户端提供了多种复杂均衡的实现，阿里默认使用带权重的随机算法(RandomLoadBalancer)，此外还有
- ConsistentHashLoaderBalancer 一致性hash，客户端和服务器之间的连接关系(谁跟谁连)比较稳定
- LocalPreferenceLoadBalancer 本地环回地址优先，提升本机调用性能
- RoundRobinLoadBalancer 循环依次来
- WeightedRoundRobinLoadBalancer 带权重的循环依次来
- RandomLoadBalancer 这个是带权重的随机，阿里的默认使用
服务器线程模型
服务器采用传统netty多线程模型，一个acceptor线程专门用来接收连接，然后扔给io线程处理读消息并解码成请求对象，最后扔给业务线程池进行处理。
心跳
客户端和服务器之间会有定时心跳检测连接的存活，默认30s来一次。tcp的关闭是通过FIN包来通知对方的，如果因为网络问题，对方连FIN包都收不到，那么即使一边关闭了套接字，另一边可能还以为连接正常。所以心跳检测存活机制在长连接应用里非常普遍。如果客户端连续发了三次心跳都没有收到服务器的回复，那么就认为连接已经关闭。服务器也会有连接存活检测，如果一个客户端连接90s内没有任何消息进来，那么也认为该连接已经断开。服务器不会主动发送心跳消息。
双工通信
RPC一般是由客户端向服务器发起一个请求，然后收到服务器的应答。Bolt的RPC是双工通信，服务器也可以向客户端主动发起请求，它们共享一个TCP连接。TCP连接本身就是双工的，所以这也不算什么奇迹。只是服务器在什么业务场景需要向客户端主动发起请求，这个蚂蚁并没有进行详细说明。
客户端作为主动连接方，它要负责重连和发起心跳消息。服务器作为被动方，它不需要处理重连，如果连接断开，它就直接将连接从集合中移除就行，不需要做特殊的处理，但是它会检测心跳消息，如果在一定时间内连接通道没有任何消息到来，它就会主动关闭。
重连
客户端的重连策略是一个单独的模块，有两个地方会成为重连的入口。一个是正常连接断开触发channelInActive回调，另一个就是重连连接不能建立成功时需要进行重试。Bolt有一个单独的重连线程，所有需要重连的连接会被包装成一个任务塞进这个线程的任务队列，该线程不断地从队列里拿任务进行重连处理，如果重连失败会尝试再将任务重新包装进队列延后继续处理。默认是1s钟处理一个重连任务。
RPC连接是延迟建立的，它在第一次客户端发送RPC请求时尝试进行连接，如果连接失败，它会立即继续重连最多默认两次。如果三次尝试连接后还是没有建立成功，就向上层爆出异常。它不需要包装一个重连任务塞进ReconnectManager，因为后续客户端请求会继续触发连接。
单向消息
RPC is usually a should answer, the client can synchronize waiting for a response, may also provide an interface to wait for results notification callback. Bolt In addition to providing the answer mode, but also provides a oneway one-way message, the server receives such a message without reply, after client sends a request immediately returned do not need to wait for the results.
oneway messages are typically used for less important for the log messages, it does not guarantee that the server will be able to receive, so this is the kind of message should allow business lost messages, similar in form to UDP, it is sacrificing reliability under can significantly enhance the throughput of messages.
Message tracking
Bolt provides a callback interface for easy monitoring system can analyze the call status request. Monitoring client can register into RPC client and server logs collected by the striking implement this interface, and then sent to the log analysis system.