foreword
The RPC framework needs to maintain the connection between the client and the server. Usually, one client corresponds to multiple servers, and the client sees the interface, not the address of the server. The address of the server is transparent to the client.
So, how to realize the network connection of such an RPC framework?
We look for answers in SOFA.
Introducing the Connection Manager
Let's start with a small demo:
ConsumerConfig<HelloService> consumerConfig = new ConsumerConfig<HelloService>()
.setInterfaceId(HelloService.class.getName()) // 指定接口
.setProtocol("bolt") // 指定协议
.setDirectUrl("bolt://127.0.0.1:9696"); // 指定直连地址
HelloService helloService = consumerConfig.refer();
while (true) {
System.out.println(helloService.sayHello("world"));
try {
Thread.sleep(2000);
} catch (Exception e) {
}
}In the above code, a ConsumerConfig corresponds to an interface service and specifies a direct connection address.
Then call the ref method. Each ConsumerConfig is bound to a ConsumerBootstrap, which is a non-singleton class.
And each ConsumerBootstrap is bound to a Cluster, which is the real client. This class contains all the key information of a client, for example:
- Router routing chain
- loadBalance load balancing
- addressHolder address manager
- connectionHolder connection manager
- filterChain filter chain
These five instances are the core of the Cluster. The normal use of a client is absolutely inseparable from these five elements.
We've analyzed 4 of the 5 before, and today we're analyzing the last one - the connection manager.
It can be said to be the core of RPC network communication.
The address manager stands for: a client can have multiple interfaces.
The connection manager stands for: a client can have multiple TCP connections.
Obviously, the address manager must have more data than the connection manager. Because usually a TCP connection (Server side) can contain multiple interfaces.
So how does SOFA implement the connection manager?
From the init method of AbstractCluster, we know that this method initializes the Cluster. The connectionHolder is also initialized.
The specific code is as follows:
// 连接管理器
connectionHolder = ConnectionHolderFactory.getConnectionHolder(consumerBootstrap);Initialization using SPI. Currently, the concrete implementation class of the RPC framework has only one AllConnectConnectionHolder. That is, the long-term connection manager.
The class requires a ConsumerConfig to initialize.
This class contains many connection-related properties, including 4 Maps, an uninitialized Map, a list of surviving nodes, a list of surviving but sub-healthy nodes, and a list of failed attempts to retry. The elements of these Maps will change as the service's network changes.
The elements in these Maps are: ConcurrentHashMap
That is, the information of each server corresponds to a client transmission. So what is this ClientTransport? Anyone who has read the previous article knows that this is a glue class for RPC and Bolt. The default implementation of this class, BoltClientTransport, includes an RpcClient property. Note that this property is static. That is, it is common to all instances. Also, BoltClientTransport contains a ProviderInfo property. There is also a Url property, the Connection property (network connection).
Let's take a look: a ConsumerConfig is bound to a Cluster, a Cluster is bound to a connectionHolder, and a connectionHolder is bound to multiple ProviderInfo and ClientTransport.
Because a client can communicate with multiple services.
How is the code implemented?
In the Cluster, the connectionHolder will be initialized, and after the Cluster gets the server list from the registry, a long connection will be established.
From here, the address manager comes into play.
The Cluster's updateAllProviders method is the source. This method will add the list of services to the connectionHolder. That is, call the connectionHolder.updateAllProviders(providerGroups) method. This method will fully update the server list.
If a new service is found during the update, a long connection will be established. The specific code is as follows:
if (!needAdd.isEmpty()) {
addNode(needAdd);
}The addNode method is to add a new node. This method establishes a TCP connection in multiple threads.
First, a ClientTransport will be created according to the ProviderInfo information, and then a task will be submitted to the thread pool. The content of the task is initClientTransport(), which is to initialize the client transmission.
The method code is as follows (simplified):
private void initClientTransport(String interfaceId, ProviderInfo providerInfo, ClientTransport transport) {
transport.connect();
if (doubleCheck(interfaceId, providerInfo, transport)) {
printSuccess(interfaceId, providerInfo, transport);
addAlive(providerInfo, transport);
} else {
printFailure(interfaceId, providerInfo, transport);
addRetry(providerInfo, transport);
}
}The key is to call the connect method of transport to establish a connection.
The default implementation of this method is in BoltClientTransport, as we expected. We know that BoltClientTransport has a static instance of RpcClient. This instance is initialized in the static block when the class is loaded. The initialization content is to initialize some of his properties, such as address parser, connection manager, connection monitoring and so on.
Let's look at the connect method of BoltClientTransport. The main logic of this method is to initialize the connection. The way is to get it through the getConnection method of RpcClient. The specific code is as follows:
connection = RPC_CLIENT.getConnection(url, url.getConnectTimeout());Pass in a URL and timeout. RpcClient is obtained by calling the getAndCreateIfAbsent method of the connection manager, and the Url is also passed in. The name of this method is very good, and the connection is obtained according to the URL. If not, create one.
It is necessary to look at the specific code:
public Connection getAndCreateIfAbsent(Url url) throws InterruptedException, RemotingException {
// get and create a connection pool with initialized connections.
ConnectionPool pool = this.getConnectionPoolAndCreateIfAbsent(url.getUniqueKey(),
new ConnectionPoolCall(url));
if (null != pool) {
return pool.get();
} else {
logger.error("[NOTIFYME] bug detected! pool here must not be null!");
return null;
}
}The method will continue to call its own getConnectionPoolAndCreateIfAbsent method, passing in the URL's unique identifier, and a ConnectionPoolCall object (which implements Callable).
Then block waiting for a connection to be returned.
Let's look at the implementation of the call method of this ConnectionPoolCall. This method calls the connection manager's doCreate method. The URL and a connection pool are passed in. Then the call method returns the connection pool.
In the doCreate method, the focus is on the create method, which passes in a url, returns a Connection, and puts it into the connection pool. There is only one persistent connection in the default pool.
The create method calls the createConnection method of the connection factory. Then call the doCreateConnection method. This method gives us a clear answer: call the connect method of Netty's Bootstrap.
code show as below:
bootstrap.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout);
ChannelFuture future = bootstrap.connect(new InetSocketAddress(targetIP, targetPort));Students who are familiar with Netty can see it at a glance. This is a connection server operation. The initialization of this BootStrap is carried out when the RpcClient is initialized. Note: BootStrap can be shared.
As you can see, the call method of ConnectionPoolCall is used to create a Netty connection. Back in the getAndCreateIfAbsent method, continue to see the implementation of the getConnectionPoolAndCreateIfAbsent method.
The method internally wraps the Callable into a FutureTask, the purpose should be to run asynchronously in the future. In short, the run method is called synchronously in the end. Then call the get method to block and wait, waiting for the connection pool returned by the call method just now. Then go back.
Get the connection pool, the connection pool calls the get method, and selects a connection from the pool according to the strategy to return. There is currently only one strategy chosen at random.
This Connection connection instance will be stored in BoltClientTransport.
When the client makes a call, RpcClient will find the corresponding connection according to the URL, and then obtain the Channel corresponding to the connection and send data to the server. The specific code is as follows:
conn.getChannel().writeAndFlush(request).addListener(new ChannelFutureListener() {
@Override
public void operationComplete(ChannelFuture f) throws Exception {
if (!f.isSuccess()) {
conn.removeInvokeFuture(request.getId());
future.putResponse(commandFactory.createSendFailedResponse(
conn.getRemoteAddress(), f.cause()));
logger.error("Invoke send failed, id={}", request.getId(), f.cause());
}
}
});The above is the principle and design of SOFA connection.
Summarize
The connection manager is the last module in our analysis of SOFA-RPC Cluster, which manages all service network connections corresponding to a client.
The connectionHolder contains multiple Maps. The key in the Map is Provider, and the value is ClientTransport. ClientTransport is the glue class between RpcClient and SOFA. Usually, a Provider corresponds to a ClientTransport. ClientTransport is actually a wrapper around a connection.
The way ClientTransport gets the connection is through the connection manager of RpcClient. The connection manager contains a connection factory internally that creates connections based on URLs. The connection is usually created through Netty's BootStrap.
When we use the RpcClient in the ClientTransport corresponding to the Provider to send data, we will find the corresponding Connection according to the URL, get its Channel, and send data to the server.
Well, the above is the analysis of SOFA-RPC connection management.
Space is limited, please correct me if I am wrong.