Analysis of spark source eight - Spark analysis of the RPC mechanism built four TransportContext analysis and TransportClientFactory

TransportContext

First official documentation of TransportContext as follows:

Contains the context to create a TransportServer, TransportClientFactory, and to setup Netty Channel pipelines with a TransportChannelHandler. There are two communication protocols that the TransportClient provides, control-plane RPCs and data-plane "chunk fetching". The handling of the RPCs is performed outside of the scope of the TransportContext (i.e., by a user-provided handler), and it is responsible for setting up streams which can be streamed through the data plane in chunks using zero-copy IO. The TransportServer and TransportClientFactory both create a TransportChannelHandler for each channel. As each TransportChannelHandler contains a TransportClient, this enables server processes to send messages back to the client on an existing channel.

First, this is a context object created TransportServer, TransportClientFactory, use TransportChannelHandler establish the context netty channel pipeline, which is its three main functions.

TransportClient provides two communication protocols: RPC control plane and data plane "chunk crawl."

User passed via the constructor rpcHandler handles RPC requests. And is responsible for setting rpcHandler streams, which may be used in the form of zero-copy IO streaming data blocks.

TransportServer and TransportClientFactory creates a TransportChannelHandler object for each channel. Each TransportChannelHandler contains a TransportClient, which makes the process server can send back to the client in an existing channel on the news.

Member variables:

1. logger: object is responsible for printing the log

2. conf: TransportConf objects

3. rpcHandler: Examples of RPCHandler

4. closeIdleConnections: idle connection is closed

5. ENCODER: Network layer data encryption, MessageEncoder examples

6. DECODER: decrypting the network layer data, MessageDecoder examples

Three methods:

1. Create TransportClientFactory, two methods are as follows:

 

2. Create TransportServer, four as follows:

 

3. Establish netty channel pipeline, it relates to a method and call the following relationship:

 

Note: TransportClient is established netty channel pipeline is invoked. Whole rpc, only this method can be instantiated TransportClient objects.

TransportClientFactory 

TransportClientFactory 

使用 TransportClientFactory 的 createClient 方法创建 TransportClient。这个factory维护到其他主机的连接池,并应为同一远程主机返回相同的TransportClient。所有TransportClients共享一个工作线程池,TransportClients将尽可能重用。

在完成新TransportClient的创建之前,将运行所有给定的TransportClientBootstraps。

其内部维护了一个连接池,如下:

TransportClientFactory 类图如下:

TransportClientFactory成员变量如下:

1. logger 日志类

2. context 是 TransportContext 实例

3. conf 是 TransportConf 实例

4. clientBootstraps是一个 List<TransportClientBootstrap>实例

5. connectionPool 是一个 ConcurrentHashMap<SocketAddress, ClientPool>实例,维护了 SocketAddress和ClientPool的映射关系,即连接到某台机器某个端口的信息被封装到

6. rand是一个Random 随机器,主要用于在ClientPool中选择TransportClient 实例

7. numConnectionsPerPeer 表示到一个rpcAddress 的连接数

8. socketChannelClass 是一个 Channel 的Class 对象

9. workerGroup 是一个EventLoopGroup 主要是为了注册channel 对象

10. pooledAllocator是一个 PooledByteBufAllocator 对象,负责分配buffer 的 11.metrics是一个 NettyMemoryMetrics对象,主要负责从 PooledByteBufAllocator 中收集内存使用metric 信息

其成员方法比较简单,简言之就是几个创建TransportClient的几个方法。

创建受管理的TransportClient,所谓的受管理,其实指的是创建的对象被放入到了connectionPool中:

创建不受管理的TransportClient,新对象创建后不需要放入connectionPool中:

上面的两个方法都调用了核心方法 createClient 方法,其源码如下:

 

其中Bootstrap类目的是为了让client 更加容易地创建channel。Bootstrap可以认为就是builder模式中的builder。

将复杂的channel初始化过程隐藏在Bootstrap类内部。

至于TransportClient是在初始化channel过程中被初始化的,由于本篇文章长度限制,我们下节剖析。

Guess you like

Origin www.cnblogs.com/johnny666888/p/11135899.html