Brief introduction
The last time we understand by analyzing KafkaProducer the main source of production processes end, and today the server at the network layer are learning what to do, look at the overall architecture diagram KafkaServer
Seen from FIG Kafka's server includes a network layer, the API layer, logging subsystem, subsystem copy of these large modules. When the client initiates the request, the network layer receives a request, and the request into the request queue shared, then removed from the request thread Handler API layer from the queue, and the execution request. For example a request is sent over KafkaProducer production message will log message written to disk, and finally the response is returned to the client
Network layer
From the above chart, you can see Kafka server is still a lot to do, there are a lot of good design, and then slowly introduced behind us, today the main learning network layer
Mainly to complete the network layer and client network connections to other Broker, using a Reactor model based on event-driven mode, written before the relevant articles (links) can refer to not repeat them here
The core layer is the SocketServer type of network, comprising Acceptor for receiving a new connection, a corresponding plurality of Acceptor Processor threads, each thread has its own Selector Processor, used to read and write requests from the connection response back
Acceptor simultaneously a plurality of threads corresponding to threads Handler, this is true processing request thread, the thread processed Handler returns a response to the request the Processor threads, and wherein threads Handler Processor data transfer through the thread RequestChannel, ReqeustChannel Processor and including the shared RequestQueue private ResponseQueue, attached below a picture
It says some abstract, we deep into the source code to see how Kafka server receives the request and the response is returned to the client
Source code analysis
kafkaserv is
Kafka KafkaServer is the main class of server, KafkaServer and network layer related components include SockerServer, KafkaApis and KafkaRequestHandlerPool. And wherein KafkaApis KafkaRequestHandlerPool network requests are processed by RequestChannel SocketServer provided, and the network layer related to the presentation of this code as follows
class KafkaServer(...) {
...
var dataPlaneRequestProcessor: KafkaApis = null
...
def startup(): Unit = {
...
// SocketServer主要关注网络层相关事宜
socketServer = new SocketServer(...)
// Processor涉及到权限相关，所以调用SocketServer.startup时，只先启动Acceptor，初始化完权限相关的凭证后，再启动Processor，所以startupProcessors为false
socketServer.startup(startupProcessors = false)
...
// KafkaApis进行真正的业务逻辑处理
dataPlaneRequestProcessor = new KafkaApis(socketServer.dataPlaneRequestChannel ,...)
// 创建KafkaRequestHandler线程池，其中KafkaRequestHandler主要用来从请求通道中取请求
dataPlaneRequestHandlerPool = new KafkaRequestHandlerPool(config.brokerId, socketServer.dataPlaneRequestChannel, dataPlaneRequestProcessor, config.numIoThreadsx,...)
...
// 初始化结束
...
// Processor涉及到权限相关的操作，所以要等初始化完成，再启动Processor
socketServer.startDataPlaneProcessors()
...
}
}复制代码We then KafkaServer network layer with a specific class shows the FIG.
Specific steps
1.客户端（NetworkClient）发送请求被接收器（Acceptor）转发给处理器（Processor）处理
2.处理器把请求放到请求通道（RequestChannel）的共享请求队列中
3.请求处理器线程（KafkaRequestHandler）从请求通道的共享请求队列中取出请求
4.业务逻辑处理器（KafkaApis）进行业务逻辑处理
5.业务逻辑处理器把响应发到请求通道中与各个处理器对应的响应队列中
6.处理器从对应的响应队列中取出响应
7.处理器将响应的结果返回给客户端复制代码SocketServer.startup方法
By KafkaServer relevant source we know about the overall process flow, since today the main source of learning related to network connectivity, let's look at what has been done SocketServer.startup
// SocketServer.startup
def startup(startupProcessors: Boolean = true): Unit = {
this.synchronized {
// 初始化控制每个IP上的最大连接数的对象，底层是Map
connectionQuotas = new ConnectionQuotas(config, time)
...
// 创建接收器和处理器
createDataPlaneAcceptorsAndProcessors(config.numNetworkThreads, config.dataPlaneListeners)
// 判断是否需要启动处理器，在KafkaServer初始化代用SocketServer.startup方法时，startipProcessors传
// 的是false，原因请看上面介绍KafkaServer的源码
if (startupProcessors) {
// 启动处理器
startControlPlaneProcessor()
...
}
}
...
} 复制代码// 创建接收器和处理器，同时并启动接收器
private def createDataPlaneAcceptorsAndProcessors(dataProcessorsPerListener: Int,endpoints: Seq[EndPoint]): Unit = synchronized {
// 遍历broker节点，为每个broker节点都创建接收器
endpoints.foreach { endpoint =>
connectionQuotas.addListener(config, endpoint.listenerName)
// 创建接收器
val dataPlaneAcceptor = createAcceptor(endpoint, DataPlaneMetricPrefix)
// 创建处理器，并把处理器添加到RequestChannel和Acceptor的处理器列表中
addDataPlaneProcessors(dataPlaneAcceptor, endpoint, dataProcessorsPerListener)
// 启动接收器线程
KafkaThread.nonDaemon(s"data-plane-kafka-socket-acceptor-${endpoint.listenerName}-${endpoint.securityProtocol}-${endpoint.port}", dataPlaneAcceptor).start()
// 等待启动完成
dataPlaneAcceptor.awaitStartup()
// 把启动好的处理器添加到map中，并和broker做好了映射
dataPlaneAcceptors.put(endpoint, dataPlaneAcceptor)
}
}复制代码From the above it can be concluded SocketServer.startup main method creates a receiver and processor, while the receiver is started, but did not start the processor, because the processor will use permissions to wait KafkaServer initialization is complete, it will start a separate processor . Processor to add the created channel list processor and the request of the receiver
Acceptor.run
Now that you created earlier and started the receiver, the receiver that we look at what has been done?
def run(): Unit = {
// 将ServerSocketChannel注册到Selector的OP_ACCEPT事件上
serverChannel.register(nioSelector, SelectionKey.OP_ACCEPT)
// 标志启动完成
startupComplete()
try {
var currentProcessorIndex = 0
while (isRunning) {
try {
// 选择器轮训请求
val ready = nioSelector.select(500)
if (ready > 0) {
// 获取所有的选择键
val keys = nioSelector.selectedKeys()
// 遍历选择键
val iter = keys.iterator()
while (iter.hasNext && isRunning) {
try {
val key = iter.next
iter.remove()
if (key.isAcceptable) {
//OP_ACCEPT事件发生，获取注册到选择键上的ServerSocketChannel，然后调用其accept方法，建立一个客户端和服务端的连接通道
accept(key).foreach { socketChannel =>
// 重试的次数=该接收器对应的处理器的个数
var retriesLeft = synchronized(processors.length)
var processor: Processor = null
do {
// 重试次数减一
retriesLeft -= 1
processor = synchronized {
// 采用轮训的方式把客户端的连接通道分配给处理器即每个处理器都会有多个 SocketChannel，对应多个客户端的连接
currentProcessorIndex = currentProcessorIndex % processors.length
// 根据索引获取对应的处理器
processors(currentProcessorIndex)
}
currentProcessorIndex += 1
// 如果新连接的队列满了，一直阻塞直到最后一个处理器可以能够接收连接
} while (!assignNewConnection(socketChannel, processor, retriesLeft == 0))
}
...复制代码Receiver service side of the main responsible for connecting the receiving client, can be seen from the above source, the receiver thread is started when you registered OP ACCEPT event, when a client initiates a connection, the receiver thread will be able to listen to OP ACCEPT event, then ServerSocketChannel bound to get the selection keys and call accept ServerSocketChannel method of establishing a connection channel client and server, see the following chart
1.服务端ServerSocketChannel在选择器上注册OP_ACCEPT事件
2.客户端在选择器上注册OP_CONNECT事件
3.服务端的选择器轮训到OP_ACCEPT事件，接收客户端的连接
4.服务端的ServerSocketChannel调用accept方法创建和客户端的通道SocketChannel复制代码Processor
By the previous description, we know that the receiver will be in rotation by way of the client SocketChannel assigned to the processor so that the processor corresponding to each of the plurality of SocketChannel, thereby also corresponding to a plurality of clients. The processor will put the receiver assigned SocketChannel own blocking queue, and then wake up its corresponding selector work, the following is the corresponding source
do {
processor = synchronized {
// 采用轮训的方式把客户端的连接通道分配给处理器即每个处理器都会有多个SocketChannel，对应多个客户端的连接
currentProcessorIndex = currentProcessorIndex % processors.length
// 根据索引获取对应的处理器
processors(currentProcessorIndex)
}
currentProcessorIndex += 1
} while (!assignNewConnection(socketChannel, processor, retriesLeft == 0)) //把SocketChannel分配给处理器复制代码private def assignNewConnection(socketChannel: SocketChannel, processor: Processor, mayBlock: Boolean): Boolean = {
if (processor.accept(socketChannel, mayBlock, blockedPercentMeter)) {
...
}复制代码def accept(socketChannel: SocketChannel, mayBlock: Boolean,
acceptorIdlePercentMeter: com.yammer.metrics.core.Meter): Boolean = {
val accepted = {
// 往处理器的阻塞队列中添加SocketChannel
if (newConnections.offer(socketChannel))
true
...
if (accepted)
// 唤醒选择器线程开始轮询，原来的轮询因为没有事件到来被阻塞
wakeup()
accepted
}复制代码Here we look at the run method Processor
override def run(): Unit = {
startupComplete()
try {
while (isRunning) {
// 把新的SocketChannel注册OP_READ事件到选择器上
configureNewConnections()
// 注册OP_WRITE事件，用来给客户端写回响应
processNewResponses()
// 选择器轮训事件，用来读取请求、发送响应，默认超时时间为300ms
poll()
// 处理已经接收完成的客户端请求
processCompletedReceives()
// 处理已经完成的响应
processCompletedSends()
...复制代码Processor.configureNewConnections
private def configureNewConnections(): Unit = {var connectionsProcessed = 0
// 从队列中取出SocketChannel
val channel = newConnections.poll()
// 注册OP_READ事件
// 根据本地的地址和端口、远程客户端的地址和端口构建唯一的connectionId
selector.register(connectionId(channel.socket), channel)
connectionsProcessed += 1
...复制代码Facie register method, has been with the innermost
public void register(String id, SocketChannel socketChannel) throws IOException {
registerChannel(id, socketChannel, SelectionKey.OP_READ);
}复制代码 protected SelectionKey registerChannel(String id, SocketChannel socketChannel, int interestedOps) throws IOException {
// 把SocketChannel注册OP_READ到选择器上，同时绑定SelectionKey
SelectionKey key = socketChannel.register(nioSelector, interestedOps);
// 构建Kafka通道KafkaChannel并将其绑定到选择键上
KafkaChannel channel = buildAndAttachKafkaChannel(socketChannel, id, key);
...
return key;
}复制代码private KafkaChannel buildAndAttachKafkaChannel(SocketChannel socketChannel, String id, SelectionKey key) throws IOException {
// 构建Kafka通道
KafkaChannel channel = channelBuilder.buildChannel(id, key, maxReceiveSize, memoryPool);
// 将Kafka通道绑定到选择键上，这样就可以根据选择键获取到对应的通道
key.attach(channel);
return channel;
...
}复制代码Here configureNewConnections () task (the new SocketChannel registered OP_READ event to the selectors) have been completed
How to look below the processor in response to the result
Processor.processNewResponses
private def processNewResponses(): Unit = {var currentResponse: RequestChannel.Response = null
// 处理器还有响应要发送
while ({currentResponse = dequeueResponse(); currentResponse != null}) {
// KafkaChannel唯一标识
val channelId = currentResponse.request.context.connectionId
try {
currentResponse match {
case response: NoOpResponse =>
...
// 没有响应发送给客户端，需要读取更多请求，注册读事件
tryUnmuteChannel(channelId)
case response: SendResponse =>
// 有响应要发送给客户端，注册写事件
sendResponse(response, response.responseSend)
...复制代码protected[network] def sendResponse(response: RequestChannel.Response, responseSend: Send): Unit = {
val connectionId = response.request.context.connectionId
...
if (openOrClosingChannel(connectionId).isDefined) {
// 将响应通过Selector标记为Send，实际发送通过poll轮询完成
selector.send(responseSend)
// 添加到 inflightResponses 底层是可变的Map key:connectionId value:response
inflightResponses += (connectionId -> response)
}
}复制代码configureNewConnections () and processNewResponses () method after the start of execution poll () method, polling event selector, read request, write a response back, and then processes the received response has been completed
Processor.processCompletedReceives()
private def processCompletedReceives(): Unit = {
selector.completedReceives.asScala.foreach { receive =>
try {
openOrClosingChannel(receive.source) match {
case Some(channel) =>
// 解析ByteBuffer 到ReuestHeader
val header = RequestHeader.parse(receive.payload)
val connectionId = receive.source
// 组装RequestContext对象
val context = new RequestContext(header, connectionId, channel.socketAddress,...)
// 构建 RequestChannel.Request对象 包含了处理器的编号，响应对象中可以获取到，这样就能保证请求和响应的处理都是在同一个处理器中完成
val req = new RequestChannel.Request(processor = id, context = context,...)
// 发送给 RequestChannel 处理
requestChannel.sendRequest(req)
// 移除OP_READ读事件，已经接收到请求了，所以就不用再读了
selector.mute(connectionId)
...复制代码Processor.processCompletedSends()
private def processCompletedSends(): Unit = {
selector.completedSends.asScala.foreach { send =>
try {
// 结束的写请求要从inflightResponses移出
val response = inflightResponses.remove(send.destination).getOrElse {...}
...
// 添加 OP_READ 读事件，这样就可以继续读取客户端发来的请求
tryUnmuteChannel(send.destination)
... 复制代码Here the server and network connections related to the source code has been introduced over, we know that the request processor into the request queue, and get a response back to the client from the response queue, then who is going to deal with the queue request additional request? And who put the response into the response queue processor in it?
Kafaka processing request thread KafkaRequestHandler
In the previous presentation, we know KafkaServer created during initialization request processing thread pool (KafkaRequestHandlerPool), request processing thread pool is created and start request processing thread (KafkaRequestHandler), each request processing threads can have access to a shared request queue, so you can get request processing thread request from the request queue, and then to KafkaApis process.
// KafkaServer 会创建 KafkaRequestHandlerPool，同时把请求队列传过去
dataPlaneRequestHandlerPool = new KafkaRequestHandlerPool(config.brokerId, socketServer.dataPlaneRequestChannel, dataPlaneRequestProcessor, ...)
复制代码// 请求处理线程的数量取决于配置 num.io.threads，默认是8
val runnables = new mutable.ArrayBuffer[KafkaRequestHandler](numThreads)
for (i <- 0 until numThreads) {
// 创建 KafkaRequestHandler Kafka请求处理线程
createHandler(i)
}
def createHandler(id: Int): Unit = synchronized {
// 每个请求处理线程都是共享同一个 RequestChannel
runnables += new KafkaRequestHandler(id, brokerId, aggregateIdleMeter, threadPoolSize, requestChannel, apis, time)
// 启动请求处理线程
KafkaThread.daemon(logAndThreadNamePrefix + "-kafka-request-handler-" + id, runnables(id)).start()
}复制代码def run(): Unit = {
while (!stopped) {
val startSelectTime = time.nanoseconds
// 从请求队列里获取下一个请求或者阻塞到超时
val req = requestChannel.receiveRequest(300)
...
req match {
...
case request: RequestChannel.Request =>
try {
// 交给KafkaApis处理
apis.handle(request)
...复制代码As can be seen the request processing threads task is simply to remove the request from the request queue shared by source code analysis, process the request and then call KafakaApis
The inlet end of the service request processing KafkaApis
When KafkaServer initialized, which a real server request processor
// 全局的服务端入口
dataPlaneRequestProcessor = new KafkaApis(socketServer.dataPlaneRequestChannel,...)复制代码handle request processing thread calls KafkaApis method of handling business logic
def request.header.apiKey match {
// 处理客户端生产消息的请求
case ApiKeys.PRODUCE => handleProduceRequest(request)
case ApiKeys.FETCH => handleFetchRequest(request)
...复制代码Kafka visible server-side request processing entrance KafkaApis Depending on the type of processor selection request, as the server for these requests did what we share next time
Reference material
1. "Kafka Inside"
2. "Apache Kafka source analysis"
3.Kafka latest trunk branch code