Kafka2.0 server side, start Source

  Kafka server by Kafka.scalathe main function mainmethod starts. KafkaServerStartableClass provides read configuration files, startup method / stop services. The Start / Stop Service is the final call of KafkaServerthe startup/shutdownmethod.

Start Process

1. Start zk client.
2. Start dynamic configuration.
3. Start scheduling thread pool.
4. Start Log Manager background threads, including cleaning up the log, the log disk brush, delete the log, log compression.
5. Start NIO Socket services .
   1. Initializing a sink Acceptor, i.e. start NIO Socket.
   2. Adding num.network.threadsreceivers to the channel request RequestChannelprocessor cache ConcurrentHashMap, key number is incremented, value for the processor Processor.
   3. AcceptorExecution CountDownLatch.awaitwaits for a notification to start.
   4. Cache Acceptorto ConcurrentHashMap, key is EndPoint, value is Acceptor.
6. Start copy manager.
7. Registered broker in zk.
8. The controller.
9. Startup group coordinator.
10. Start transaction coordinator.
11. Initialization KafkaApis.
12. The processor cache initialization thread pool .
    1. Start num.io.threadsrequest processor threads KafkaRequestHandler.
    2. From blocking queue ArrayBlockingQueueacquisition request, calling KafkaApis.handlethe method, for centralized processing request.
13. Start processor thread .
    1. First CountDownLatch.countDownnotice wake up Acceptorthreads.
       1. The use of NIO.selectpolling.
       2. If an event can receive ready, then the current SocketChanneladdition of buffer queueConcurrentLinkedQueue
    2. Taken out from the queue buffer SocketChannel, to bind KafkaChannel.
    3. The received request to the cache limit blocking queue lengthArrayBlockingQueue

Request processing flow

Detailed source code analysis

Acceptor threads

def run() {
  serverChannel.register(nioSelector, SelectionKey.OP_ACCEPT) // 注册接收事件
  startupComplete() // 通知 Acceptor 线程
  var currentProcessor = 0
  while (isRunning) {
    val ready = nioSelector.select(500) // 轮询事件
    if (ready > 0) {
      val keys = nioSelector.selectedKeys()
      val iter = keys.iterator()
      while (iter.hasNext && isRunning) {
        val key = iter.next
        iter.remove()
        if (key.isAcceptable) { // 有可接受事件
          val processor = synchronized {
            currentProcessor = currentProcessor % processors.size
            processors(currentProcessor) // 缓存 Processor 
          }
          accept(key, processor) // 将 SocketChannel 缓存到队列
        }
      }
    }
  }
}

Processor thread

override def run() {
  startupComplete() // CountDownLatch.countDown 唤醒 Acceptor 线程。
  while (isRunning) {
    configureNewConnections() // 从缓存队列取出 SocketChannel，绑定到 KafkaChannel
    processNewResponses() // 处理返回客户端的响应
    poll() // Kafka.Selector 轮询读取/写入事件
    processCompletedReceives() // 处理客户端的请求，放到阻塞队列
    processCompletedSends() // 处理返回客户端响应后的回调
    processDisconnected() // 断开连接后的处理
  }
}

KafkaRequestHandler thread blocking queue

def run() {
  while (!stopped) {
    val startSelectTime = time.nanoseconds
    // 从阻塞队列拉取请求
    val req = requestChannel.receiveRequest(300) 

    req match {
      case request: RequestChannel.Request =>
        try {
          apis.handle(request) // 调用`KafkaApis.handle`方法，进行集中处理请求。
        }
    }
  }
}

KSelector

  Analysis of client reference source.