1 Overview
This article is a supplement to https://www.cnblogs.com/huxi2b/p/11262995.html
Kafka Broker uses the Reactor model to process requests. Each Broker has an Acceptor thread similar to Dispatcher, and several Processor threads that process requests (Of course, the thread that actually processes the request logic is not the Processor, but actually KafkaRequestHandler). After each Processor thread is started, it roughly does the following things:
-
Set up a new inbound connection
-
Process the new request response (the so-called processing is put into the response queue)
-
Perform the Selector.select operation to get those prepared IO operations
-
Receive new inbound request
-
Execute the callback logic of the sent response
-
Deal with disconnected
After each Broker starts, the Processor thread it creates will continue to perform the above actions, cyclically, until the Broker is closed.
Let's focus on the logic in the first step. The following is the source code of version 1.1.1 (the selection of version 1.1.1 is not intentional, in fact, all versions before 2.3 are similar):
/**
* Register any new connections that have been queued up
*/
private def configureNewConnections() {
while (!newConnections.isEmpty) {
val channel = newConnections.poll()
try {
debug(s"Processor $id listening to new connection from ${channel.socket.getRemoteSocketAddress}")
selector.register(connectionId(channel.socket), channel)
} catch {
// We explicitly catch all exceptions and close the socket to avoid a socket leak.
case e: Throwable =>
val remoteAddress = channel.socket.getRemoteSocketAddress
// need to close the channel here to avoid a socket leak.
close(channel)
processException(s"Processor $id closed connection from $remoteAddress", e)
}
}
}
Pay attention to the sentence I marked in red. Basically, the way the Processor thread sets up a new inbound connection is to process it all at once. The newConnections in the code is an instance of java.util.concurrent.ArrayBlockingQueue. The Acceptor thread will also access newConnections, so it must be thread-safe.
This kind of one-time processing before closing is risky in some cases. For example, when the Kafka cluster encounters a DDOS attack, the external IP will create a large number of inbound connections and smash them into newConnections. At this time, the processor thread will always try to consume these new connections while it is running, otherwise it will not do other things-such as processing requests. In other words, Kafka currently processes new inbound connections with higher priority than existing connections. When encountering a connection storm, the Kafka Broker will give priority to new connections, which may cause the processing of requests on existing connections to be suspended and eventually lead to timeouts. In this way, the client will further send new requests after being notified of the request timeout, resulting in an avalanche effect.
In addition, it is not without overhead for the Broker to maintain each connection. The connection information itself must take up some content resources. If it is an SSL-enabled connection, Kafka maintains an additional 48KB temporary buffer for it. Therefore, OOM errors are very common in the event of a connection storm.
For these reasons, the community has improved the way the Broker handles new connection requests in version 2.3. First of all, the number of new connections stored in the blocking queue is no longer unlimited, but is fixed at 20, that is, the maximum number of new connection queues for each processor is 20 connections-this is hard-coded in the code and cannot be modified at present . Second, the community introduced a new parameter max.connections to control the maximum number of connections allowed on the Broker side. You can adjust this parameter to control how many inbound connections a Broker can receive at most. This parameter can be set in server.properties, or it can be dynamically modified using the kafka-configs script. max.connections is global, and you can also set a different upper limit for the number of connections for each listener. For example, if your listener uses both PLAINTEXT and SSL, then you can use listener.name.plaintext.max.connections and listener.name.ssl.max.connections to configure the number of connections for these two listeners. The command is as follows :
$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-name 0 --alter --add-config max.connections=100$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-name 0 --alter --add-config listener.name.plaintext.max.connections=80
Completed updating config for broker: 0.
$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-name 0 --alter --add-config listener.name.ssl.max.connections=80
Completed updating config for broker: 0.
The third is that each Processor thread of Kafka Broker will try to close redundant connections before the end of each round of tasks. There are two reasons for judging whether you need to close redundant connections: 1. The total number of connections exceeds the max.connections value; 2. You have set up multiple listeners for the Broker, but Kafka will protect the listener used by the Broker internal connections. For example, if you set up multiple listeners: PLAINTEXT://9092, SSL://9093, SASL://9094, and then set inter.broker.listener.name=SSL, then the connection under the SSL listener is It will not be forcibly closed by the Processor.
Finally, if the blocking queues of all processors are full, the previous Acceptor thread will be blocked and will not receive any more inbound requests. The community has added a new JMX indicator to calculate the percentage of time that Acceptor threads are blocked: kafka.network:type=Acceptor,name=AcceptorBlockedPercent,listener={listenerName}
2. Kafka 2.3 code
/**
* Register any new connections that have been queued up. The number of connections processed
* in each iteration is limited to ensure that traffic and connection close notifications of
* existing channels are handled promptly.
*
* 注册任何已排队的新连接。每个迭代中处理的连接数量有限，以确保及时处理现有通道的流量和连接关闭通知。
*
* 从并发队列Q1里取出SocketChannel,添加到自身的nio selector中,监听读事件;
* 队列中有新的SocketChannel，首先为通道注册OP_READ事件到统一的选择器上
*/
private def configureNewConnections() {
var connectionsProcessed = 0
// 队列中有新的SocketChannel，遍历newConnections队列
while (connectionsProcessed < connectionQueueSize && !newConnections.isEmpty) {
// 从队列中弹出SocketChannel，一个SocketChannel只会注册一个
val channel = newConnections.poll()
try {
debug(s"Processor $id listening to new connection from ${channel.socket.getRemoteSocketAddress}")
// connectionId 从通道中获取本地服务端和远程客户端的地址和端口，构造唯一的 connectionId
// selector.register 注册通道的读事件
selector.register(connectionId(channel.socket), channel)
connectionsProcessed += 1
} catch {
// We explicitly catch all exceptions and close the socket to avoid a socket leak.
case e: Throwable =>
val remoteAddress = channel.socket.getRemoteSocketAddress
// need to close the channel here to avoid a socket leak.
close(listenerName, channel)
processException(s"Processor $id closed connection from $remoteAddress", e)
}
}
}