Kafka producer network layer summary

1 Hierarchy

NetworkClient is responsible for making network IO requests. The main hierarchy is as follows

ClusterConnectionStates reports the status of each node, with node as key and node status as value; inFlightRequets saves requests that have been sent by each node, but has not yet returned, with node as key and List<ClientRequest > is value. inFlightRequets can also be seen from the name, which means "flying in the air" request.

2 How to ensure that only one request is sent at a time

After the sender thread is started, if there is a message in the RecordBatch, the message will be rearranged according to the node where it is located, and each node will create a ClientRequest for sending. Each node can only send one ClientRequest at a time, as follows
KafkaChannel#setSend(.. )

public void setSend(Send send) {
        if (this.send != null) // 如果已经有send，会抛出异常
            throw new IllegalStateException("Attempt to begin a send operation with prior send operation still in progress.");
        this.send = send;
        this.transportLayer.addInterestOps(SelectionKey.OP_WRITE);
 }

So how does kafka ensure that there is already send in the KafkaChannel when setSend is avoided? The key is that NetworkClient#ready(..) will be called in the sender thread, and the node that is not ready will be removed, so that it will not be on the node. setSend:

while (iter.hasNext()) {
            Node node = iter.next();
            if (!this.client.ready(node, now)) { // 关键
                iter.remove();
                notReadyTimeout = Math.min(notReadyTimeout, this.client.connectionDelay(node, now));
            }
}

3 NetworkClient#ready(..)

NetworkClient#ready(..) checks whether the node is ready to determine whether the message can be encapsulated as a ClientRequest and placed on the KafkaChannel.

public boolean ready(Node node, long now) {
        if (node.isEmpty())
            throw new IllegalArgumentException("Cannot connect to empty node " + node);

        if (isReady(node, now)) // 关键
            return true;

        if (connectionStates.canConnect(node.idString(), now))
            initiateConnect(node, now);

        return false;
    }

Let's analyze isReady

public boolean isReady(Node node, long now) {
        return !metadataUpdater.isUpdateDue(now) && canSendRequest(node.idString());
    }

isReady has two main conditions, one is to judge whether it is time to update the metadata. If the metadata needs to be updated, then this request will not be sent, that is, the metadata update has a high priority. The second is to determine whether the node canSendRequest.

private boolean canSendRequest(String node) {
        return connectionStates.isConnected(node) && selector.isChannelReady(node) 
        && inFlightRequests.canSendMore(node); // 重点
    }

The inFlightRequests report saves the request for "flying in the air"

public boolean canSendMore(String node) {
        Deque<ClientRequest> queue = requests.get(node);
        return queue == null || queue.isEmpty() ||
               (queue.peekFirst().request().completed() && queue.size() < this.maxInFlightRequestsPerConnection);
    }

If the following conditions are met, it means that you can continue to send

1. The queue is empty, that is, the node has no "flying in the air" request
2. queue is not empty. The first request in the queue has been completed and the size of the queue is less than the maximum allowed. How to understand it? The queue is a double-ended queue, which is inserted at the head of the queue every time it is set, so the first one in the queue is the one being sent, or the send in the KafkaChannel. Only when send is sent to the network can it continue to send. This guarantees the aforementioned "how to ensure that only one request is sent at a time".

4 Reference

1. https://blog.csdn.net/chunlongyu/article/details/52651960