Kafka consumer: preparation for consumer polling (15)

• Before pulling messages, you need to make sure
  • The client has connected to the coordinator
  • The consumer receives the partition assigned to it by the coordinator
• Preparations for consumers to pull messages
  • connection coordinator
  • Send a request to the coordinator to join the consumer group, and get the assigned partition from the coordinator

1. The sequence before consumers pull messages

• The consumer applies to the coordinator to join the consumer group
• There is a coordinator who manages consumers on the server side, and the coordinator adds consumers to the consumer group
• The coordinator assigns partitions to all consumers
• The consumer gets the partitions assigned to it from the coordinator
• Consumers pull messages from partitions

2. Send a request and get the result

• Each consumer has to send a JoinGroupRequest (request to join the consumer group) to the coordinator, and each consumer gets the assigned partition from the JoinGroupResponse (response to join the consumer group)
• The pseudo code of the process is as follows, the real code is in the joinGroupIfNeeded method of AbstractCoordinator

// 消费者向协调者发送加入消费组的请求，并得到分配分区
// 发送 JoinGroup 请求
JoinGroupResponse joinResult = sendJoinGroupRequest();
// 获取分区
Assignment assignment = partitionAssignment(joinResult);
// 更新状态
subscriptions.assignFromSubscribed(assignment.partitions());

• The actual situation is to wait for all consumers to join the consumer group before executing the algorithm for allocating partitions, so what is returned at the beginning is an asynchronous object Future

// 在异步请求对象上轮询，如果没有完成，会一直循环
RequestFuture<JoinGroupResponse> future = sendJoinGroupRequest();
client.poll(future);
Assignment assignment = partitionAssignment(future.get());

3. Consumers join the consumer group

• Call ensureActiveGroup to join the consumer step
  • onJoinPrepare in the preparation phase
  • sendJoinGroupRequest that initiates the request
  • onJoinComplete for successful join

// 进行加入前准备
onJoinPrepare(generation.generationId, generation.memberId);
// TODO 发送一个注册请求
joinFuture = sendJoinGroupRequest();
// 成功加入消费组
onJoinComplete(generation.generationId, generation.memberId, generation.protocol, future.value());

• Note: The member number when the request is initiated is UNKNOWN_MEMBER_ID, and an actual member number will be returned after the first join, and the actual member number needs to be specified when rejoining or rebalancing occurs
• The logic for consumers to connect to the coordinator, send "join consumer group" requests, and obtain partitions are all in AbstractCoordinator

public abstract class AbstractCoordinator implements Closeable {
    
    
	// 准备加入消费组
	private boolean rejoinNeeded = true;
	// 是否需要重新加入消费组
	private boolean needsJoinPrepare = true;
	protected synchronized boolean needRejoin() {
    
    
        return rejoinNeeded;
    }
	public void ensurePartitionAssignment(){
    
    
		// 根据消费者订阅状态的变量，来判断是否需要执行重新加入消费组的逻辑
		if(subscriptions.partitionsAutoAssigned())
			ensureActiveGroup();
	}
	public void ensureActiveGroup(){
    
    
		if(!needRejoin()) return;
		if (needsJoinPrepare) {
    
    
		    // 进行加入前准备
		    onJoinPrepare(generation.generationId, generation.memberId);
		    needsJoinPrepare = false;
		}
		while (needRejoin() || rejoinIncomplete()) {
    
    
			// 再次判断是否已经确定好了 coordinator
			ensureCoordinatorReady();
			// TODO 发送加入消费者请求
			RequestFuture<ByteBuffer> future = sendJoinGroupRequest();
			client.poll(future);
			if (future.succeeded()) {
    
    
                needsJoinPrepare = true;
                // 成功加入消费组
                onJoinComplete(generation.generationId, generation.memberId, generation.protocol, future.value());
            }
		}
	}
}

• Make sure to assign to the partition to execute the process
  • The initial needJoinPrepare and rejoinNeeded are both true, and the consumer will join the consumer group by default when starting
  • To meet needRejoin(), first call onJoinPrepare() to make preparations, such as submitting offsets, etc.
  • After the preparation work is completed, modify needsJoinPrepare=false to prevent the preparation work from being performed multiple times before joining the consumer group
  • Satisfy needRejoin(), execute the loop body: send a request, call the client polling once to try to get the result
  • Consumers are assigned to partitions, update rejoinNeeded to false, and reset needsJoinPrepare to true
• The onJoinComplete() method will update the needsPartitionsAssigned of the subscription status to false after the consumer successfully joins the consumer group and assigns it to a partition. This variable is to determine whether to rejoin the consumer group
• When the consumer starts, it registers listeners for different events (such as partition changes, consumer member changes, and session timeouts) with ZK. When the registered events occur, it will trigger the rebalancing operation of ZKRebalanceListener