1 Overview
Reprinted: https://www.cnblogs.com/huxi2b/p/13278206.html
Kafka 2.6 introduced a new feature: consumers can actively trigger Rebalance. For a long time, the triggering of Rebalance has been executed by the Coordinator, but in some scenarios, it is necessary for the consumer to trigger Rebalance proactively.
For example, there is a subscriptionUserData method in the ConsumerPartitionAssignor interface to implement custom user data. Later, when performing Rebalance, Leader consumers can execute specific logical partition consumption allocation plans based on these customized user data. Currently, if these user data changes, Kafka will not enable Rebalance. If these custom data greatly affect the formulation of the allocation plan, the only way is to call the unsubscribe method of KafkaConsumer to explicitly inform the Coordinator members to leave the group. The disadvantage of this is that the unsubscribe method will reclaim the currently allocated partition.
In response to this scenario, if consumers can actively initiate Rebalance to refresh or submit the latest custom user data in a "do not leave the group" method, then the problems encountered in this scenario can be solved. Based on these considerations, the community introduced this feature in 2.6. Mainly reflected in the increase of the enforceRebalance method in KafkaConsumer. As follows:
@Override
public void enforceRebalance() {
acquireAndEnsureOpen();
try {
if (coordinator == null) {
throw new IllegalStateException("Tried to force a rebalance but consumer does not have a group.");
}
coordinator.requestRejoin();
} finally {
release();
}
}
It can be seen that the enforceRebalance method calls the requestRejon method to request Rebalance, instead of triggering Rebalance by explicitly leaving the group like unsubscribe. The following is the source code of unsubscribe, you can compare and feel it:
public void unsubscribe() {
acquireAndEnsureOpen();
try {
fetcher.clearBufferedDataForUnassignedPartitions(Collections.emptySet());
if (this.coordinator != null) {
this.coordinator.onLeavePrepare();
this.coordinator.maybeLeaveGroup("the consumer unsubscribed from all topics");
}
this.subscriptions.unsubscribe();
log.info("Unsubscribed all topics or patterns and assigned partitions");
} finally {
release();
}
}
It can be seen that the logic of unsubscribe is much more "heavier". It allows consumers to actively initiate a leave-group operation, thereby reclaiming all the partitions assigned to it.
The enforceRebalance method is a non-blocking method, and it does not cause Rebalance by itself. It only sets the boolean variable that identifies whether to rejoin the group to true, so if Rebalance is to be triggered, the poll method needs to be called explicitly. The common usage is as follows:
consumer.enforceRebalance();
consumer.poll(Duration.ofMillis(500));
So, when should this feature be used? Normally, you don't need to use this function, because Coordinator will help you take care of Rebalance. There are two conditions for using this function:
- You have to use the ConsumerPartitionAssignor interface to execute the formulation of the allocation plan
- You need to specify subscriptionUserData and use it to help formulate the distribution plan
If these two conditions are not met at the same time, the use of the enforceRebalance method is meaningless.
Finally, talk about the processing logic of calling this method in two scenarios.
The first case: call the enforceRebalance method when Rebalance has occurred. This is likely to happen, that is, Rebalance has already occurred when the consumer calls enforceRebalance. Consumers may be waiting for Coordinator allocation plan, or waiting for other members to join the group. At this time, we are faced with several choices: 1. Let the current Rebalance complete normally, and then not trigger Rebalance again; 2. 1. Let the current Rebalance complete normally, and then trigger Rebalance again. Obviously, no matter which is reasonable, there is no absolute right or wrong. The current conclusion is that Kafka gives the user the right to trigger again. After the current Rebalance is completed normally, the user can check whether the received distribution plan meets expectations or whether the changed userData is used. Then decide whether to retry the enforceRebalance method just now.
The second situation: The consumer does not belong to the consumer group, possibly because it has been kicked out of the group or has not yet joined the group. If the consumer group is not in the Rebalance process at this time, then the consumer has not had time to send the most timely userData, then the next Rebalance must contain the latest userData. You can try Rebalance again at this time.