Balance weight (rebalance of)
Relying on the old version of Kafka Zk be rebalance, the new version of the consumer using the built-in Kafka a whole new set of coordination protocols. For each group, a broker Kafka will be elected as the group coordinator (coordinator).
Triggering conditions:
1. The members of the group changed.
2. group subscriptions topic changed. Such as the use of subscription based on regular expressions, regular expression matching when a new topic is created it will trigger rebalance
3. The number of partitions of the group subscribe to topic changed. For example, use the command-line script to increase the number of partitions of subscription topic.
Partitioning strategy:
range, round-robin, sticky strategy
rebalance generation：
After the session members expressed rebalance, it is an integer in the consumer, starting at 0. Mainly to protect consumer group, in particular to prevent invalid offset submitted, such as the last member of the consumer for some reason delayed submission of the offset, but then rebalance the group created the new group members, and this new delay offset submit information to carry an older generation, so this submission will be rejected consumer group.
rebalance the agreement:
JoinGroup request: consumer requests to join the group
SyncGroup request: group leader synchronized to the allocation scheme all members of the group
Heartbeat request: consumer report regularly show their heartbeat to the coordinator still alive
LeaveGroup request: consumer initiative to inform the coordinator of the conumer about to leave the group.
DescribeGroup request: for administrators, see the group all the information, including membership information, protocol information, allocation and subscription information
在成功rebalance之后，组内所有consumer都需要定期向coordinator发送Heartbeat请求，而每个consumer也是根据Heartbeat请求的响应中是否包含REBALANCE_IN_PROGRESS来判断当前group是否开启了新一轮rebalance