1, consumers and Partition
The following from "kafak The Definitive Guide," Chapter 4.
Theme T1 assume there are four partitions.
1.1, a consumer group
1.1.1, the number of consumers is less than the number of partitions
Only a consumer, the consumer will receive a message all four partitions.
When there are two consumers, each consumer will receive either a message from the two partitions.
1.1.2, the number of consumers is equal to the number of partitions
When there are four consumers, each consumer can accept a partition of the message.
1.1.3, the number of consumers is greater than the number of partitions
When there are five consumers, consumers will be idle.
1.2, two consumer groups
Between consumer groups are independently of each other, as shown:
2, the partition allocation policy
When consumers join the group, it will determine which partition to partition allocation policy according to which the consumer.
Kafka has two allocation strategies: Range and RoundRobin.
- The Range
Topic of several consecutive Partition assigned to the consumer. - RoundRobin
all Partition Topic allocated one by one to consumers.
3, partition Rebalance (rebalancing)
- New consumers to join the group of consumers
- Consumers withdraw existing consumer groups
- Subscribed threads partition changes
These three cases will trigger the reallocation of partitions, re-allocation process is called Rebalance (rebalancing).
Rebalance to the consumer group brings high availability and scalability, but during Rebalance, consumers can not read the message, the entire group is not available for a short time, and when the partition is re-assigned to another consumer, consumer who reads the current state is lost.
