The relationship between kafka's partition and costomer

 

From the official website:

Each partition is consumed by only one consumer in the same consumer group. And make sure that the consumer is the only consumer of the partition and consumes data in order. If each topic has multiple partitions, load balancing needs to be performed on multiple consumers, but please note that 相同的消费者组中不能有比分区更多的消费者,否则多出的消费者一直处于空等待,不会收到消息 .

 
1. The number of consumers in the same group is greater than the partition

 When the number of consumers in the same group is greater than the partition, some consumers must be idle. That is to say, if there is only one partition, it doesn't matter how many consumers you start in the same group. The number of partitions determines the degree to which this topic can be balanced in the same group.

2. The number of consumers in the same group is less than or equal to the partition

Graphically represented as:

 
The number of consumers 2 is less than the number of partitions 3, at this time, send messages 1, 2, 3, 4, 5, 6, 7, 8, 9 to test2
C1 receives 1, 3, 4, 6, 7, 9
C2 receives At 2 , 5, and 8
, P1 and P2 correspond to C1, that is, multiple partitions correspond to one consumer, and the amount of messages received by C1 is twice that of C2.
Then, another consumer is started in the g3 group to make the number of consumers 3 equals the number of partitions in topic2
    After adding a C3, it can ensure that all messages are consumed in 3 Cs evenly
 
3. Multiple groups of consumer consumption problems
 

Graphically represented as

As shown in the figure above, after sending messages 1, 2, 3, 4, 5, 6, 7, 8, and 9 to test2,
what are the messages received by consumers in group g3 and consumers in group g4?
Answer:
The messages are equally divided by the consumers in the g3 group, and the consumers in the g4 group receive all the messages.
Group g3:
C1 received: 2, 5, 8
C2 received: 3, 6, 9
C3 received: 1, 4, 7
Group g4:
C1 received: 1, 2, 3, 4, 5, 6, 7 , 8, 9
start multiple groups, the same message will be consumed multiple times

Guess you like

Origin http://43.154.161.224:23101/article/api/json?id=326447688&siteId=291194637