Scene: in a thread, and the same group id brokers like configuration, according to the number of incoming topic N, respectively, define N Consumer, according to the definition of the order of successively calling consumers consumption
Phenomenon: After the program starts, kafka consumer thread to initialize properly, try calling the consumer poll, there has been no return message program starts about 10 minutes after the start of normal consumption.
Positioning Analysis:
After the program starts, it found no consumption
1> View kafka debug log
Sending Heartbeat request to coordinator 192.168.44.88:9191 (id: 2147483647 rack: null)
Attempt to heartbeat failed since group is rebalancing
2> using kafka own script to see consumer group
bin/kafka-consumer-groups.sh --bootstrap-server 192.168.44.88:9191 --describe --group sample-consumer-group
发现Warning: Consumer group 'sample-consumer-group' is rebalancing.
The topic LAG is not 0, that is the message backlog.
3> after trying to update kafka cluster and increase the topic partitions test found that the problem still exists, identified as procedural issues.
4> to try to reduce the number of consumer and topic, only to pass a topic, normal consumption.
Incoming three topic and spread into a topic, for each topic, the only one consumer consumption of a topic, which is why the former is not normal two normal? Why the former will lead to kafka group rebalancing?
Comparison test conditions vary, the initial guess is the same group id lead.
After passing three topic and were let consumers use three different groupid consumption, we found that normal consumption, the problem is solved.
Search for information, find a similar situation, reference
https://www.gonever.com/post/108
TODO: zookeeper data analysis and kakfa source.