kakfa common mistakes (long term update)

1. Consumer related

1.1 Consumer group

1.1.1 View consumer group command cannot find consumer group

Details: After running the following command, there is no output on the console interface

kafka-consumer-groups.sh --bootstrap-server 192.168.2.83:9092 --list

As shown below:

Solution:
Specify a consumer group for consumption, and the specified consumer group will be created automatically after successful consumption.

Springboot specifies the consumer group: add the following code to the consumer configuration class (consumer configuration class can view springboot connection kafka )

props.put(ConsumerConfig.GROUP_ID_CONFIG, "kakfa-test");

The result is shown in the figure:

1.2 Theme

1.2.1 kafka default topic _consumer_offsets accidentally deleted

Details: Removed _consumer_offsets topic

As shown below:

Solution :

• Recreate topic __consumer_offsets

bin/kafka-topics.sh --bootstrap-server 192.168.2.86:9092 --topic __consumer_offsets --create

• Restart the kafka service

• to view

  The result is shown in the figure, you can see the topic __consumer_offsets:
  

Unsuccessfully check the kafka service configuration, add:

auto.create.topic.enable=true

2. Related to the publisher

3. Spring Boot connection related

3.1 Consumer related

3.1.1 Connection error

Details: After springboot is started, an error is reported when connecting to the kafka cluster: the coordinator is incorrect, and the marked coordinator is unknown.

Request joining group due to: rebalance failed due to 'This is not the correct coordinator.' (NotCoordinatorException)

As shown below:

Solution

The reason for this problem is generally that the configuration of the consumer class is incorrect, or there is a problem with providing the group id. If you do not know the group id, please display the designated partition to avoid writing the wrong group id.

 @KafkaListener(topicPartitions  =   {
    
     @TopicPartition(topic = "kafka-topic-test", partitions = {
    
     "0", "8" })
    })

3.1.1.2 Consumption error report

Details: The consumption is successful, but an error is reportedThis error handler cannot process 'org.apache.kafka.common.errors.InvalidGroupIdException's; no record information is available

As shown below:

Solution

This prompt is obvious, no available group id is provided. We only need to set one in the consumer configuration. The reason for this error is that the group id is not set, but the subscribe(topic) or Kafka-based offset management strategy is used to use the group management function.