Kafka抛出异常Offset commit cannot be completed since the consumer is not part of an active group for...

Summary/Zhu Jiqian

During a test of Kafka's consumption process of specifying offset offset through consumer.subscribe(), an exception message appeared due to improper setting of parameters——

2024-01-04 16:06:32.552main[Consumer clientId=consumer-group.id-1, groupId=group.id] Offset commit with offsets {topic-123-0=OffsetAndMetadata{offset=124, leaderEpoch=null, metadata=''}} failed org.apache.kafka.clients.consumer.CommitFailedException: Offset commit cannot be completed since the consumer is not part of an active group for auto partition assignment; it is likely that the consumer was kicked out of the group. at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.sendOffsetCommitRequest(ConsumerCoordinator.java:1109) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.doCommitOffsetsAsync(ConsumerCoordinator.java:929) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsAsync(ConsumerCoordinator.java:896) at org.apache.kafka.clients.consumer.KafkaConsumer.commitAsync(KafkaConsumer.java:1582) at org.apache.kafka.clients.consumer.KafkaConsumer.commitAsync(KafkaConsumer.java:1550) at org.apache.kafka.clients.consumer.KafkaConsumer.commitAsync(KafkaConsumer.java:1527)

This exception translates to "The offset commit cannot be completed because the consumer is not part of the active group for automatic partition allocation; this consumer is most likely kicked out of the group."

This indicates that the consumer group is disconnected.

When this problem occurs, you need to pay attention to a parameter

properties.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, 500)。

This ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG is max.poll.interval.ms, which represents the maximum polling interval. If it is manually set to 500, it means that the consumer can only wait up to 500 milliseconds between two consecutive polls. If this maximum polling time is exceeded, the consumer is considered to have lost its connection, triggering a rebalance operation to distribute it to other consumers.

If this parameter is set to a small value, it may cause frequent rebalancing. If there is no problem with the consumer itself, setting it to a small value may cause the consumer to fail to work properly, and the above exception will be thrown. However, if the setting is too large, the consumer may not be able to process new records for a long time.

Therefore, this parameter needs to be set more reasonably.

At the same time, there is another parameter that needs to be paid attention to——

ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));

This line of code represents an attempt to retrieve a batch of record objects from a Kafka topic within a maximum of 500 milliseconds.

The waiting time between two consecutive polls for consumers is not only related to business processing, but also related to the number of items pulled. If too many items are pulled at one time, the polling time will inevitably become longer.

Simulate the online pull code consumption and processing business logic as follows -

while (true){
    long start = System.currentTimeMillis();
    ConsumerRecords<Integer, String> records = consumer.poll(Duration.ofMillis(500));
    for (ConsumerRecord<Integer, String> record : records){
        //模拟处理业务
        Thread.sleep(10);
        System.out.println("处理业务中");
    }
    long end = System.currentTimeMillis();
    System.out.println("耗时：" + ( end- start) );
    consumer.commitAsync();
}

The size set by max.poll.interval.ms should be based on Duration.ofMillis(500), plus the time it takes for business processing.

Test it and observe the running time:

If the average time-consuming of this processing logic is: 1151 milliseconds, then max.poll.interval.ms should be set larger than 1151 milliseconds. Of course, some additional burst time-consuming situations need to be taken into account.

Anyway, it cannot be smaller than 1151 milliseconds. If it is smaller than 1151 milliseconds, an org.apache.kafka.clients.consumer.CommitFailedException will be thrown.

In addition to adjusting max.poll.interval.ms, which is more time-consuming than consumption logic, you can also adjust consumer.poll(Duration.ofMillis(500)) and max.poll.records to control the time-consuming reduction of each poll processing.