Spring Kafka's Offset submission timing

1. Premise

The spring-kafka1.3 version is used in the project, and the 2.5 version is also used. However, the timing of submitting the offset is vague, and this time it is further clarified through source code analysis and data.

2. Know the offset of KafkaConsumer

KafkaConsumer is an entrance of the Kafka client. Through KafkaConsumer, data on the Kafka service can be pulled, heartbeat packets can be sent, and the offset (offset) of the consumption partition can be reported.

In order to ensure that records that have not been read by consumers can always be returned when calling the KafkaConsumer.poll() method, consumers need to maintain the offset offset corresponding to the read messages in each partition, and send the offset to Kafka by mobile phone or regularly The server, so that the message is not lost.

There are two main categories for submitting offsets, one is automatic and the other is manual. Controlled by the enable.auto.commit parameter.

When =true, combined with the auto.commit.interval.ms parameter, the consumer will check whether the offset should be submitted each time it polls, and automatically submit the offset returned by the latest poll.

When =false, the developer needs to submit the offset manually, see the following code:

  Properties properties = new Properties();
  properties.setProperty("bootstrap.servers", "10.74.20.54:9092");
  properties.setProperty("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
  properties.setProperty("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
  properties.setProperty("enable.auto.commit", "false"); // 禁止自动提交
  properties.setProperty("group.id", "my-group"); // 

  KafkaConsumer<String, String> consumer = new KafkaConsumer<>(properties);
  consumer.subscribe(Collections.singletonList("topic")); // 订阅主题
  try {
      while (! Thread.currentThread().isInterrupted()) {
          ConsumerRecords<String, String> records = consumer.poll(1000);
          records.forEach(System.out::println);
          consumer.commitAsync(); // 异步提交偏移量
      }
  } catch (WakeupException ignore) {
      // 忽略关闭异常
  } finally {
      try {
          consumer.commitSync(); // 同步提交偏移量
      } finally {
          consumer.close();
      }
  }

Here it is divided into asynchronous submission or synchronous submission:

Synchronous submission: Using KafkaConsumer.commitSync() will submit the latest offset and wait for the broker to respond to the submission request. Continuous retrying until a successful commit or an unrecoverable error is encountered will cause the application to remain blocked, limiting the throughput of the application.

Asynchronous submission: Using KafkaConsumer.commitAsync() will submit the latest offset without waiting for the broker's response and without retrying. No retry is done because there may be a larger offset that has been submitted successfully, and the retry may overwrite the latest value, resulting in duplicate messages after rebalancing. This method executes a user-specified callback when the broker responds. Callbacks are often used to log commit errors or generate metrics. However, if you want to retry in it, you must pay attention to the order of submission.

From the perspective of performance: adopt the form of the above code, asynchronous + synchronous

3. Implementation of Spring-kafka

After spring-kafka 2.3 enable.auto.commit is set to false by default (true before this version), and AckMode defaults to batch. By setting the value to false, kafkaConsumer no longer automatically submits the offset (offset). Instead, use spring to help us achieve it.

That sping is how to achieve? From the figure below, we can see that KafkaMessageListenerContainer.run() implements a while loop internally, and then calls pollAndInvoke() to complete the data reading. At the top of the pollAndInvoke method, it is judged whether it is a manual submission, and if it is, submit the last time The offset to read.