[Kafka] Kafka 1.1.0 consumer group displacement reset

1 Overview

Reprinted: https://www.cnblogs.com/huxi2b/p/7284767.html

This article explains how to use the kafka-consumer-groups.sh script that comes with Kafka to freely set the displacement of the consumer group. What needs special emphasis is that这是0.11.0.0版本提供的新功能且只适用于新版本consumer。

Before the new version, if you want to adjust the displacement for an existing consumer group, you must manually write a Java program to call the KafkaConsumer#seek method, which is time-consuming and labor-intensive and error-prone. Version 0.11.0.0 enriches the functions of the kafka-consumer-groups script. Users can directly use this script to easily reset the displacement for an existing consumer group, but the premise is:consumer group状态必须是inactive的，即不能是处于正在工作中的状态。

Retreat first. In general, the process of resetting displacement consists of 3 steps, as shown in the following figure:

Determine the scope of topic-currently there are 3 ways to specify scope:

--all-topics（为consumer group下所有topic的所有分区调整位移）
--topic t1 --topic t2（为指定的若干个topic的所有分区调整位移）
--topic t1:0,1,2（为指定的topic分区调整位移）

Determine the displacement reset strategy-currently supports 8 setting rules:

--to-earliest：把位移调整到分区当前最小位移
--to-latest：把位移调整到分区当前最新位移
--to-current：把位移调整到分区当前位移
--to-offset <offset>： 把位移调整到指定位移处
--shift-by N： 把位移调整到当前位移 + N处，注意N可以是负数，表示向前移动
--to-datetime <datetime>：把位移调整到大于给定时间的最早位移处，datetime格式是yyyy-MM-ddTHH:mm:ss.xxx，比如2017-08-04T00:00:00.000
--by-duration <duration>：把位移调整到距离当前时间指定间隔的位移处，duration格式是PnDTnHnMnS，比如PT0H5M0S
--from-file <file>：从CSV文件中读取调整策略

Determine the implementation plan-currently supports 3 plans:

什么参数都不加：只是打印出位移调整方案，不具体执行
--execute：执行真正的位移调整
--export：把位移调整方案按照CSV格式打印，方便用户成csv文件，供后续直接使用

Aiming at the above 8 strategies, this article focuses on demonstrating the first 7 strategies.
First, we create a test topic, 5 partitions, and send 5,000,000 test messages:

bin/kafka-topics.sh --zookeeper localhost:2181 --create --partitions 5 --replication-factor 1 --topic topic_lcc

Created topic "topic_lcc".

Simulate sending data

[lcc@lcc ~/soft/kafka/kafka_2.11-1.1.0]$ bin/kafka-producer-perf-test.sh --topic  topic_lcc --num-records 5000000 --throughput -1 --record-size 100 --producer-props bootstrap.servers=localhost:9092 acks=-1
254093 records sent, 50345.4 records/sec (4.80 MB/sec), 279.0 ms avg latency, 675.0 max latency.
525399 records sent, 104828.2 records/sec (10.00 MB/sec), 1454.9 ms avg latency, 2530.0 max latency.
739109 records sent, 147821.8 records/sec (14.10 MB/sec), 2224.5 ms avg latency, 2659.0 max latency.

Then, start a console consumer program and set the group name to test-group:

[lcc@lcc ~/soft/kafka/kafka_2.11-1.1.0]$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic_lcc --from-beginning --consumer-property group.id=test-group

Processed a total of 133596 messages

After running for a period of time, close the consumer program and set the group to inactive. Now run the kafka-consumer-groups.sh script to first determine the consumption progress of the current group:

[lcc@lcc ~/soft/kafka/kafka_2.11-1.1.0]$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group test-group --describe
Note: This will not show information about old Zookeeper-based consumers.
Consumer group 'test-group' has no active members.

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
topic_lcc       0          298155          2064715         1766560         -               -               -
[lcc@lcc ~/soft/kafka/kafka_2.11-1.1.0]$

It can be seen from the above output that the current value of the LAG column of the 5 partitions is not 0, which means that the consumption is not completed. Now we demonstrate how to reset the displacement.

2.–to-earliest

[lcc@lcc ~/soft/kafka/kafka_2.11-1.1.0]$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group test-group --reset-offsets --all-topics --to-earliest --execute
Note: This will not show information about old Zookeeper-based consumers.

TOPIC                          PARTITION  NEW-OFFSET
topic_lcc                      0          164559
[lcc@lcc ~/soft/kafka/kafka_2.11-1.1.0]$

The above output shows that the displacement of all partitions has been reset to 164559

3.–to-latest

[lcc@lcc ~/soft/kafka/kafka_2.11-1.1.0]$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group test-group --reset-offsets --all-topics --to-latest --execute
Note: This will not show information about old Zookeeper-based consumers.

TOPIC                          PARTITION  NEW-OFFSET
topic_lcc                      0          2064715
[lcc@lcc ~/soft/kafka/kafka_2.11-1.1.0]$

The above output shows that the displacement of all partitions has been reset to the latest displacement, which is 2064715

4.–to-offset <offset>

[lcc@lcc ~/soft/kafka/kafka_2.11-1.1.0]$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group test-group --reset-offsets --all-topics --to-offset 10000  --execute
Note: This will not show information about old Zookeeper-based consumers.
[2020-10-28 23:04:13,055] WARN New offset (10000) is lower than earliest offset for topic partition topic_lcc-0. Value will be set to 164559 (kafka.admin.ConsumerGroupCommand$)

TOPIC                          PARTITION  NEW-OFFSET
topic_lcc                      0          164559
[lcc@lcc ~/soft/kafka/kafka_2.11-1.1.0]$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group test-group --reset-offsets --all-topics --to-offset 200000  --execute
Note: This will not show information about old Zookeeper-based consumers.

TOPIC                          PARTITION  NEW-OFFSET
topic_lcc                      0          200000
[lcc@lcc ~/soft/kafka/kafka_2.11-1.1.0]$

The above output shows that the displacement of all partitions has been adjusted to the given 200000. If your value is less than the oldest Offset, it will automatically locate to the oldest position

5.–to-current

[lcc@lcc ~/soft/kafka/kafka_2.11-1.1.0]$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group test-group --reset-offsets --all-topics --to-current --execute
Note: This will not show information about old Zookeeper-based consumers.

TOPIC                          PARTITION  NEW-OFFSET
topic_lcc                      0          200000
[lcc@lcc ~/soft/kafka/kafka_2.11-1.1.0]$

The output indicates that the displacement of all partitions has been moved to the current displacement (this is a bit silly, because the displacement distance has not changed from the previous step)

6.–shift-by N

[lcc@lcc ~/soft/kafka/kafka_2.11-1.1.0]$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group test-group --reset-offsets --all-topics --to-offset 200000  --execute
Note: This will not show information about old Zookeeper-based consumers.

TOPIC                          PARTITION  NEW-OFFSET
topic_lcc                      0          200000
[lcc@lcc ~/soft/kafka/kafka_2.11-1.1.0]$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group test-group --reset-offsets --all-topics --shift-by -1000 --execute
Note: This will not show information about old Zookeeper-based consumers.

TOPIC                          PARTITION  NEW-OFFSET
topic_lcc                      0          199000
[lcc@lcc ~/soft/kafka/kafka_2.11-1.1.0]$

The output shows that the displacement of all partitions is moved to (200000-1000) = 199000

7.–to-datetime

[lcc@lcc ~/soft/kafka/kafka_2.11-1.1.0]$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group test-group --reset-offsets --all-topics --to-datetime 2017-08-04T14:30:00.000
WARN: In a future major release, the default behavior of this command will be to prompt the user before executing the reset rather than doing a dry run. You should add the --dry-run option explicitly if you are scripting this command and want to keep the current default behavior without prompting.
Note: This will not show information about old Zookeeper-based consumers.

TOPIC                          PARTITION  NEW-OFFSET
topic_lcc                      0          164559
[lcc@lcc ~/soft/kafka/kafka_2.11-1.1.0]$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group test-group --reset-offsets --all-topics --to-datetime 2020-10-28T20:00:00.000
WARN: In a future major release, the default behavior of this command will be to prompt the user before executing the reset rather than doing a dry run. You should add the --dry-run option explicitly if you are scripting this command and want to keep the current default behavior without prompting.
Note: This will not show information about old Zookeeper-based consumers.

TOPIC                          PARTITION  NEW-OFFSET
topic_lcc                      0          2064715
[lcc@lcc ~/soft/kafka/kafka_2.11-1.1.0]$

Adjust the displacement of all partitions to the earliest displacement after 2020-10-28T20:00:00.000

8.–by-duration

[lcc@lcc ~/soft/kafka/kafka_2.11-1.1.0]$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group test-group --reset-offsets --all-topics --by-duration PT0H30M0S
WARN: In a future major release, the default behavior of this command will be to prompt the user before executing the reset rather than doing a dry run. You should add the --dry-run option explicitly if you are scripting this command and want to keep the current default behavior without prompting.
Note: This will not show information about old Zookeeper-based consumers.

TOPIC                          PARTITION  NEW-OFFSET
topic_lcc                      0          164571
[lcc@lcc ~/soft/kafka/kafka_2.11-1.1.0]$

Adjust all partition displacements to the earliest displacement 30 minutes ago