Kafka consumer group displacement Reset

    This article explains how to use Kafka's own kafka-consumer-groups.sh scripts are free to set the displacement of consumer groups (consumer group) is. It needs to be stressed that this is a new feature version 0.11.0.0 offers and applies only to new versions consumer.

    Prior to the new version, calling KafkaConsumer # seek method if you want to adjust the displacement of existing consumer group had to manually write Java programs, not that time consuming and error-prone. 0.11.0.0 version rich functionality kafka-consumer-groups scripts, users can directly use the script to easily reset the displacement of existing consumer group, but the premise must be consumer group must be inactive, that is not in being work state.

    To retreat a bit. In general, the process of resetting a 3-step displacement of the composition, as shown below:

  • Determining topic Scope - Scope currently has three targeting methods: - all-topics (all partitions for all the topic of the displacement adjustment consumer group), - topic t1 --topic t2 (for the specified number of topic displacement adjusting all partitions), - topic t1: 0,1,2 (partition adjusting path is specified topic)

  • To determine the displacement Reset Strategy - eight kinds of set rules currently supports:

    • --to-earliest: the displacement adjusting to the current minimum displacement partition

    • --to-latest: the shift to adjust to the current date shift partition

    • --to-current: the shift to adjust to the current displacement partition

    • --to-offset <offset>: the displacement of the displacement adjusting to the specified

    • --shift-by N: the displacement of the displacement adjusting to the current + N, the N may be noted that negative indicating forward movement

    • --to-datetime <datetime>: to be greater than the displacement of the first displacement at a given time, datetime format yyyy-MM-ddTHH: mm: ss.xxx, such as 2017-08-04T00: 00: 00.000

    • --by-duration <duration>: the displacement distance is adjusted to the specified interval at a current time displacement, duration format PnDTnHnMnS, such PT0H5M0S

    • --from-file <file>: read adjustment strategy from a CSV file

  • Determine the implementation of the program - currently supports three kinds of programs:

    • What parameters do not add: just print out the displacement adjustment programs, not the specific implementation

    • --execute: perform the actual displacement adjustment

    • --export: The print program according to the displacement adjusting CSV format, user into a csv file, for subsequent direct use

    For eight kinds of strategies above, the focus of this article demonstrate in front of seven kinds of tactics. First, we create a test topic, 5 partitions, and send test messages 5,000,000 Article:

> bin/kafka-topics.sh --zookeeper localhost:2181 --create --partitions 5 --replication-factor 1 --topic test

Created topic "test".

> bin/kafka-producer-perf-test.sh --topic test --num-records 5000000 --throughput -1 --record-size 100 --producer-props bootstrap.servers=localhost:9092 acks=-1

 

1439666 records sent, 287760.5 records/sec (27.44 MB/sec), 75.7 ms avg latency, 317.0 max latency.
1541123 records sent, 308163.0 records/sec (29.39 MB/sec), 136.4 ms avg latency, 480.0 max latency.
1878025 records sent, 375529.9 records/sec (35.81 MB/sec), 58.2 ms avg latency, 600.0 max latency.
5000000 records sent, 319529.652352 records/sec (30.47 MB/sec), 86.33 ms avg latency, 600.00 ms max latency, 38 ms 50th, 319 ms 95th, 516 ms 99th, 591 ms 99.9th.

 然后,启动一个console consumer程序,组名设置为test-group:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --consumer-property group.id=test-group

..............

待运行一段时间后关闭consumer程序将group设置为inactive。现在运行kafka-consumer-groups.sh脚本首先确定当前group的消费进度:

bogon:kafka_0.11 huxi$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group test-group --describe
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).

TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
test 0 1000000 1000000 0 consumer-1-8688633a-2f88-4c41-89ca-fd0cd6d19ec7 /127.0.0.1 consumer-1
test 1 1000000 1000000 0 consumer-1-8688633a-2f88-4c41-89ca-fd0cd6d19ec7 /127.0.0.1 consumer-1
test 2 1000000 1000000 0 consumer-1-8688633a-2f88-4c41-89ca-fd0cd6d19ec7 /127.0.0.1 consumer-1
test 3 1000000 1000000 0 consumer-1-8688633a-2f88-4c41-89ca-fd0cd6d19ec7 /127.0.0.1 consumer-1
test 4 1000000 1000000 0 consumer-1-8688633a-2f88-4c41-89ca-fd0cd6d19ec7 /127.0.0.1 consumer-1

由上面输出可知,当前5个分区LAG列的值都是0,表示全部消费完毕。现在我们演示下如何重设位移。

1. --to-earliest

bogon:kafka_0.11 huxi$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group test-group --reset-offsets --all-topics --to-earliest --execute
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).

TOPIC PARTITION NEW-OFFSET 
test 0 0 
test 1 0 
test 4 0 
test 3 0 
test 2 0

上面输出表明,所有分区的位移都已经被重设为0

2. --to-latest

bogon:kafka_0.11 huxi$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group test-group --reset-offsets --all-topics --to-latest --execute
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).

TOPIC PARTITION NEW-OFFSET 
test 0 1000000 
test 1 1000000 
test 4 1000000 
test 3 1000000 
test 2 1000000

上面输出表明,所有分区的位移都已经被重设为最新位移,即1,000,000

3.  --to-offset <offset>

bogon:kafka_0.11 huxi$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group test-group --reset-offsets --all-topics --to-offset 500000 --execute
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).

TOPIC PARTITION NEW-OFFSET 
test 0 500000 
test 1 500000 
test 4 500000 
test 3 500000 
test 2 500000

上面输出表明,所有分区的位移都已经调整为给定的500000

4.  --to-current

bogon:kafka_0.11 huxi$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group test-group --reset-offsets --all-topics --to-current --execute
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).

TOPIC PARTITION NEW-OFFSET 
test 0 500000 
test 1 500000 
test 4 500000 
test 3 500000 
test 2 500000

输出表明所有分区的位移都已经被移动到当前位移(这个有点傻,因为位移距上一步没有变动)

5. --shift-by N

bogon:kafka_0.11 huxi$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group test-group --reset-offsets --all-topics --shift-by -100000 --execute
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).

TOPIC PARTITION NEW-OFFSET 
test 0 400000 
test 1 400000 
test 4 400000 
test 3 400000 
test 2 400000

输出表明所有分区的位移被移动到(500000 - 100000) = 400000处

6. --to-datetime

bogon:kafka_0.11 huxi$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group test-group --reset-offsets --all-topics --to-datetime 2017-08-04T14:30:00.000
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).

TOPIC PARTITION NEW-OFFSET 
test 0 1000000 
test 1 1000000 
test 4 1000000 
test 3 1000000 
test 2 1000000

将所有分区的位移调整为2017年8月4日14:30之后的最早位移

7. --by-duration

bogon:kafka_0.11 huxi$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group test-group --reset-offsets --all-topics --by-duration PT0H30M0S
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).

TOPIC PARTITION NEW-OFFSET 
test 0 0 
test 1 0 
test 4 0 
test 3 0 
test 2 0

将所有分区位移调整为30分钟之前的最早位移

 

文章转载自:http://muxiulin.cn/archives/254

Guess you like

Origin www.cnblogs.com/chongaizhen/p/11206133.html