版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/a2011480169/article/details/82888172
Kafka常用命令汇总:
1、列出集群里的所有主题。
kafka-topics.sh --zookeeper localhost:2181 --list
[bdp_client@BJLFRZ-Client-50-162 ~]$ kafka-topics.sh --zookeeper localhost:2181 --list
ClearFdmDirtyDataCheckTopic
ClearFdmDirtyData_CheckTopic
ClearFdmDirtyData_Check_Topic
ClearFdmDirtyData_ExeTopic
TaskTag_Check_topic
TaskTag_Exe_topic
2、 创建一个叫作my-topic的主题,主题包含8分区,每个分区拥有两个副本。
kafka-topics.sh --zookeeper localhost:2181 --create --topic my-topic --replication-factor 3 --partitions 3
[bdp_client@BJLFRZ-Client-50-162 ~]$ kafka-topics.sh --zookeeper localhost:2181 --create --topic my-topic --replication-factor 3 --partitions 3
Created topic "my-topic".
3、列出集群里所有主题的详细信息。
kafka-topics.sh --zookeeper localhost:2181 --describe
[bdp_client@BJLFRZ-Client-50-162 ~]$ kafka-topics.sh --zookeeper localhost:2181 --describe
Topic:ClearFdmDirtyDataCheckTopic PartitionCount:1 ReplicationFactor:1 Configs:
Topic: ClearFdmDirtyDataCheckTopic Partition: 0 Leader: 163 Replicas: 163 Isr: 163
Topic:ClearFdmDirtyData_CheckTopic PartitionCount:1 ReplicationFactor:3 Configs:
Topic: ClearFdmDirtyData_CheckTopic Partition: 0 Leader: 163 Replicas: 163,164,135 Isr: 163,164,135
Topic:ClearFdmDirtyData_Check_Topic PartitionCount:1 ReplicationFactor:1 Configs:
Topic: ClearFdmDirtyData_Check_Topic Partition: 0 Leader: 135 Replicas: 135 Isr: 135
Topic:ClearFdmDirtyData_ExeTopic PartitionCount:1 ReplicationFactor:1 Configs:
Topic: ClearFdmDirtyData_ExeTopic Partition: 0 Leader: 164 Replicas: 164 Isr: 164
Topic:TaskTag_Check_topic PartitionCount:3 ReplicationFactor:3 Configs:
Topic: TaskTag_Check_topic Partition: 0 Leader: 135 Replicas: 135,163,164 Isr: 135,163,164
Topic: TaskTag_Check_topic Partition: 1 Leader: 136 Replicas: 136,164,135 Isr: 136,164,135
Topic: TaskTag_Check_topic Partition: 2 Leader: 162 Replicas: 162,135,136 Isr: 162,135,136
Topic:TaskTag_Exe_topic PartitionCount:3 ReplicationFactor:3 Configs:
Topic: TaskTag_Exe_topic Partition: 0 Leader: 164 Replicas: 164,162,163 Isr: 164,162,163
Topic: TaskTag_Exe_topic Partition: 1 Leader: 135 Replicas: 135,163,164 Isr: 135,163,164
Topic: TaskTag_Exe_topic Partition: 2 Leader: 136 Replicas: 136,164,135 Isr: 136,164,135
Topic:__consumer_offsets PartitionCount:50 ReplicationFactor:3 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
Topic: __consumer_offsets Partition: 0 Leader: 136 Replicas: 136,162,163 Isr: 162,163,136
Topic: __consumer_offsets Partition: 1 Leader: 162 Replicas: 162,163,164 Isr: 162,163,164
Topic: __consumer_offsets Partition: 2 Leader: 163 Replicas: 163,164,135 Isr: 163,164,135
Topic: __consumer_offsets Partition: 3 Leader: 164 Replicas: 164,135,136 Isr: 164,135,136
Topic: __consumer_offsets Partition: 4 Leader: 135 Replicas: 135,136,162 Isr: 162,135,136
Topic: __consumer_offsets Partition: 5 Leader: 136 Replicas: 136,163,164 Isr: 163,164,136
Topic: __consumer_offsets Partition: 6 Leader: 162 Replicas: 162,164,135 Isr: 162,164,135
Topic: __consumer_offsets Partition: 7 Leader: 163 Replicas: 163,135,136 Isr: 163,135,136
Topic: __consumer_offsets Partition: 8 Leader: 164 Replicas: 164,136,162 Isr: 162,164,136
Topic: __consumer_offsets Partition: 9 Leader: 135 Replicas: 135,162,163 Isr: 162,163,135
Topic: __consumer_offsets Partition: 10 Leader: 136 Replicas: 136,164,135 Isr: 164,135,136
Topic: __consumer_offsets Partition: 11 Leader: 162 Replicas: 162,135,136 Isr: 162,135,136
Topic: __consumer_offsets Partition: 12 Leader: 163 Replicas: 163,136,162 Isr: 162,163,136
Topic: __consumer_offsets Partition: 13 Leader: 164 Replicas: 164,162,163 Isr: 162,163,164
Topic: __consumer_offsets Partition: 14 Leader: 135 Replicas: 135,163,164 Isr: 163,164,135
Topic: __consumer_offsets Partition: 15 Leader: 136 Replicas: 136,135,162 Isr: 162,135,136
Topic: __consumer_offsets Partition: 16 Leader: 162 Replicas: 162,136,163 Isr: 162,163,136
Topic: __consumer_offsets Partition: 17 Leader: 163 Replicas: 163,162,164 Isr: 162,163,164
Topic: __consumer_offsets Partition: 18 Leader: 164 Replicas: 164,163,135 Isr: 163,164,135
Topic: __consumer_offsets Partition: 19 Leader: 135 Replicas: 135,164,136 Isr: 164,135,136
Topic: __consumer_offsets Partition: 20 Leader: 136 Replicas: 136,162,163 Isr: 162,163,136
Topic: __consumer_offsets Partition: 21 Leader: 162 Replicas: 162,163,164 Isr: 162,163,164
Topic: __consumer_offsets Partition: 22 Leader: 163 Replicas: 163,164,135 Isr: 163,164,135
Topic: __consumer_offsets Partition: 23 Leader: 164 Replicas: 164,135,136 Isr: 164,135,136
Topic: __consumer_offsets Partition: 24 Leader: 135 Replicas: 135,136,162 Isr: 162,135,136
Topic: __consumer_offsets Partition: 25 Leader: 136 Replicas: 136,163,164 Isr: 163,164,136
Topic: __consumer_offsets Partition: 26 Leader: 162 Replicas: 162,164,135 Isr: 162,164,135
Topic: __consumer_offsets Partition: 27 Leader: 163 Replicas: 163,135,136 Isr: 163,135,136
Topic: __consumer_offsets Partition: 28 Leader: 164 Replicas: 164,136,162 Isr: 162,164,136
Topic: __consumer_offsets Partition: 29 Leader: 135 Replicas: 135,162,163 Isr: 162,163,135
Topic: __consumer_offsets Partition: 30 Leader: 136 Replicas: 136,164,135 Isr: 164,135,136
Topic: __consumer_offsets Partition: 31 Leader: 162 Replicas: 162,135,136 Isr: 162,135,136
Topic: __consumer_offsets Partition: 32 Leader: 163 Replicas: 163,136,162 Isr: 162,163,136
Topic: __consumer_offsets Partition: 33 Leader: 164 Replicas: 164,162,163 Isr: 162,163,164
Topic: __consumer_offsets Partition: 34 Leader: 135 Replicas: 135,163,164 Isr: 163,164,135
Topic: __consumer_offsets Partition: 35 Leader: 136 Replicas: 136,135,162 Isr: 162,135,136
Topic: __consumer_offsets Partition: 36 Leader: 162 Replicas: 162,136,163 Isr: 162,163,136
Topic: __consumer_offsets Partition: 37 Leader: 163 Replicas: 163,162,164 Isr: 162,163,164
Topic: __consumer_offsets Partition: 38 Leader: 164 Replicas: 164,163,135 Isr: 163,164,135
Topic: __consumer_offsets Partition: 39 Leader: 135 Replicas: 135,164,136 Isr: 164,135,136
Topic: __consumer_offsets Partition: 40 Leader: 136 Replicas: 136,162,163 Isr: 162,163,136
Topic: __consumer_offsets Partition: 41 Leader: 162 Replicas: 162,163,164 Isr: 162,163,164
Topic: __consumer_offsets Partition: 42 Leader: 163 Replicas: 163,164,135 Isr: 163,164,135
Topic: __consumer_offsets Partition: 43 Leader: 164 Replicas: 164,135,136 Isr: 164,135,136
Topic: __consumer_offsets Partition: 44 Leader: 135 Replicas: 135,136,162 Isr: 162,135,136
Topic: __consumer_offsets Partition: 45 Leader: 136 Replicas: 136,163,164 Isr: 163,164,136
Topic: __consumer_offsets Partition: 46 Leader: 162 Replicas: 162,164,135 Isr: 162,164,135
Topic: __consumer_offsets Partition: 47 Leader: 163 Replicas: 163,135,136 Isr: 163,135,136
Topic: __consumer_offsets Partition: 48 Leader: 164 Replicas: 164,136,162 Isr: 162,164,136
Topic: __consumer_offsets Partition: 49 Leader: 135 Replicas: 135,162,163 Isr: 162,163,135
4、列出集群里特定主题的详细信息。
kafka-topics.sh --zookeeper localhost:2181 --describe --topic my-topic
[bdp_client@BJLFRZ-Client-50-162 ~]$ kafka-topics.sh --zookeeper localhost:2181 --describe --topic my-topic
Topic:my-topic PartitionCount:3 ReplicationFactor:3 Configs:
Topic: my-topic Partition: 0 Leader: 135 Replicas: 135,162,163 Isr: 135,162,163
Topic: my-topic Partition: 1 Leader: 136 Replicas: 136,163,164 Isr: 136,163,164
Topic: my-topic Partition: 2 Leader: 162 Replicas: 162,164,135 Isr: 162,164,135
5、删除一个叫作my-topic的主题。
kafka-topics.sh --zookeeper localhost:2181 --delete --topic my-topic
[bdp_client@BJLFRZ-Client-50-162 ~]$ kafka-topics.sh --zookeeper localhost:2181 --delete --topic my-topic
Topic my-topic is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.
6、列出旧版本的所有消费者群组。
kafka-consumer-groups.sh --zookeeper localhost:2181 --list
7、列出新版本的所有消费者群组。
kafka-consumer-groups.sh --new-consumer --bootstrap-server 172.21.50.162:9092 --list
[bdp_client@BJLFRZ-Client-50-162 ~]$ kafka-consumer-groups.sh --new-consumer --bootstrap-server 172.21.50.162:9092 --list
ClearFDM_Check_ConsumerGroup_1
ClearFDM_Exe_ConsumerGroup_1
consumer_group_TaskTag_Check_topic
consumer_group_TaskTag_Exec_topic
8、获取旧版本消费者群组testgroup的详细信息。
kafka-consumer-groups.sh --zookeeper localhost:2181 --describe --group testgroup
9、获取新版本消费者群组testgroup的详细信息。
kafka-consumer-groups.sh --new-consumer --bootstrap-server 172.21.50.162:9092 --describe --group testgroup
[bdp_client@BJLFRZ-Client-50-162 ~]$ kafka-consumer-groups.sh --new-consumer --bootstrap-server 172.21.50.162:9092 --describe --group consumer_group_TaskTag_Check_topic
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER
consumer_group_TaskTag_Check_topic TaskTag_Check_topic 2 unknown 0 unknown fda52f5c-3d16-434e-8ba7-d26975dfe2a5_/172.21.50.10
consumer_group_TaskTag_Check_topic TaskTag_Check_topic 1 unknown 0 unknown cbd08551-8380-40ff-bcda-da59cfb9480c_/172.21.50.10
consumer_group_TaskTag_Check_topic TaskTag_Check_topic 0 unknown 22 unknown 50d7c78c-7528-4aad-ba94-68724628f2cd_/172.21.50.10
10、查看某一个topic对应的消息数量。
kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 172.21.50.162:9092 --topic my-topic --time -1
[bdp_client@BJLFRZ-Client-50-162 ~]$ kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 172.21.50.162:9092 --topic consumer_group_TaskTag_Check_topic --time -1
consumer_group_TaskTag_Check_topic:0:0
11、查看log日志片段的内容,显示查看日志的内容。
kafka-run-class.sh kafka.tools.DumpLogSegments --files 00000000000000000000.log --print-data-log
[bdp_client@BJLFRZ-Client-50-162 t9-1]$ kafka-run-class.sh kafka.tools.DumpLogSegments --files 00000000000000000000.log
Dumping 00000000000000000000.log
Starting offset: 0
offset: 0 position: 0 CreateTime: 1537343431271 isvalid: true payloadsize: 19 magic: 1 compresscodec: NoCompressionCodec crc: 2502058395 keysize: 13
offset: 1 position: 66 CreateTime: 1537343431272 isvalid: true payloadsize: 19 magic: 1 compresscodec: NoCompressionCodec crc: 2236582723 keysize: 13
offset: 2 position: 132 CreateTime: 1537343431272 isvalid: true payloadsize: 19 magic: 1 compresscodec: NoCompressionCodec crc: 3270504851 keysize: 13
offset: 3 position: 198 CreateTime: 1537343708182 isvalid: true payloadsize: 19 magic: 1 compresscodec: NoCompressionCodec crc: 450329064 keysize: 13
12、控制台生产者:向主题 my-topic 生成两个消息。
kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic
[bdp_client@BJLFRZ-Client-50-162 ~]$ kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic
1
2
3
4
5
6
7
13、控制台消费者:从主题 my-topic 获取消息。
kafka-console-consumer.sh --zookeeper localhost:2181 --topic my-topic --from-beginning
kafka-console-consumer.sh --new-consumer --bootstrap-server 172.21.50.162:9092 --topic my-topic --from-beginning