kafka之运维命令

其他 2018-10-25 10:41:09 阅读次数: 0
版权声明：本文是作者在学习与工作中的总结与笔记，如有内容是您的原创，请评论留下链接地址，我会在文章开头声明。 https://blog.csdn.net/usagoole/article/details/82587624

文章目录

kafka版本声明

  1. 使用的是kafka 0.10.0.1版本

常用命令

查看kafka版本

  1. 查看kafka版本

    # find / -name \*kafka_\* | head -1 | grep -o '\kafka[^\n]*'

主题

  1. 创建Topic、Partition(5个)，replication(1个)

    # bin/kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 5 --topic testTopic

  2. 查看Topic最大(最小)的offset,--time,-1表示最大，-2表示最小

     # bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 -topic topic-A --time -1

topic-A:2:60
topic-A:4:60
topic-A:1:60
topic-A:3:60
topic-A:0:60

  3. 列出集群中所有的topic

    # bin/kafka-topics.sh --zookeeper zookeeper:2181 --list

  4. 查看Topic明细

     # bin/kafka-topics.sh --describe --zookeeper zookeeper:2181 --topic testTopic

Topic:testTopic	  PartitionCount:5	ReplicationFactor:1 Configs:
Topic: testTopic	Partition: 0	Leader: 1  Replicas: 1 Isr: 1
Topic: testTopic	Partition: 1	Leader: 1  Replicas: 1 Isr: 1
Topic: testTopic	Partition: 2	Leader: 1  Replicas: 1 Isr: 1
Topic: testTopic	Partition: 3	Leader: 1  Replicas: 1 Isr: 1
Topic: testTopic	Partition: 4	Leader: 1  Replicas: 1 Isr: 1

  5. 修改topic参数,partitions个数只能被增加不能减少。如果一定要减少partitions个数，只能删除整个主题，重新建立

     # bin/kafka-topics.sh --zookeeper zookeeper:2181 --partition 6 --topic testTopic --alter

  6. 删除Topic，broker的delete.topic.enable=true才可以删除，否则删除topic的请求会被忽略.删除topic会丢弃topic里所有数据。

    # bin/kafka-topics.sh --zookeeper zookeeper:2181 --delete --topic topicAgroup

生产

  1. 生产数据

    # bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test --producer.config config/producer.properties

消费

  1. 消费

    # bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --new-consumer --from-beginning --consumer.config config/consumer.properties

# bin/kafka-simple-consumer-shell.sh --brist localhost:9092 --topic test --partition 0 --offset 1234  --max-messages 10

  2. 查看消费组offset信息，CURRENT-OFFSET表示最近提交的offset,LOG-END-OFFSET最近一个被提交到集群的offset，LOG-END-OFFSET减去CURRENT-OFFSET可以得到消息积压的个数(LAG的值)

    1. 运行状态查询	
# bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --new-consumer --describe --group topicAGroup

GROUP           TOPIC            PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             OWNER
topicAGroup     topic-A           0          20              60              40     consumer-1_/172.17.0.1
topicAGroup     topic-A           1          20              60              40     consumer-1_/172.17.0.1
topicAGroup     topic-A           2          20              60              40     consumer-1_/172.17.0.1
topicAGroup     topic-A           3          20              60              40     consumer-1_/172.17.0.1
topicAGroup     topic-A           4          20              60              40     consumer-1_/172.17.0.1
	2. 未运行状态查询	
	bash-4.3# bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --new-consumer --describe --group topicAGroup
Consumer group `topicAGroup` does not exist or is rebalancing.

  3. 查看消费组列表

     # bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server localhost:9092 --list

topicAGroup

  4. 读取__consumer_offsets.所有的consumer的offset都以消息的形式写入到这个topic.为了解码这个主题的消息，需要kafka.coordinator.GroupMetadataManager$OffsetsMessageFormatter这个格式化器.通过Math.abs(groupID.hashCode()) % numPartitions确定在__consumer_offsets哪个分区，numPartitions默认是50，这里要查看的groupId是testgroup,Math.abs("testgroup".hashCode()) % 50=27，确定partition为27.输出的结果中有offset(消费者已经提交的偏移量)，此处是100

    # bin/kafka-simple-consumer-shell.sh --topic  __consumer_offsets --partition 27 --broker-list localhost:9092 --formatter "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter"
[testgroup,testTopic,0]::[OffsetMetadata[100,NO_METADATA],CommitTime 1535613323875,ExpirationTime 1535699723875]
[testgroup,testTopic,0]::[OffsetMetadata[100,NO_METADATA],CommitTime 1535613324884,ExpirationTime 1535699724884]
[testgroup,testTopic,0]::[OffsetMetadata[100,NO_METADATA],CommitTime 1535613325888,ExpirationTime 1535699725888]
[testgroup,testTopic,0]::[OffsetMetadata[100,NO_METADATA],CommitTime 1535613326893,ExpirationTime 1535699726893]
[testgroup,testTopic,0]::[OffsetMetadata[100,NO_METADATA],CommitTime 1535613327903,ExpirationTime 1535699727903]
[testgroup,testTopic,0]::[OffsetMetadata[100,NO_METADATA],CommitTime 1535613328912,ExpirationTime 1535699728912]
[testgroup,testTopic,0]::[OffsetMetadata[100,NO_METADATA],CommitTime 1535613329920,ExpirationTime 1535699729920]
....

集群

  1. 查看集群描述

     # bin/kafka-topics.sh --describe --zookeeper zookeeper:2181
Topic:__consumer_offsets	PartitionCount:50	ReplicationFactor:1	Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
	Topic: __consumer_offsets	Partition: 0	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 1	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 2	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 3	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 4	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 5	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 6	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 7	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 8	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 9	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 10	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 11	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 12	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 13	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 14	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 15	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 16	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 17	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 18	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 19	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 20	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 21	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 22	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 23	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 24	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 25	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 26	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 27	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 28	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 29	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 30	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 31	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 32	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 33	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 34	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 35	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 36	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 37	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 38	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 39	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 40	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 41	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 42	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 43	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 44	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 45	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 46	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 47	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 48	Leader: 1	Replicas: 1	Isr: 1
	Topic: __consumer_offsets	Partition: 49	Leader: 1	Replicas: 1	Isr: 1
Topic:testTopic	PartitionCount:5	ReplicationFactor:1	Configs:
	Topic: testTopic	Partition: 0	Leader: 1	Replicas: 1	Isr: 1
	Topic: testTopic	Partition: 1	Leader: 1	Replicas: 1	Isr: 1
	Topic: testTopic	Partition: 2	Leader: 1	Replicas: 1	Isr: 1
	Topic: testTopic	Partition: 3	Leader: 1	Replicas: 1	Isr: 1
	Topic: testTopic	Partition: 4	Leader: 1	Replicas: 1	Isr: 1
Topic:testTopic3	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: testTopic3	Partition: 0	Leader: 1	Replicas: 1	Isr: 1
Topic:topic-A	PartitionCount:5	ReplicationFactor:1	Configs:
	Topic: topic-A	Partition: 0	Leader: 1	Replicas: 1	Isr: 1
	Topic: topic-A	Partition: 1	Leader: 1	Replicas: 1	Isr: 1
	Topic: topic-A	Partition: 2	Leader: 1	Replicas: 1	Isr: 1
	Topic: topic-A	Partition: 3	Leader: 1	Replicas: 1	Isr: 1
	Topic: topic-A	Partition: 4	Leader: 1	Replicas: 1	Isr: 1
Topic:topicAGroup	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: topicAGroup	Partition: 0	Leader: 1	Replicas: 1	Isr: 1
Topic:topicAgroup	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: topicAgroup	Partition: 0	Leader: 1	Replicas: 1	Isr: 1

压测命令

  1. 压测

    # bin/kafka-producer-perf-test.sh --topic test --num-records 100 --record-size 1 --throughput 100  --producer-props bootstrap.servers=localhost:9092

猜你喜欢

转载自blog.csdn.net/usagoole/article/details/82587624
今日推荐
火速冲上 GitHub 热榜 —— 开源编程语言、框架哪有这么可爱？
北京人形机器人创新中心发布全球首个纯电驱拟人奔跑的全尺寸人形机器人“天工”
LFOSSA 源来如此公开课 | 掌握云原生未来：CNCF 认证全面攻略与备考秘籍
国产云输入法——仅华为无云端数据上传安全问题
开源日报 | 工业开源项目OGG 1.0；姐姐，你要和我一起配置火狐吗；苹果AI遥遥落后？Fedora 40
开放签电子签章：停止新增，优化体验，前进更进（五一假期前工作）
开源日报 | 中学生开源前端动画引擎；全球首个Llama3 8B中文版开源模型；联想电脑恐出局；Linus讽刺AI炒作
周排行
浏览器对同一域名进行请求的最大并发连接数
React Hook之自定义Hook
【转】MyBatis缓存机制
-Java-泛型
自动化测试常用脚本-发送邮件
LeetCode#859: Buddy Strings
java、Python处理字符串
第二篇の博客
Hadoop伪分布式环境安装
SQL Server进阶（十一）临时表、表变量
每日归档
更多
2024-04-27(56)
2024-04-26(39)
2024-04-25(22)
2024-04-24(36)
2024-04-23(26)
2024-04-22(39)
2024-04-21(0)
2024-04-20(6)
2024-04-19(5)
2024-04-18(0)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022