问题描述:
storm 起动topology的在stormui中可以看到类似错误
storm 的KafkaSpout 接收kafka 消息时报错，错误如下
Unable to get offset lags for kafka. Reason: java.lang.IllegalArgumentException: zk-node '/kafka_storm/webTraceId' dose not exists. at org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.getOldConsumerOffsetsFromZk(KafkaOffsetLagUtil.java:387) at org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.getOffsetLags(KafkaOffsetLagUtil.java:268) at org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.main(KafkaOffsetLagUtil.java:124)
或者如下
Unable to get offset lags for kafka. Reason: java.lang.NullPointerException at org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.getOffsetLags(KafkaOffsetLagUtil.java:272) at org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.main(KafkaOffsetLagUtil.java:124)
原因:
storm 读取自身存储的kafka信息时，没读到或者解析不正确。这个消息存储在zookeeper中。在
org.apache.storm.kafka.SpoutConfig
public SpoutConfig(BrokerHosts hosts, String topic, String zkRoot, String id)中的zkRoot和id, storm 会在zookeepr的中创建如下目录 /zkRoot/id，在这个目录下面会存放一些kafka相关信息。大致如下:
{"topology":{"id":"alarmTopology-11-1528251428","name":"alarmTopology"},"offset":397,"partition":0,"broker":{"host":"172.16.125.134","port":9092},"topic":"alarm-fe"}
zookeeper路径如下:
[zk: localhost:2181(CONNECTED) 81] ls /kafka_storm/webTraceId [partition_0, partition_1, partition_2]
解决办法:
分2步:
1. 要zookeeper中删除对应的目录
delete /kafka_storm/webTraceId/partition_0 delete /kafka_storm/webTraceId/partition_1 delete /kafka_storm/webTraceId/partition_2 delete /kafka_storm/webTraceId
2. 往kafka中生产数据
等数据被storm 的kafkaSpout消费之后，/kafka_storm/webTraceId/partition_0 这些目录会被重新建立
问题解决