flume编译源码后启动报错

19/07/30 11:13:22 INFO util.RestClientHelper: es_username: null, es_password: null, es_hostName: 172.16.10.10,172.16.10.11, es_port: 9200
Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.NoSuchFieldError: INSTANCE
   at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$InternalAddressResolver.<init>(PoolingNHttpClientConnectionManager.java:591)
   at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.<init>(PoolingNHttpClientConnectionManager.java:163)
   at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.<init>(PoolingNHttpClientConnectionManager.java:147)
   at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.<init>(PoolingNHttpClientConnectionManager.java:119)
   at org.apache.http.impl.nio.client.HttpAsyncClientBuilder.build(HttpAsyncClientBuilder.java:668)
   at org.elasticsearch.client.RestClientBuilder$1.run(RestClientBuilder.java:213)
   at org.elasticsearch.client.RestClientBuilder$1.run(RestClientBuilder.java:210)
   at java.security.AccessController.doPrivileged(Native Method)
   at org.elasticsearch.client.RestClientBuilder.createHttpClient(RestClientBuilder.java:210)
   at org.elasticsearch.client.RestClientBuilder.build(RestClientBuilder.java:184)
   at com.geo.cn.util.RestClientHelper.<init>(RestClientHelper.java:82)
   at com.geo.cn.util.RestClientHelper.getClient(RestClientHelper.java:114)
   at com.geo.cn.util.EsUtil.getRestHighLevelClient(EsUtil.java:43)
   at com.geo.cn.DASink.process(DASink.java:182)
   at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
   at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
   at java.lang.Thread.run(Thread.java:745)」

原因：flume lib(classPath) 存在多个版本的http-client.jar,http-core.jar 。有flume自带的低版本 maven重新编译的flume-ng-core自带的高版本

解决：删除 flume lib目录下的低版本的http-client.jar,http-core.jar

flume解决后再次测试数据：

flume监控功能测试：

1.将测试数据cat出来，赋值一部分到kafka producer中

(1)创建topic:
kafka-topics.sh --create --zookeeper zookeeper地址(localhost:2181) --replication-factor 1 --partitions 1 --topic topic名称

扫描二维码关注公众号，回复： 9419078 查看本文章

(2)将日志重定向到kafka-console-producer.sh中：
/usr/kafka_2.10-0.9.0.1/bin/kafka-console-producer.sh --topic 生产新建测试topic --broker-list 生成环境kafka的broker-list对应的 ip：port << EOF
`cat kafka日志文件全路径`
EOF

/usr/kafka_2.10-0.9.0.1/bin/kafka-console-producer.sh --topic dp_monitor_rh_out-20190717 --broker-list hadoop26.geotmt.com:6667,hadoop206.geotmt.com:6667,hadoop207.geotmt.com:6667 << EOF
`cat da.log`
EOF

2.将数据放入后，查看flume日志，看是否正常插入elasticsearch ，查看elasticsearch中是否有对应模型id数据

dymkkj

发布了150 篇原创文章 · 获赞 15 · 访问量 10万+

私信关注

flume编译源码后启动报错

猜你喜欢