flume启动过程中配置错误出现的异常问题

其他 2020-02-19 10:50:47 阅读次数: 0

flume的官网提供了source channel sink的详细介绍，以及配置的示例，有不会的可以直接查看相应的文档。
官方文档网址：http://flume.apache.org/releases/content/1.9.0/FlumeUserGuide.html

1.hdfs sink的配置问题，会出现以下的错误形式

java.lang.NullPointerException: hdfs.path is required
错误的代码如下所示：

19/10/11 17:20:48 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting
19/10/11 17:20:48 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:collector1.conf
19/10/11 17:20:48 INFO conf.FlumeConfiguration: Processing:memory-channel
19/10/11 17:20:48 INFO conf.FlumeConfiguration: Processing:memory-channel
19/10/11 17:20:48 INFO conf.FlumeConfiguration: Processing:avro-source
19/10/11 17:20:48 INFO conf.FlumeConfiguration: Processing:avro-source
19/10/11 17:20:48 INFO conf.FlumeConfiguration: Processing:avro-source
19/10/11 17:20:48 INFO conf.FlumeConfiguration: Processing:hdfs-sink
19/10/11 17:20:48 INFO conf.FlumeConfiguration: Processing:avro-source
19/10/11 17:20:48 INFO conf.FlumeConfiguration: Processing:hdfs-sink
19/10/11 17:20:48 INFO conf.FlumeConfiguration: Added sinks: hdfs-sink Agent: avro-memory-hdfs
19/10/11 17:20:48 INFO conf.FlumeConfiguration: Processing:hdfs-sink
19/10/11 17:20:48 INFO conf.FlumeConfiguration: Processing:hdfs-sink
19/10/11 17:20:48 INFO conf.FlumeConfiguration: Processing:hdfs-sink
19/10/11 17:20:48 INFO conf.FlumeConfiguration: Processing:avro-source
19/10/11 17:20:48 INFO conf.FlumeConfiguration: Processing:avro-source
19/10/11 17:20:48 INFO conf.FlumeConfiguration: Processing:avro-source
19/10/11 17:20:48 INFO conf.FlumeConfiguration: Processing:memory-channel
19/10/11 17:20:48 INFO conf.FlumeConfiguration: Processing:hdfs-sink
19/10/11 17:20:48 INFO conf.FlumeConfiguration: Processing:avro-source
19/10/11 17:20:48 INFO conf.FlumeConfiguration: Processing:hdfs-sink
19/10/11 17:20:48 WARN conf.FlumeConfiguration: Agent configuration for 'avro-memory-hdfs' has no configfilters.
19/10/11 17:20:48 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [avro-memory-hdfs]
19/10/11 17:20:48 INFO node.AbstractConfigurationProvider: Creating channels
19/10/11 17:20:48 INFO channel.DefaultChannelFactory: Creating instance of channel memory-channel type memory
19/10/11 17:20:48 INFO node.AbstractConfigurationProvider: Created channel memory-channel
19/10/11 17:20:48 INFO source.DefaultSourceFactory: Creating instance of source avro-source, type avro
19/10/11 17:20:48 INFO sink.DefaultSinkFactory: Creating instance of sink: hdfs-sink, type: hdfs
19/10/11 17:20:48 ERROR node.AbstractConfigurationProvider: Sink hdfs-sink has been removed due to an error during configuration
java.lang.NullPointerException: hdfs.path is required
        at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
        at org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:195)
        at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
        at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:453)
        at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:106)
        at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:145)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
19/10/11 17:20:48 INFO node.AbstractConfigurationProvider: Channel memory-channel connected to [avro-source]
19/10/11 17:20:48 INFO node.Application: Starting new configuration:{ sourceRunners:{avro-source=EventDrivenSourceRunner: { source:Avro source avro-source: { bindAddress: 192.168.141.140, port: 44444 } }} sinkRunners:{} channels:{memory-channel=org.apache.flume.channel.MemoryChannel{name: memory-channel}} }
19/10/11 17:20:48 INFO node.Application: Starting Channel memory-channel
19/10/11 17:20:48 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: memory-channel: Successfully registered new MBean.
19/10/11 17:20:48 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: memory-channel started
19/10/11 17:20:48 INFO node.Application: Starting Source avro-source
19/10/11 17:20:48 INFO source.AvroSource: Starting Avro source avro-source: { bindAddress: 192.168.141.140, port: 44444 }...
19/10/11 17:20:48 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: avro-source: Successfully registered new MBean.
19/10/11 17:20:48 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: avro-source started
19/10/11 17:20:48 INFO source.AvroSource: Avro source avro-source started.

修改：hdfs sink必要的配置属性包含channel hdfs.type hdfs.path,以及选配的属性。
官网提供的示例：
官网地址：http://flume.apache.org/releases/content/1.9.0/FlumeUserGuide.html#hdfs-sink

a1.channels = c1
a1.sinks = k1
a1.sinks.k1.type = hdfs
a1.sinks.k1.channel = c1
a1.sinks.k1.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S
a1.sinks.k1.hdfs.filePrefix = events-
a1.sinks.k1.hdfs.round = true
a1.sinks.k1.hdfs.roundValue = 10
a1.sinks.k1.hdfs.roundUnit = minute

容易出错的地方：a1.sinks.k1.后面忘记hdfs,就会出现上面的错误提示，或者是hdfs.path的路径配置错误

2.channel通道配置错误

channel通道配置过程中，忘记配置source和sink与channel之间的对应关系，或者channel的配置出现拼写错误或者其它问题，可能会出现 does not contain any valid channels 和 org.jboss.netty.channel.ChannelException:等异常提示。
异常代码1：

19/10/11 17:30:21 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting
19/10/11 17:30:21 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:collector2.conf
19/10/11 17:30:21 INFO conf.FlumeConfiguration: Processing:hdfs-sink
19/10/11 17:30:21 INFO conf.FlumeConfiguration: Processing:hdfs-sink
19/10/11 17:30:21 INFO conf.FlumeConfiguration: Processing:memory-channel
19/10/11 17:30:21 INFO conf.FlumeConfiguration: Processing:memory-channel
19/10/11 17:30:21 INFO conf.FlumeConfiguration: Processing:avro-source
19/10/11 17:30:21 INFO conf.FlumeConfiguration: Processing:avro-source
19/10/11 17:30:21 INFO conf.FlumeConfiguration: Processing:avro-source
19/10/11 17:30:21 INFO conf.FlumeConfiguration: Processing:hdfs-sink
19/10/11 17:30:21 INFO conf.FlumeConfiguration: Processing:hdfs-sink
19/10/11 17:30:21 INFO conf.FlumeConfiguration: Processing:avro-source
19/10/11 17:30:21 INFO conf.FlumeConfiguration: Processing:avro-source
19/10/11 17:30:21 INFO conf.FlumeConfiguration: Added sinks: hdfs-sink Agent: avro-memory-hdfs
19/10/11 17:30:21 INFO conf.FlumeConfiguration: Processing:hdfs-sink
19/10/11 17:30:21 INFO conf.FlumeConfiguration: Processing:avro-source
19/10/11 17:30:21 INFO conf.FlumeConfiguration: Processing:avro-source
19/10/11 17:30:21 INFO conf.FlumeConfiguration: Processing:hdfs-sink
19/10/11 17:30:21 INFO conf.FlumeConfiguration: Processing:memory-channel
19/10/11 17:30:21 INFO conf.FlumeConfiguration: Processing:avro-source
19/10/11 17:30:21 INFO conf.FlumeConfiguration: Processing:hdfs-sink
19/10/11 17:30:21 WARN conf.FlumeConfiguration: Agent configuration for 'avro-memory-hdfs' has no configfilters.
19/10/11 17:30:21 WARN conf.FlumeConfiguration: Agent configuration for 'avro-memory-hdfs' does not contain any valid channels. Marking it as invalid.
19/10/11 17:30:21 WARN conf.FlumeConfiguration: Agent configuration invalid for agent 'avro-memory-hdfs'. It will be removed.

异常代码2：

19/10/11 17:32:10 INFO source.AvroSource: Avro source avro-source stopping: Avro source avro-source: { bindAddress: 192.168.141.130, port: 52020 }
19/10/11 17:32:10 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: avro-source stopped
19/10/11 17:32:10 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SOURCE, name: avro-source. source.start.time == 0
19/10/11 17:32:10 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SOURCE, name: avro-source. source.stop.time == 1570786330084
19/10/11 17:32:10 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SOURCE, name: avro-source. src.append-batch.accepted == 0
19/10/11 17:32:10 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SOURCE, name: avro-source. src.append-batch.received == 0
19/10/11 17:32:10 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SOURCE, name: avro-source. src.append.accepted == 0
19/10/11 17:32:10 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SOURCE, name: avro-source. src.append.received == 0
19/10/11 17:32:10 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SOURCE, name: avro-source. src.channel.write.fail == 0
19/10/11 17:32:10 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SOURCE, name: avro-source. src.event.read.fail == 0
19/10/11 17:32:10 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SOURCE, name: avro-source. src.events.accepted == 0
19/10/11 17:32:10 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SOURCE, name: avro-source. src.events.received == 0
19/10/11 17:32:10 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SOURCE, name: avro-source. src.generic.processing.fail == 0
19/10/11 17:32:10 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SOURCE, name: avro-source. src.open-connection.count == 0
19/10/11 17:32:10 INFO source.AvroSource: Avro source avro-source stopped. Metrics: SOURCE:avro-source{src.events.accepted=0, src.open-connection.count=0, src.append.received=0, src.channel.write.fail=0, src.append-batch.received=0, src.generic.processing.fail=0, src.append-batch.accepted=0, src.append.accepted=0, src.events.received=0, src.event.read.fail=0}
19/10/11 17:32:10 ERROR lifecycle.LifecycleSupervisor: Unable to start EventDrivenSourceRunner: { source:Avro source avro-source: { bindAddress: 192.168.141.130, port: 52020 } } - Exception follows.
org.apache.flume.FlumeException: Failed to set up server socket
        at org.apache.flume.source.AvroSource.start(AvroSource.java:207)
        at org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
        at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:249)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.jboss.netty.channel.ChannelException: Failed to bind to: /192.168.141.130:52020
        at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
        at org.apache.avro.ipc.NettyServer.<init>(NettyServer.java:106)
        at org.apache.flume.source.AvroSource.start(AvroSource.java:202)
        ... 9 more
Caused by: java.net.BindException: Cannot assign requested address
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.jboss.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193)
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:391)
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:315)
        at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        ... 3 more

修改：检查自己channel通道的配置是否正确，检查channel和sink source的对应关系是否配置。

无为二里
发布了27 篇原创文章 · 获赞 13 · 访问量 1万+
私信 关注

猜你喜欢

转载自blog.csdn.net/weixin_40122615/article/details/102505736
今日推荐
openKylin 社区生态委员会第六次会议圆满召开
阿里云正式发布通义千问 2.5
Python 3.13 发布首个 Beta：实验性自由线程模式和 JIT、改进交互式解释器
Stack Overflow 拿我的代码去训练 AI 大模型，还封了我的账号
Pop!_OS 的 COSMIC 桌面完成 App Store 上架工作
报告：Django 仍然是 74% 开发者的首选
《2024 年一季度互联网投融资运行情况》研究报告
15 年前上了“FFmpeg 耻辱柱”，今天他还得谢谢咱——腾讯QQPlayer一雪前耻？
TIOBE 5 月榜单：Fortran “复活”进入 Top 10
GCC 14.1 发布
面壁智能发布 Eurux-8x22B 开源大模型 —— 堪称「理科状元」
开源日报 | 谷歌扶持鸿蒙上位；开源Rabbit R1；Docker加持的安卓手机；微软的焦虑和野心；海尔电器把开放平台关了
周排行
计算机组成与设计（七）—— 除法器
Integer Approximation(分治+枚举)
大话数据库索引
windows10系统JDK的配置及下载地址
mysql实现秒值转换中原六仔平台搭建
Codeforces Round #556 (Div. 1)
百练1064 网线主管
Codeforces 995F Cowmpany Cowmpensation
子集生成之增量构造法，位向量法，二进制法
ERROR: cmd.exe failed with args /c "/APK\gradle\rungradle.bat...
每日归档
更多
2024-05-10(38)
2024-05-09(35)
2024-05-08(42)
2024-05-07(14)
2024-05-06(40)
2024-05-05(0)
2024-05-04(7)
2024-05-03(19)
2024-05-02(0)
2024-05-01(4)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022