日志采集系统Flume启动的一些问题简述

版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/a247027417/article/details/82744723

Flume官网下载了binary版本的安装包之后,使用sftp上传至虚拟机。

使用tar -zxvf 命令行解压flume

在第一次启动时,直接使用官网上的例子进行启动,为了对配置文件留下印象,这个配置文件是自己手动敲上去的,修改前代码如下:

#Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

#Describe /configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = 127.0.0.1
a1.sources.r1.port = 44444

#Describe /configure the source
a1.sink.k1.type = logger

# use a channe; which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

#bind the source and sink to the channels
a1.sources.r1.channels = c1
a1.sinks.k1.channels = c1

cd 到flume目录下,使用命令

bin/flume-ng agent --conf conf --conf-file conf/netcat-logger.conf --name a1 -Dflume.root.logger=INFO,console

flume开始启动,在另外一台虚拟机使用telnet连接flume发送消息:

[root@pawn1 ~]# telnet 127.0.0.1 44444
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
hi\
OK

但是在flume端,没有在控制台打印任何消息,按ctrl+c退出flume出现了一堆信息

2018-09-17 06:16:51,123 (agent-shutdown-hook) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.stop(LifecycleSupervisor.java:78)] Stopping lifecycle supervisor 10
2018-09-17 06:16:51,125 (agent-shutdown-hook) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:149)] Component type: CHANNEL, name: c1 stopped
2018-09-17 06:16:51,125 (agent-shutdown-hook) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:155)] Shutdown Metric for type: CHANNEL, name: c1. channel.start.time == 1537136176961
2018-09-17 06:16:51,125 (agent-shutdown-hook) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:161)] Shutdown Metric for type: CHANNEL, name: c1. channel.stop.time == 1537136211124
2018-09-17 06:16:51,125 (agent-shutdown-hook) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:177)] Shutdown Metric for type: CHANNEL, name: c1. channel.capacity == 1000
2018-09-17 06:16:51,125 (agent-shutdown-hook) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:177)] Shutdown Metric for type: CHANNEL, name: c1. channel.current.size == 18
2018-09-17 06:16:51,125 (agent-shutdown-hook) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:177)] Shutdown Metric for type: CHANNEL, name: c1. channel.event.put.attempt == 18
2018-09-17 06:16:51,125 (agent-shutdown-hook) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:177)] Shutdown Metric for type: CHANNEL, name: c1. channel.event.put.success == 18
2018-09-17 06:16:51,126 (agent-shutdown-hook) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:177)] Shutdown Metric for type: CHANNEL, name: c1. channel.event.take.attempt == 0
2018-09-17 06:16:51,126 (agent-shutdown-hook) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:177)] Shutdown Metric for type: CHANNEL, name: c1. channel.event.take.success == 0
2018-09-17 06:16:51,126 (agent-shutdown-hook) [INFO - org.apache.flume.source.NetcatSource.stop(NetcatSource.java:197)] Source stopping
2018-09-17 06:16:51,627 (agent-shutdown-hook) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.stop(PollingPropertiesFileConfigurationProvider.java:84)] Configuration provider stopping

可以看到,发送的消息通过source进入了channel,但是却没有被sink取出,再详细查看启动日志,发现了错误位置

2018-09-17 06:26:48,855 (conf-file-poller-0) [WARN - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:697)] Could not configure sink  k1 due to: No channel configured for sink: k1
org.apache.flume.conf.ConfigurationException: No channel configured for sink: k1
        at org.apache.flume.conf.sink.SinkConfiguration.configure(SinkConfiguration.java:52)
        at org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:680)
        at org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:347)
        at org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.access$000(FlumeConfiguration.java:212)
        at org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:126)
        at org.apache.flume.conf.FlumeConfiguration.<init>(FlumeConfiguration.java:108)
        at org.apache.flume.node.PropertiesFileConfigurationProvider.getFlumeConfiguration(PropertiesFileConfigurationProvider.java:194)
        at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:93)
        at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:141)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

可以看到,sink没有channel的配置,返回查看conf,与正确的对比,发现在a1.sinks应该是sinks需要加上s,在a1.sinks.channel这个地方是channel而不是channels,所以,在配置文件上需要注意s的这些细小错误,下面附上正确的配置文件。

#Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

#Describe /configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = 127.0.0.1
a1.sources.r1.port = 44444

#Describe /configure the source
a1.sinks.k1.type = logger

# use a channe; which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

#bind the source and sink to the channels
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

猜你喜欢

转载自blog.csdn.net/a247027417/article/details/82744723
今日推荐