说在前面
- 工作环境：VMware® Workstation 12 Pro 12.5.6 build-5528349
- linux版本：CentOS-7-x86_64-Minimal-1611.iso
- Flume版本：apache-flume-1.8.0-bin.tar.gz
Flume的安装
-
下载 Flume：apache-flume-1.8.0-bin.tar.gz
-
解压到 /soft 目录下，并创建符号链接
//解压
$soft/> tar -xzvf apache-flume-1.8.0-bin.tar.gz
//创建符号链接
$soft/> ln -s apache-flume-1.8.0-bin flume
- 配置 /etc/profile 文件，source profile(立即生效)
# flume
export FLUME_HOME=/soft/flume
export PATH=$PATH:$FLUME_HOME/bin
- 验证是否安装成功
$> flume-ng version
配置Flume并使用不同的源测试
使用netcat源
安装nc：sudo yum install nmap-ncat.x86_64
- 进入 /soft/flume/conf 目录下配置
[helloworld.conf]
agent.sources = r1
agent.channels = c1
agent.sinks = k1
agent.sources.r1.type=netcat
agent.sources.r1.bind=localhost
agent.sources.r1.port=8888
agent.sinks.k1.type=logger
agent.channels.c1.type=memory
agent.sources.r1.channels=c1
agent.sinks.k1.channel=c1
- 运行flume
$> flume-ng agent -f ../conf/helloworld.conf -n agent -Dflume.root.logger=INFO,console
exec源：实时日志收集
[exec_r.conf]
agent.sources = r1
agent.channels = c1
agent.sinks = k1
agent.sources.r1.type=exec
agent.sources.r1.command=tail -F /home/centosmin0/test.txt
agent.sinks.k1.type=logger
agent.channels.c1.type=memory
agent.sources.r1.channels=c1
agent.sinks.k1.channel=c1
//在 /home/centosmin0/ 下创建一个 test.txt 文件
$> touch ~/test.txt
spooldir源
监控一个文件夹，收集完成后会重命名为新的文件
[spooldir_r.conf]
agent.sources = r1
agent.channels = c1
agent.sinks = k1
agent.sources.r1.type=spooldir
agent.sources.r1.spoolDir=/home/centosmin0/spool
agent.sources.r1.fileHeader=true
agent.sinks.k1.type=logger
agent.channels.c1.type=memory
agent.sources.r1.channels=c1
agent.sinks.k1.channel=c1
//在 /home/centosmin0/ 创建一个spool文件夹
$> mkdir ~/spool
Sequence源
[seq_r.conf]
agent.sources = r1
agent.channels = c1
agent.sinks = k1
agent.sources.r1.type=seq
agent.sources.r1.totalEvents=1000
agent.sinks.k1.type=logger
agent.channels.c1.type=memory
agent.sources.r1.channels=c1
agent.sinks.k1.channel=c1