flume官网(官网介绍及使用方式都比较全面):http://flume.apache.org/index.html
1、netcat–logger
# example.conf: A single-node Flume configuration
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444
# Describe the sink
a1.sinks.k1.type = logger
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
启动flume
flume-ng agent --conf-file /root/flume/tt --name a1 -Dflume.root.logger=INFO,console
在source对应的节点上启动telnet
telnet localhost 44444
在telnet输入相应东西,回车,flume端会显示打印
2、exec–logger
a1.sources=r1
a1.channels=c1
a1.sinks=k1
a1.sources.r1.type=exec
a1.sources.r1.command=tail -F /root/flume/log
a1.sinks.k1.type=logger
a1.channels.c1.type=memory
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity=100
a1.sources.r1.channels=c1
a1.sinks.k1.channel=c1
启动flume对应文件
flume-ng agent --conf-file /root/flume/exec --name a1 -Dflume.root.logger=INFO,console
此时会监测/root/flume/log文件,一旦有文件输入,flume会获取并打印展示
3、avro–logger
a1.sources = r1
a1.sinks = k1
a1.channels = c1
a1.sources.r1.type=avro
a1.sources.r1.channels = c1
a1.sources.r1.bind=192.168.30.101
a1.sources.r1.port=55555
# Describe the sink
a1.sinks.k1.type = logger
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sinks.k1.channel = c1
a1.sinks.k1.type=logger
启动flume
flume-ng agent --conf-file /root/flume/avro --name a1 -Dflume.root.logger=INFO,console
flume-ng avro-client -H localhost -p 55555 -F /root/flume/log
4、netcat–hdfs
a1.channels=c1
a1.sources=r1
a1.sinks=k1
a1.sources.r1.type=netcat
a1.sources.r1.bind=localhost
a1.sources.r1.port=41416
a1.sinks.k1.type=hdfs
a1.sinks.k1.hdfs.path=hdfs://sx/myflume/%y-%m-%d
a1.sinks.k1.hdfs.useLocalTimeStamp=true
a1.channels.c1.type=memory
a1.sources.r1.channels=c1
a1.sinks.k1.channel=c1
启动flume
flume-ng agent --conf-file /root/flume/netcat_hdfs --name a1 -Dflume.root.logger=INFO,console
启动telnet(对应端口)
telnet localhost 41416
5、avro–hdfs
# a1 which ones we want to activate.
a1.channels = c1
a1.sources = r1
a1.sinks = k1
a1.sources.r1.type = avro
a1.sources.r1.bind=localhost
a1.sources.r1.port=55555
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = hdfs://sxt/myflume
# Define a memory channel called c1 on a1
a1.channels.c1.type = memory
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity = 100
# Define an Avro source called r1 on a1 and tell it
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
flume
flume-ng agent --conf-file /root/flume/avro_hdfs --name a1 -Dflume.root.logger=INFO,console
flume-ng avro-client -H localhost -p 55555 -F /root/flume/log
6、exec–hdfs
a1.sources=r1
a1.channels=c1
a1.sinks=k1
a1.sources.r1.type=exec
a1.sources.r1.command=tail -F /root/flume/log
a1.sinks.k1.type=hdfs
a1.sinks.k1.hdfs.path = hdfs://sxt/myflume/qq
a1.channels.c1.type=memory
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity=100
a1.sources.r1.channels=c1
a1.sinks.k1.channel=c1
启动flume
flume-ng agent --conf-file /root/flume/exec_hdfs --name a1 -Dflume.root.logger=INFO,console
此时会监测/root/flume/log文件,一旦有文件输入,会写入hdfs