Flume Cloudera is provided to a highly available, highly reliable, distributed massive log collection, aggregation and transmission systems, to support various types of customized data Flume sender log system for collecting data; simultaneously, providing Flume simple data processing, and the ability to write to a variety of data recipients (customizable) is.
1.flume node design services
2. Download and install the Flume
1) Download the Apache version of the Flume.
2) Download Cloudera version of the Flume.
3) Select here to download the version of Apache apache-flume-1.7.0-bin.tar.gz, and then uploaded to bigdata-pro01.kfk.com node / opt / softwares / directory
4) extract the Flume
tar -zxf apache-flume-1.7.0-bin.tar.gz -C /opt/modules/
5) The flume distributed to two other nodes
scp -r flume-1.7.0-bin bigdata-pro02.kfk.com:/opt/modules/
scp -r flume-1.7.0-bin bigdata-pro03.kfk.com:/opt/modules/
3.flume agent-1 collection node configuration service
1) bigdata-pro02.kfk.com node configuration flume, the collected data to a node bigdata-pro01.kfk.com
we flume-conf.properties
agent2.sources = r1
agent2.channels = c1
agent2.sinks = k1
agent2.sources.r1.type = exec
agent2.sources.r1.command = tail -F /opt/datas/weblogs.log
agent2.sources.r1.channels = c1
agent2.channels.c1.type = memory
agent2.channels.c1.capacity = 10000
agent2.channels.c1.transactionCapacity = 10000
agent2.channels.c1.keep-alive = 5
agent2.sinks.k1.type = avro
agent2.sinks.k1.channel = c1
agent2.sinks.k1.hostname = bigdata-pro01.kfk.com
agent2.sinks.k1.port = 5555
2) bigdata-pro03.kfk.com node configuration flume, the collected data to a node bigdata-pro01.kfk.com
we flume-conf.properties
agent3.sources = r1
agent3.channels = c1
agent3.sinks = k1
agent3.sources.r1.type = exec
agent3.sources.r1.command = tail -F /opt/datas/weblogs.log
agent3.sources.r1.channels = c1
agent3.channels.c1.type = memory
agent3.channels.c1.capacity = 10000
agent3.channels.c1.transactionCapacity = 10000
agent3.channels.c1.keep-alive = 5
agent3.sinks.k1.type = avro
agent3.sinks.k1.channel = c1
agent3.sinks.k1.hostname = bigdata-pro01.kfk.com
agent3.sinks.k1.port = 5555