Outline
Apache Flume is a distributed, reliable and available system for efficiently collect, aggregate and a large amount of log data from a plurality of different sources of centralized data storage to the mobile.
Apache Flume log data is not limited to the use of the polymerization. Since the data source is customizable, thus Flume number of events can be used to transmit data, including but not limited to network traffic data, social media generated data, and e-mail messages may be virtually any data source.
Apache Flume is a top-level project of Apache Software Foundation, there are two lines of code release, version 0.9.x and 1.x.
System Requirements
1.Java runtime environment - must match (for example flume1.8.0, use jdk version 1.8 or later)
2. Memory - configuration source or receiver used in the channel having sufficient memory
3. Disk space - or receiver channel configured to use sufficient disk space
4. directory permissions - directory used by the agent read / write permissions
Installation configuration
Flume single node deployment. This configuration allows the user to generate an event and then record it to the console.
1. official website to download flume version you want to upload to linux and extract the path you want to install the next
2. Configure environment variables: vim / etc / profile
3. The immediate effect source / etc / profile
Find their installation path:
For example /opt/apache-flume-1.8.0-bin/conf, in this path vim example.conf, and then copy the following configuration
# example.conf: single node configuration Flume
# named components on this proxy
a1.sources R1 =
a1.sinks = K1
a1.channels = C1
# Description / configuration source
a1.sources.r1.type netcat =
A1. = localhost sources.r1.bind
a1.sources.r1.port = 44444
# Description sink
a1.sinks.k1.type = Logger
# of events using the channel buffer memory
a1.channels.c1.type = Memory
a1.channels .c1 = 1000 .capacity
a1.channels.c1.transactionCapacity = 100
# the source and sink bound to channel
a1.sources.r1.channels = C1
a1.sinks.k1.channel = C1
This configuration defines a named agents of a1. a1 one of the listeners of the source data port 44444, a cache memory channel event data, the event data recorder and a receiver to the console. Profile naming the various components, and then describes their type and configuration parameters. Given profile may define multiple named agent; when given Flume process starts, it will pass a flag that tells it what to display the designated agent.
Start flume, in the directory / opt / apache-flume-1.6.0-bin / bin, run flume
flume-ng agent -n a1 -f /conf/example.conf -Dflume.root.logger=INFO,console
Parameters :
-n Specifies the agent name
-c specifies the profile directory
-f specify the configuration file
-Dflume.root.logger = DEBUG, console log level settings
Open another connection terminal to the remote port 44444, and then transmits the test events flume
telnet localhost 44444