版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/zzw0221/article/details/81454956
一、概述
Flume是Cloudera提供的一个高可用的，高可靠的，分布式的海量日志采集、聚合和传输的系统，Flume支持在日志系统中定制各类数据发送方，用于收集数据；同时，Flume提供对数据进行简单处理，并写到各种数据接受方（可定制）的能力。
二、安装
1、解压：
tar -zxvf flume-ng-1.6.0-cdh5.14.2.tar.gz -C /opt/cdh5.14.2/2、查看conf目录：
[root@master conf]# ll
total 16
-rw-r--r-- 1 1106 4001 1661 Mar 28 04:47 flume-conf.properties.template
-rw-r--r-- 1 1106 4001 1455 Mar 28 04:47 flume-env.ps1.template
-rw-r--r-- 1 1106 4001 1565 Mar 28 04:47 flume-env.sh.template
-rw-r--r-- 1 1106 4001 3107 Mar 28 04:47 log4j.properties3、重命名flume-env.sh.template为flume-env.sh
4、配置flume-env.sh
查看Java_home的路径：
[root@master conf]# echo $JAVA_HOME
/opt/java/jdk1.7.0_80修改flume-env.sh：
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# If this file is placed at FLUME_CONF_DIR/flume-env.sh, it will be sourced
# during Flume startup.
# Enviroment variables can be set here.
export JAVA_HOME=/opt/java/jdk1.7.0_80
# Give Flume more memory and pre-allocate, enable remote monitoring via JMX
# export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote"
# Let Flume write raw event data and configuration information to its log files for debugging
# purposes. Enabling these flags is not recommended in production,
# as it may result in logging sensitive user information or encryption secrets.
# export JAVA_OPTS="$JAVA_OPTS -Dorg.apache.flume.log.rawdata=true -Dorg.apache.flume.log.printconfig=true "
# Note that the Flume conf directory is always included in the classpath.
#FLUME_CLASSPATH=""
5、将Hadoop的lib目录下的四个jar包拷贝到flume的lib目录：
commons-configuration-1.6.jar，hadoop-auth-2.6.0-cdh5.14.2.jar，hadoop-common-2.6.0-cdh5.14.2.jar，hadoop-hdfs-2.6.0-cdh5.14.2.jar
[root@master hadoop-2.6.0]# cp -r share/hadoop/tools/lib/commons-configuration-1.6.jar /opt/cdh5.14.2/flume-1.6.0/lib/
[root@master hadoop-2.6.0]# cp -r share/hadoop/common/lib/hadoop-auth-2.6.0-cdh5.14.2.jar /opt/cdh5.14.2/flume-1.6.0/lib/
[root@master hadoop-2.6.0]# cp -r share/hadoop/common/hadoop-common-2.6.0-cdh5.14.2.jar /opt/cdh5.14.2/flume-1.6.0/lib/
[root@master hadoop-2.6.0]# cp -r share/hadoop/hdfs/hadoop-hdfs-2.6.0-cdh5.14.2.jar /opt/cdh5.14.2/flume-1.6.0/lib/三、基本使用
1、查看当前flume版本：
[root@master flume-1.6.0]# bin/flume-ng version
Flume 1.6.0-cdh5.14.2
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
Revision: 50436774fa1c7eaf0bd9c89ac6ee845695fbb687
Compiled by jenkins on Tue Mar 27 13:55:10 PDT 2018
From source with checksum 30217fe2b34097676ff5eabb51f4a11d2、查看帮助：
[root@master flume-1.6.0]# bin/flume-ng help
Usage: bin/flume-ng <command> [options]...
commands:
help display this help text
agent run a Flume agent
avro-client run an avro Flume client
version show Flume version info
global options:
--conf,-c <conf> use configs in <conf> directory
--classpath,-C <cp> append to the classpath
--dryrun,-d do not actually start Flume, just print the command
--plugins-path <dirs> colon-separated list of plugins.d directories. See the
plugins.d section in the user guide for more details.
Default: $FLUME_HOME/plugins.d
-Dproperty=value sets a Java system property value
-Xproperty=value sets a Java -X option
agent options:
--name,-n <name> the name of this agent (required)
--conf-file,-f <file> specify a config file (required if -z missing)
--zkConnString,-z <str> specify the ZooKeeper connection to use (required if -f missing)
--zkBasePath,-p <path> specify the base path in ZooKeeper for agent configs
--no-reload-conf do not reload config file if changed
--help,-h display help text
avro-client options:
--rpcProps,-P <file> RPC client properties file with server connection params
--host,-H <host> hostname to which events will be sent
--port,-p <port> port of the avro source
--dirname <dir> directory to stream to avro source
--filename,-F <file> text file to stream to avro source (default: std input)
--headerFile,-R <file> File containing event headers as key/value pairs on each new line
--help,-h display help text
Either --rpcProps or both --host and --port must be specified.
Note that if <conf> directory is specified, then it is always included first
in the classpath.
四、Flume Agent应用例子：将hive的log存储到hdfs
1、在flume的conf目录下创建文件catchhivelogs.conf，文件的内容的格式可以参照conf目录下的flume-conf.properties.template
2、编写catchhivelogs.conf
http://flume.apache.org/FlumeUserGuide.html#exec-source
http://flume.apache.org/FlumeUserGuide.html#memory-channel
http://flume.apache.org/FlumeUserGuide.html#hdfs-sink
# The configuration file needs to define the sources,
# the channels and the sinks.
# Sources, channels and sinks are defined per agent,
# in this case called 'agent'
#define agent
a1.sources = r1
a1.channels = c1
a1.sinks = k1
#define sources
a1.sources.r1.type = exec
a1.sources.r1.command = tail -f /opt/cdh5.14.2/hive-1.1.0/logs/hive.log
a1.sources.r1.shell = /bin/bash -c
#define channels
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
#define sinks
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = hdfs://master.cdh.com:8020/user/flume/hive-logs/
a1.sinks.k1.hdfs.fileType = DataStream
a1.sinks.k1.hdfs.writeFormat = Text
a1.sinks.k1.hdfs.batchSize = 10
#bind the sources and sinks to the channels
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c13、运行：
bin/flume-ng agent -c conf -n a1 -f conf/catchhivelogs.conf -Dflume.root.logger=DEBUG,console发现报错了：
2018-07-31 17:43:21,039 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:447)] process failed
java.lang.NoClassDefFoundError: org/apache/htrace/core/Tracer$Builder
at org.apache.hadoop.fs.FsTracer.get(FsTracer.java:42)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2803)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:98)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2853)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2835)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:186)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:260)
at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:252)
at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:701)
at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:698)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.htrace.core.Tracer$Builder
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 18 more解决办法：
将hadoop-2.6.0/share/hadoop/common/lib目录下的htrace-core4-4.0.1-incubating.jar拷贝到flume的lib目录下。
4、运行结果：
查看hdfs:
5、优化：
在第2步中的conf:
a1.sinks.k1.hdfs.path = hdfs://master.cdh.com:8020/user/flume/hive-logs/
当Hadoop 为HA时候，这种写法肯定有问题。
解决办法：1、将hdfs-site.xml和core-site.xml复制到flume-1.6.0/conf的目录下
[root@master hadoop]# cp -r hdfs-site.xml /opt/cdh5.14.2/flume-1.6.0/conf/
[root@master hadoop]# cp -r core-site.xml /opt/cdh5.14.2/flume-1.6.0/conf/
2、修改a1.sinks.k1.hdfs.path = hdfs://ns1/user/flume/hive-logs/
五、flume在企业大数据中的架构：
