flume log4j日志接收

flume 安装与配置：

下载flume，然后解压：

tar xvf apache-flume-1.5.2-bin.tar.gz -C ./

配置flume，在conf/flume-conf.properties下（没有则创建，反正有template）:

# example.conf: A single-node Flume configuration

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44446

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

启动flume：

./bin/flume-ng agent --conf conf --conf-file conf/flume-conf.properties --name a1 -Dflume.root.logger=INFO,LOGFILE

flume java日志实例

启动flume后，可以建一个maven工程，pom.xml如下：

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>flume-test</groupId>
    <artifactId>flume-test</artifactId>
    <version>1.0</version>

    <dependencies>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-log4j12</artifactId>
            <version>1.7.10</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flume</groupId>
            <artifactId>flume-ng-core</artifactId>
            <version>1.5.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flume.flume-ng-clients</groupId>
            <artifactId>flume-ng-log4jappender</artifactId>
            <version>1.5.0</version>
        </dependency>
    </dependencies>


</project>

注意依赖一定要有flume-ng-log4jappender，后面配置log4j要用。
新建一个实例：

package per.flume;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.Date;

/**
 * Created by hp on 15-3-16.
 */
public class WriteLog {

    protected static final Logger logger = LoggerFactory.getLogger(WriteLog.class);

    public static void main(String[] args) throws Exception {
        while (true) {
            logger.info(String.valueOf(new Date().getTime()));
            Thread.sleep(2000);
        }
    }

}

resources下log4j配置如下：

### set log levels ###
log4j.rootLogger=INFO, stdout, file, flume
log4j.logger.per.flume=INFO

### flume ###
log4j.appender.flume=org.apache.flume.clients.log4jappender.Log4jAppender
log4j.appender.flume.layout=org.apache.log4j.PatternLayout
log4j.appender.flume.Hostname=localhost
log4j.appender.flume.Port=44446

### stdout ###
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Threshold=INFO
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %c{1} [%p] %m%n

### file ###
log4j.appender.file=org.apache.log4j.DailyRollingFileAppender
log4j.appender.file.Threshold=INFO
log4j.appender.file.File=./logs/tracker/tracker.log
log4j.appender.file.Append=true
log4j.appender.file.DatePattern='.'yyyy-MM-dd
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %c{1} [%p] %m%n

建立一个shell(run.sh)脚本用于讲flume的包加入classpath：

#!/bin/sh
jarlist=$(ls /home/hp/servers/apache-flume-1.5.2-bin/lib/*.jar)
CLASSPATH=/home/hp/projects/flume-test/target/flume-test-1.0.jar
for jar in ${jarlist}
do
    CLASSPATH=${CLASSPATH}:${jar}
done
echo ${CLASSPATH}

java -classpath $CLASSPATH per.flume.WriteLog

build 项目：

mvn clean package

我这里生成的是flume-test-1.0.jar,然后启动shell:

./run.sh

此时进入apache-flume-1.5.2-bin/logs/flume.log,可以看到我们程序打的日志了。