flume入门 log4j 输出日志到flume

将log4j产生的日志直接输出到flume控制台

1.编写客户端

package com.study.test;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

import java.util.Date;

public class WriteLog {
    protected static final Log logger = LogFactory.getLog(WriteLog.class);


    public static void main(String[] args) throws InterruptedException {

        while (true) {
            // 每隔两秒log输出一下当前系统时间戳
            logger.info(new Date().getTime());
            //System.out.println(new Date().getTime());
            Thread.sleep(2000);
            try {
                throw new Exception("exception msg");
            }
            catch (Exception e) {
                logger.error("error:" + e.getMessage());
            }
        }
    }
}

2.pom.xml

    <?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.study.test</groupId>
    <artifactId>log4j2flume</artifactId>
    <version>1.0-SNAPSHOT</version>

    <dependencies>
        <dependency>
            <groupId>org.apache.flume.flume-ng-clients</groupId>
            <artifactId>flume-ng-log4jappender</artifactId>
            <version>1.7.0</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-simple</artifactId>
            <version>1.6.1</version>
        </dependency>
    </dependencies>
</project>

3. log4j.properties

log4j.rootLogger=INFO,flume
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.out
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern="%d{yyyy-MM-dd HH:mm:ss} %p [%c:%L] - %m%n

log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender
log4j.appender.flume.Hostname = 10.45.17.66
log4j.appender.flume.Port = 4444
log4j.appender.flume.UnsafeMode = true
log4j.appender.flume.layout=org.apache.log4j.PatternLayout
log4j.appender.flume.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %p [%c:%L] - %m%n

4.配置flume 建立testlog2flume.conf

a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind =0.0.0.0
a1.sources.r1.port = 4444

# Describe the sink
a1.sinks.k1.type=logger
#a1.sinks.k1.type = file_roll
#a1.sinks.k1.sink.directory = /data/soft/flume/tmp
#a1.sinks.k1.sink.rollInterval=86400
#a1.sinks.k1.sink.batchSize=100
#a1.sinks.k1.sink.serializer=text
#a1.sinks.k1.sink.serializer.appendNewline = false

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 1000

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

该配置文件中 配置了一个代理a1
在代理agent中配置了一个source（源）一个sink（接收器）和一个channel（通道），分别为：r1，k1，c1
r1的类型定义为avro(序列化)，对应该类型参数为bind和port 分别为0.0.0.0和4141
k1的类型定义为logger，直接输出到日志文件中
cl的类型定义为内存方式，设置其参数capactiy和transactionCapacity分别为1000和1000
指定r1和k1的channel为c1

将上述配置中a1.sinks.k1.type=logger注释，把a1.sinks其他的给打开，就是将日志输入到文件

5.到flume目录下启动flume

bin/flume-ng agent -c conf -f conf/testlog2flume.conf --name a1 -Dflume.root.logger=INFO,console

flume-ng ：flume 命令
agent：运行一个Flume Agent
-c：在指定目录下使用配置 use configs in directory
-f：指定配置文件，这个配置文件必须在全局选项的–conf（-c）参数定义的目录下
–name:Agent的名称（必填）
-Dflume.root.logger=INFO,console 该参数将会把flume的日志输出到console