Flume 从 0 到 1 学习 —— 第二章 Flume 快速入门

其他 2021-03-04 10:17:48 阅读次数: 0

文章目录

1. Flume 安装地址

  1. Flume官网地址

    http://flume.apache.org/

  2. 文档查看地址

    http://flume.apache.org/FlumeUserGuide.html

  3. 下载地址

    http://archive.apache.org/dist/flume/

2. Flume 安装部署

  1. 将 apache-flume-1.9.0-bin.tar.gz上传到 linux 的 /opt/software目录下

  2. 解压 apache-flume-1.9.0-bin.tar.gz到 /opt/module/目录下

    [dwjf321@hadoop102 software]$ tar -zxf apache-flume-1.9.0-bin.tar.gz -C /opt/module/

  3. 修改 apache-flume-1.9.0-bin的名称为 flume

    [dwjf321@hadoop102 module]$ mv apache-flume-1.9.0-bin flume

  4. 将 flume/conf下的 flume-env.sh.template文件修改为 flume-env.sh，并配置 flume-env.sh文件

[dwjf321@hadoop102 conf]$ mv flume-env.sh.template flume-env.sh
[dwjf321@hadoop102 conf]$ vim flume-env.sh
export JAVA_HOME=/opt/module/jdk1.8.0_144

3. Flume 配置

3.1 配置 TailDir Source 监控文件

# 定义组件
# 定义source
a1.sources=r1

# configure source  配置 source
# taildir 方式读取文件
a1.sources.r1.type = TAILDIR
# 定义读取位置的存储位置
a1.sources.r1.positionFile = /opt/module/flume/test/log_position.json
a1.sources.r1.filegroups = f1
a1.sources.r1.filegroups.f1 = /tmp/logs/app.+
a1.sources.r1.fileHeader = true
a1.sources.r1.channels = c1 c2


a1.sources.r1.selector.type = multiplexing
a1.sources.r1.selector.header = topic
a1.sources.r1.selector.mapping.topic_start = c1
a1.sources.r1.selector.mapping.topic_event = c2

Taildir Source 相比Exec Source、Spooling Directory Source的优势：

  1. TailDir Source：断点续传、多目录。Flume1.6以前需要自己自定义Source记录每次读取文件位置，实现断点续传。

  2. Exec Source： 可以实时搜集数据，但是在Flume不运行或者Shell命令出错的情况下，数据将会丢失。

  3. Spooling Directory Source： 用于监控目录，不支持断点续传。

3.2 配置 Kafka Channel

# 定义channel
a1.channels=c1 c2

# configure channel 定义channel
a1.channels.c1.type = org.apache.flume.channel.kafka.KafkaChannel
a1.channels.c1.kafka.bootstrap.servers = hadoop102:9092,hadoop103:9092,hadoop104:9092
a1.channels.c1.kafka.topic = topic_start
a1.channels.c1.parseAsFlumeEvent = false
a1.channels.c1.kafka.consumer.group.id = flume-consumer

a1.channels.c2.type = org.apache.flume.channel.kafka.KafkaChannel
a1.channels.c2.kafka.bootstrap.servers = hadoop102:9092,hadoop103:9092,hadoop104:9092
a1.channels.c2.kafka.topic = topic_event
a1.channels.c2.parseAsFlumeEvent = false
a1.channels.c2.kafka.consumer.group.id = flume-consumer

采用Kafka Channel，省去了Sink，提高了效率。

3.3 配置 Kafka Source

## 组件
a1.sources=r1 r2
a1.channels=c1 c2

## source1
a1.sources.r1.type = org.apache.flume.source.kafka.KafkaSource
a1.sources.r1.batchSize = 5000
a1.sources.r1.batchDurationMillis = 2000
a1.sources.r1.kafka.bootstrap.servers = hadoop102:9092,hadoop103:9092,hadoop104:9092
a1.sources.r1.kafka.topics=topic_start

## source2
a1.sources.r2.type = org.apache.flume.source.kafka.KafkaSource
a1.sources.r2.batchSize = 5000
a1.sources.r2.batchDurationMillis = 2000
a1.sources.r2.kafka.bootstrap.servers = hadoop102:9092,hadoop103:9092,hadoop104:9092
a1.sources.r2.kafka.topics=topic_event


## channel1
a1.channels.c1.type = file
a1.channels.c1.checkpointDir = /opt/module/flume/checkpoint/behavior1
a1.channels.c1.dataDirs = /opt/module/flume/data/behavior1/
a1.channels.c1.maxFileSize = 2146435071
a1.channels.c1.capacity = 1000000
a1.channels.c1.keep-alive = 6

## channel2
a1.channels.c2.type = file
a1.channels.c2.checkpointDir = /opt/module/flume/checkpoint/behavior2
a1.channels.c2.dataDirs = /opt/module/flume/data/behavior2/
a1.channels.c2.maxFileSize = 2146435071
a1.channels.c2.capacity = 1000000
a1.channels.c2.keep-alive = 6

3.4 配置 HDFS Sink

a1.sinks=k1 k2
## sink1
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = /origin_data/gmall/log/topic_start/%Y-%m-%d
a1.sinks.k1.hdfs.filePrefix = logstart-
a1.sinks.k1.hdfs.round = true
a1.sinks.k1.hdfs.roundValue = 10
a1.sinks.k1.hdfs.roundUnit = second

##sink2
a1.sinks.k2.type = hdfs
a1.sinks.k2.hdfs.path = /origin_data/gmall/log/topic_event/%Y-%m-%d
a1.sinks.k2.hdfs.filePrefix = logevent-
a1.sinks.k2.hdfs.round = true
a1.sinks.k2.hdfs.roundValue = 10
a1.sinks.k2.hdfs.roundUnit = second

## 不要产生大量小文件
a1.sinks.k1.hdfs.rollInterval = 10
a1.sinks.k1.hdfs.rollSize = 134217728
a1.sinks.k1.hdfs.rollCount = 0

# 每10s 滚动一个文件
a1.sinks.k2.hdfs.rollInterval = 10
# 每 128M 滚动一个文件
a1.sinks.k2.hdfs.rollSize = 134217728
# 多少个事件之后滚动一个文件 0忽略
a1.sinks.k2.hdfs.rollCount = 0

## 控制输出文件是原生文件。
a1.sinks.k1.hdfs.fileType = CompressedStream
a1.sinks.k2.hdfs.fileType = CompressedStream

a1.sinks.k1.hdfs.codeC = lzop
a1.sinks.k2.hdfs.codeC = lzop

## 拼装
a1.sources.r1.channels = c1
a1.sinks.k1.channel= c1

a1.sources.r2.channels = c2
a1.sinks.k2.channel= c2

3.5 定义自定义拦截器

创建 maven 工程，添加 pom.xml依赖

<dependency>
    <groupId>org.apache.flume</groupId>
    <artifactId>flume-ng-core</artifactId>
    <version>1.9.0</version>
</dependency>

3.5.1 Log 解析器

package com.big.data.flume.interceptor;

import org.apache.commons.lang.math.NumberUtils;

public class LogUtils {
    
    

    public static boolean validateEvent(String log) {
    
    
        // 服务器时间 | json
        // 1549696569054 | {"cm":{"ln":"-89.2","sv":"V2.0.4","os":"8.2.0","g":"[email protected]","nw":"4G","l":"en","vc":"18","hw":"1080*1920","ar":"MX","uid":"u8678","t":"1549679122062","la":"-27.4","md":"sumsung-12","vn":"1.1.3","ba":"Sumsung","sr":"Y"},"ap":"weather","et":[]}

        // 1 切割
        String[] logContents = log.split("\\|");

        // 2 校验
        if(logContents.length != 2){
    
    
            return false;
        }

        //3 校验服务器时间
        if (logContents[0].length()!=13 || !NumberUtils.isDigits(logContents[0])){
    
    
            return false;
        }

        // 4 校验json
        if (!logContents[1].trim().startsWith("{") || !logContents[1].trim().endsWith("}")){
    
    
            return false;
        }

        return true;
    }

    public static boolean validateStart(String log) {
    
    

        if (log == null){
    
    
            return false;
        }

        // 校验json
        if (!log.trim().startsWith("{") || !log.trim().endsWith("}")){
    
    
            return false;
        }

        return true;
    }
}

3.5.2 ETL 拦截器

对数据进行简单的校验，拦截不合格的数据。

package com.big.data.flume.interceptor;

import org.apache.flume.Context;
import org.apache.flume.Event;
import org.apache.flume.interceptor.Interceptor;

import java.nio.charset.Charset;
import java.util.ArrayList;
import java.util.List;

public class LogETLInterceptor implements Interceptor {
    
    
    @Override
    public void initialize() {
    
    

    }

    @Override
    public Event intercept(Event event) {
    
    

        // 1. 获取数据
        byte[] body = event.getBody();
        String log = new String(body, Charset.forName("UTF-8"));

        // 2. 校验 启动日志(json)  事件日志(服务器事件|json)
        if (log.contains("start")) {
    
    
            if (LogUtils.validateStart(log)) {
    
    
                return event;
            }
        } else {
    
    
            if (LogUtils.validateEvent(log)) {
    
    
                return event;
            }
        }
        // 3. 返回校验结果
        return null;
    }

    @Override
    public List<Event> intercept(List<Event> list) {
    
    

        ArrayList<Event> events = new ArrayList<>();
        for (Event event : list) {
    
    
            Event intercept = intercept(event);
            if (intercept == null) {
    
    
                continue;
            }
            events.add(intercept);
        }
        return events;
    }

    @Override
    public void close() {
    
    

    }


    public static class Builder implements Interceptor.Builder {
    
    

        @Override
        public Interceptor build() {
    
    
            return new LogETLInterceptor();
        }

        @Override
        public void configure(Context context) {
    
    

        }
    }
}

3.5.3 日志类型拦截器

判断日志类型，发送到不同的 kafka channel

package com.big.data.flume.interceptor;

import org.apache.flume.Context;
import org.apache.flume.Event;
import org.apache.flume.interceptor.Interceptor;

import java.nio.charset.Charset;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;

public class LogTypeInterceptor implements Interceptor {
    
    
    @Override
    public void initialize() {
    
    

    }

    @Override
    public Event intercept(Event event) {
    
    
        // 1. 获取body 数据
        byte[] body = event.getBody();
        String log = new String(body, Charset.forName("UTF-8"));

        // 2. 获取 header
        Map<String, String> headers = event.getHeaders();

        // 3. 判断数据类型并向 header 中赋值
        if (log.contains("start")) {
    
    
            headers.put("topic","topic_start");
        } else {
    
    
            headers.put("topic","topic_event");
        }
        return event;
    }

    @Override
    public List<Event> intercept(List<Event> list) {
    
    
        ArrayList<Event> events = new ArrayList<>();
        for (Event event : list) {
    
    
            Event intercept = intercept(event);
            if (intercept == null) {
    
    
                continue;
            }
            events.add(intercept);
        }
        return events;
    }

    @Override
    public void close() {
    
    

    }

    public static class Builder implements Interceptor.Builder {
    
    

        @Override
        public Interceptor build() {
    
    
            return new LogTypeInterceptor();
        }

        @Override
        public void configure(Context context) {
    
    

        }
    }
}

3.5.4 配置拦截器

#interceptor 定义拦截器
a1.sources.r1.interceptors =  i1 i2
a1.sources.r1.interceptors.i1.type = com.big.data.flume.interceptor.LogETLInterceptor$Builder
a1.sources.r1.interceptors.i2.type = com.big.data.flume.interceptor.LogTypeInterceptor$Builder

3.6 完整的配置示例

3.6.1 TailDir Source 配置示例

编写 file-flume-kafka.conf配置文件

# 定义组件
# 定义source
a1.sources=r1
# 定义channel
a1.channels=c1 c2

# configure source  配置 source
# taildir 方式读取文件
a1.sources.r1.type = TAILDIR
# 定义读取位置的存储位置
a1.sources.r1.positionFile = /opt/module/flume/test/log_position.json
a1.sources.r1.filegroups = f1
a1.sources.r1.filegroups.f1 = /tmp/logs/app.+
a1.sources.r1.fileHeader = true
a1.sources.r1.channels = c1 c2

#interceptor 定义拦截器
a1.sources.r1.interceptors =  i1 i2
a1.sources.r1.interceptors.i1.type = com.big.data.flume.interceptor.LogETLInterceptor$Builder
a1.sources.r1.interceptors.i2.type = com.big.data.flume.interceptor.LogTypeInterceptor$Builder

a1.sources.r1.selector.type = multiplexing
a1.sources.r1.selector.header = topic
a1.sources.r1.selector.mapping.topic_start = c1
a1.sources.r1.selector.mapping.topic_event = c2

# configure channel 定义channel
a1.channels.c1.type = org.apache.flume.channel.kafka.KafkaChannel
a1.channels.c1.kafka.bootstrap.servers = hadoop102:9092,hadoop103:9092,hadoop104:9092
a1.channels.c1.kafka.topic = topic_start
a1.channels.c1.parseAsFlumeEvent = false
a1.channels.c1.kafka.consumer.group.id = flume-consumer

a1.channels.c2.type = org.apache.flume.channel.kafka.KafkaChannel
a1.channels.c2.kafka.bootstrap.servers = hadoop102:9092,hadoop103:9092,hadoop104:9092
a1.channels.c2.kafka.topic = topic_event
a1.channels.c2.parseAsFlumeEvent = false
a1.channels.c2.kafka.consumer.group.id = flume-consumer

3.6.2 Kafka Source 示例

编写 kafka-flume-hdfs.conf配置文件

## 组件
a1.sources=r1 r2
a1.channels=c1 c2
a1.sinks=k1 k2

## source1
a1.sources.r1.type = org.apache.flume.source.kafka.KafkaSource
a1.sources.r1.batchSize = 5000
a1.sources.r1.batchDurationMillis = 2000
a1.sources.r1.kafka.bootstrap.servers = hadoop102:9092,hadoop103:9092,hadoop104:9092
a1.sources.r1.kafka.topics=topic_start

## source2
a1.sources.r2.type = org.apache.flume.source.kafka.KafkaSource
a1.sources.r2.batchSize = 5000
a1.sources.r2.batchDurationMillis = 2000
a1.sources.r2.kafka.bootstrap.servers = hadoop102:9092,hadoop103:9092,hadoop104:9092
a1.sources.r2.kafka.topics=topic_event

## channel1
a1.channels.c1.type = file
a1.channels.c1.checkpointDir = /opt/module/flume/checkpoint/behavior1
a1.channels.c1.dataDirs = /opt/module/flume/data/behavior1/
a1.channels.c1.maxFileSize = 2146435071
a1.channels.c1.capacity = 1000000
a1.channels.c1.keep-alive = 6

## channel2
a1.channels.c2.type = file
a1.channels.c2.checkpointDir = /opt/module/flume/checkpoint/behavior2
a1.channels.c2.dataDirs = /opt/module/flume/data/behavior2/
a1.channels.c2.maxFileSize = 2146435071
a1.channels.c2.capacity = 1000000
a1.channels.c2.keep-alive = 6

## sink1
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = /origin_data/gmall/log/topic_start/%Y-%m-%d
a1.sinks.k1.hdfs.filePrefix = logstart-
a1.sinks.k1.hdfs.round = true
a1.sinks.k1.hdfs.roundValue = 10
a1.sinks.k1.hdfs.roundUnit = second

##sink2
a1.sinks.k2.type = hdfs
a1.sinks.k2.hdfs.path = /origin_data/gmall/log/topic_event/%Y-%m-%d
a1.sinks.k2.hdfs.filePrefix = logevent-
a1.sinks.k2.hdfs.round = true
a1.sinks.k2.hdfs.roundValue = 10
a1.sinks.k2.hdfs.roundUnit = second

## 不要产生大量小文件
a1.sinks.k1.hdfs.rollInterval = 10
a1.sinks.k1.hdfs.rollSize = 134217728
a1.sinks.k1.hdfs.rollCount = 0

# 每10s 滚动一个文件
a1.sinks.k2.hdfs.rollInterval = 10
# 每 128M 滚动一个文件
a1.sinks.k2.hdfs.rollSize = 134217728
# 多少个事件之后滚动一个文件 0忽略
a1.sinks.k2.hdfs.rollCount = 0

## 控制输出文件是原生文件。
a1.sinks.k1.hdfs.fileType = CompressedStream
a1.sinks.k2.hdfs.fileType = CompressedStream

a1.sinks.k1.hdfs.codeC = lzop
a1.sinks.k2.hdfs.codeC = lzop

## 拼装
a1.sources.r1.channels = c1
a1.sinks.k1.channel= c1

a1.sources.r2.channels = c2
a1.sinks.k2.channel= c2

4. 启动 Flume

nohup /opt/module/flume/bin/flume-ng agent --conf-file /opt/module/flume/conf/file-flume-kafka.conf --name a1 -Dflume.root.logger=INFO,LOGFILE >/dev/null 2>&1 &

5. 停止 Flume

ps -ef | grep file-flume-kafka | grep -v grep |awk '{print \$2}' | xargs kill

猜你喜欢

转载自blog.csdn.net/dwjf321/article/details/112480349
今日推荐
Linus “吃狗粮”最积极！
开源日报 | Winamp播放器即将开源；生成式AI之战升级第二轮；Linus“吃狗粮”最积极；AI进入泡沫前期；吴泳铭为阿里云带来了什么？
NetBSD 禁止提交由 AI 生成的代码
Apache Doris 2.0.10 版本正式发布！
开源日报 | 大模型开战；大模型独角兽被曝卖身；周鸿祎建议谷歌开源所有产品；最大开源AI社区提供1000万美元共享GPU
开源日报 | Chrome内置Gemini的意义不在于Gemini；中国AI追随之路的五大误区；ECharts创始人“下海”养鱼；谷歌I/O开发者大会什么都有，只是没有惊喜
微软回应中国区AI团队“打包赴美”传闻
周排行
LogN级别的区间查询算法(线段树), 你学会了吗
数论概论(英文版.第4版)
idea 更新后和新的直接安装前，都需要配置 idea64.exe.vmoptions 后再使用
CANOpen系列教程04_CAN总线波特率、位时序、帧类型及格式说明
Java序列化基础
java排序算法整理
异常：org.apache.ibatis.reflection.ReflectionException
（算法练习）——二路归并排序
go 闭包函数
好程序员web前端技术分享媒体查询
每日归档
更多
2024-05-21(8)
2024-05-20(36)
2024-05-19(0)
2024-05-18(4)
2024-05-17(34)
2024-05-16(6)
2024-05-15(24)
2024-05-14(0)
2024-05-13(18)
2024-05-12(0)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022