Project implementation -09 (butt Flume)

SpringBoot docking Flume

Among the large data stream processing, we need from the project, log user behavior collected by kafkaconveying to the calculation engine, calculation and analysis. This is one of the log files collected, we need Flumeto complete the docking.

① rely coordinates

	  <!--引入Flume-->
        <dependency>
            <groupId>org.apache.flume</groupId>
            <artifactId>flume-ng-sdk</artifactId>
            <version>1.9.0</version>
        </dependency>

② introduction Flume docking project dependencies

flumeProviding open source projects on the official website so that users with a 日志采集complete fast docked.
Get the address: https://pan.baidu.com/s/1chY1Ia7aBPtY7AVXZ_rrdQ
acquisition code: cgi8

This open source project package to comthe outermost layer package, this package is introduced directly into their own projects javaunder the direct subdirectory (the first stage of the project with its own package at the same level), you can.

③ configuration log file

Configuration items in log files logback.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration scan="true" scanPeriod="60 seconds" debug="false">
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender" >
        <encoder>
            <pattern>%p %c#%M %d{yyyy-MM-dd HH:mm:ss} %m%n</pattern>
            <charset>UTF-8</charset>
        </encoder>
    </appender>
    <appender name="flume" class="com.gilt.logback.flume.FlumeLogstashV1Appender">
        <flumeAgents>
            pro1:44444, <!--这里配置集群，此处搭建的是个伪集群-->
            pro2:44444,
            pro1:44444
        </flumeAgents>
        <flumeProperties>
            connect-timeout=4000;
            request-timeout=8000
        </flumeProperties>
        <!--这里为了测试调低了，生成环境下要调高-->
        <batchSize>1</batchSize> 
        <!--每1ms发送一次这里为了测试调低了，生成环境下要调高-->
        <reportingWindow>1</reportingWindow>
        <additionalAvroHeaders>
            myHeader=myValue
        </additionalAvroHeaders>
        <application>smapleapp</application>
        <layout class="ch.qos.logback.classic.PatternLayout">
            <pattern>%p %c#%M %d{yyyy-MM-dd HH:mm:ss} %m%n</pattern>
        </layout>
    </appender>
    <!-- 控制台输出⽇志级别 -->
    <root level="ERROR">
        <appender-ref ref="STDOUT" />
    </root>
    <logger name="com.baizhi.service" level="DEBUG" additivity="false">
    <appender-ref ref="STDOUT" />
    <appender-ref ref="flume" />
    </logger>
    <logger name="com.baizhi.interceptor" level="DEBUG" additivity="false">
        <appender-ref ref="STDOUT" />
        <appender-ref ref="flume" />
    </logger>

</configuration>

④ referenced in the project

项目中，要对接 flume 就变的很容易了，只要是 log4j日志对象的输出，都将直接通过网络，输送到接收端。

• Example:

package com.baizhi.interceptor;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;
import org.springframework.web.servlet.HandlerInterceptor;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

//将此组件交给工厂
@Component
public class UserInputFeatureInterceptor implements HandlerInterceptor {
    private static final Logger LOGGER = LoggerFactory.getLogger(UserInputFeatureInterceptor.class);

    @Override
    public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
    
        //获取信息
        String name = request.getParameter("name");
        String password = request.getParameter("password");
      
        LOGGER.info("请求参数为\t"+name+"\t"+password);
        /*2020-03-29 15:56:35.783  INFO 12620 --- [nio-9999-exec-8] c.b.i.UserInputFeatureInterceptor        : 请求参数为	测试数据2	000000	 */

        return true;
    }
}

⑤flume side settings

In /apache-flume-1.9.0-binthe root directory in a conffolder, create (custom) example02.perprotiesfiles , configuration is as follows:

# 声明基本条件 Source Channel Sink
a1.sources = s1
a1.sinks = sk1
a1.channels = c1

#配置Source组件，从Socket中接收文本数据
a1.sources.s1.type = avro
a1.sources.s1.bind = pro2
a1.sources.s1.port = 44444

#配置Sink组件 将接收数据打印在日志控制台
a1.sinks.sk1.type = logger

#配置Channel通道，主要负责数据缓冲
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

#进行组件间的绑定
a1.sources.s1.channels = c1
a1.sinks.sk1.channel = c1

启动flume

准备接收数据

[root@pro2 apache-flume-1.9.0-bin] ./bin/flume-ng agent --conf conf/ --name a1 -conf-file conf/example02.perproties -Dflume.root.logger=INFO,console