实战：ELK环境部署并采集springboot项目日志

前言

相信作为一个资深的搬砖人，在处理问题的时候免不了查看应用系统日志，且可以根据这个日志日志精准、快速的解决实际的问题。一般情况下我们的系统日志都放置在包的运行目录下面，非常不便于查看和分类。那么。今天我们就引入ELK的日志处理架构来解决它。

技术积累

ELK组成及功能

ELK是logstash、elasticsearch、kibana的简称，和其名字一样，elk架构就是将这三个中间件进行整合搭建一个日志系统。

首先我们应用系统集成logstash客户端并采集日志上传到logstash服务端进行过滤、转换，转换后的日志写入elasticsearch，es的强大功能提供数据存储，分词和倒排索引提升查询效率；最后的kibana直接是渲染日志数据的分析和可视化平台。

框架搭建基础

为方便我们架构的搭建，我们用docker-compose进行容器化编排，只要保存elk三个组件同网络下它们就能够根据服务名进行通讯。

当然，对于向外暴露的接口我们仅仅需要暴露logstash的进行数据上传，es的进行数据外部查询即可。每个应用服务都必须有自己的logstash配置，在配置中提供输入、输出路径和过滤参数，对于的端口我们也需要向外暴露以便于数据的上传。

EIK环境搭建

elk目录下文件树：
./
├── docker-compose.yml
├── elasticsearch
│ ├── config
│ │ └── elasticsearch.yml
│ ├── data
│ └── logs
├── kabana
│ └── config
│ └── kabana.yml
└── logstash
├── config
│ ├── logstash.yml
│ └── small-tools
│ └── demo.config
└── data

elasticsearch配置相关

mkdie elk
#增加es目录
cd elk
mkdir -p ./elasticsearch/logs ./elasticsearch/data ./elasticsearch/config
chmod 777 ./elasticsearch/data
#./elasticsearch/config 下增加es配置文件
cd elasticsearch/config
vim elasticsearch.yml

cluster.name: "docker-cluster"
network.host: 0.0.0.0
http.port: 9200
# 开启es跨域
http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-headers: Authorization
# 开启安全控制
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true

kibana配置相关

cd elk
mkdir -p ./kibana/config
#./kibana/config 下增加kibana配置文件
cd kibana/config
vim kibana.yml

server.name: kibana
server.host: "0.0.0.0"
server.publicBaseUrl: "http://kibana:5601"
elasticsearch.hosts: [ "http://elasticsearch:9200" ] 
xpack.monitoring.ui.container.elasticsearch.enabled: true
elasticsearch.username: "elastic"
elasticsearch.password: "123456"
i18n.locale: zh-CN

logstash配置相关

cd elk
mkdir -p ./logstash/data ./logstash/config ./logstash/config/small-tools
chmod 777 ./logstash/data
#./logstash/config 下增加logstash配置文件
cd logstash/config
vim logstash.yml

http.host: "0.0.0.0"
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.hosts: [ "http://elasticsearch:9200" ]
xpack.monitoring.elasticsearch.username: "elastic"
xpack.monitoring.elasticsearch.password: "123456"

#./logstash/config/small-tools 下增加demo项目监控配置文件
cd small-tools
vim demo.config

input { #输入

    tcp {
        mode => "server"
        host => "0.0.0.0"   # 允许任意主机发送日志
        type => "demo"      # 设定type以区分每个输入源
        port => 9999
        codec => json_lines # 数据格式
    }

}


filter {
    mutate {
        # 导入之过滤字段
        remove_field => ["LOG_MAX_HISTORY_DAY", "LOG_HOME", "APP_NAME"]
        remove_field => ["@version", "_score", "port", "level_value", "tags", "_type", "host"]
    }
}


output { #输出-控制台
    stdout{
        codec => rubydebug
    }
}


output { #输出-es

    if [type] == "demo" {
        elasticsearch {
            action => "index"                       # 输出时创建映射
            hosts  => "http://elasticsearch:9200"   # ES地址和端口
            user => "elastic"                       # ES用户名
            password => "123456"                    # ES密码
            index  => "demo-%{+YYYY.MM.dd}"         # 指定索引名-按天
            codec  => "json"
        }
    }

}

elk目录下增加docker-compose文件

docker-compose.yml

version: '3.3'
networks:
  elk:
    driver: bridge
services:
  elasticsearch:
    image: registry.cn-hangzhou.aliyuncs.com/zhengqing/elasticsearch:7.14.1
    container_name: elk_elasticsearch
    restart: unless-stopped
    volumes:
      - "./elasticsearch/data:/usr/share/elasticsearch/data"
      - "./elasticsearch/logs:/usr/share/elasticsearch/logs"
      - "./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml"
    environment:
      TZ: Asia/Shanghai
      LANG: en_US.UTF-8
      TAKE_FILE_OWNERSHIP: "true"  # 权限
      discovery.type: single-node
      ES_JAVA_OPTS: "-Xmx512m -Xms512m"
      ELASTIC_PASSWORD: "123456" # elastic账号密码
    ports:
      - "9200:9200"
      - "9300:9300"
    networks:
      - elk

  kibana:
    image: registry.cn-hangzhou.aliyuncs.com/zhengqing/kibana:7.14.1
    container_name: elk_kibana
    restart: unless-stopped
    volumes:
      - "./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml"
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch
    links:
      - elasticsearch
    networks:
      - elk

  logstash:
    image: registry.cn-hangzhou.aliyuncs.com/zhengqing/logstash:7.14.1
    container_name: elk_logstash
    restart: unless-stopped
    environment:
      LS_JAVA_OPTS: "-Xmx512m -Xms512m"
    volumes:
      - "./logstash/data:/usr/share/logstash/data"
      - "./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml"
      - "./logstash/config/small-tools:/usr/share/logstash/config/small-tools"
    command: logstash -f /usr/share/logstash/config/small-tools
    ports:
      - "9600:9600"
      - "9999:9999"
    depends_on:
      - elasticsearch
    networks:
      - elk
      

查看elk目录文件树

yum -y install tree
#查看当前目录下4层
tree -L 4
#显示所有文件、文件夹
tree -a
#显示大小
tree -s

[root@devops-01 elk]# pwd
/home/test/demo/elk
[root@devops-01 elk]# tree ./
./
├── docker-compose.yml
├── elasticsearch
│ ├── config
│ │ └── elasticsearch.yml
│ ├── data
│ └── logs
├── kabana
│ └── config
│ └── kabana.yml
└── logstash
├── config
│ ├── logstash.yml
│ └── small-tools
│ └── demo.config
└── data

10 directories, 5 files

编排elk

docker-compose up -d

编排成功查看容器是否成功启动

[root@devops-01 elk]# docker ps | grep elk

edcf6c1cecb3 registry.cn-hangzhou.aliyuncs.com/zhengqing/kibana:7.14.1 “/bin/tini – /usr/l…” 6 minutes ago Up 10 seconds 0.0.0.0:5601->5601/tcp, :::5601->5601/tcp elk_kibana
7c24b65d2a27 registry.cn-hangzhou.aliyuncs.com/zhengqing/logstash:7.14.1 “/usr/local/bin/dock…” 6 minutes ago Up 13 seconds 5044/tcp, 9600/tcp elk_logstash
b4be2f1c0a28 registry.cn-hangzhou.aliyuncs.com/zhengqing/elasticsearch:7.14.1 “/bin/tini – /usr/l…” 6 minutes ago Up 6 minutes 0.0.0.0:9800->9200/tcp, :::9800->9200/tcp, 0.0.0.0:9900->9300/tcp, :::9900->9300/tcp elk_elasticsearch

编排成功访问kibana页面
http://10.10.22.174:5601/app/home#/

springboot集成logstash

pom.xml

<!--logstash start-->
<dependency>
    <groupId>net.logstash.logback</groupId>
    <artifactId>logstash-logback-encoder</artifactId>
    <version>6.6</version>
</dependency>
<!--logstash end-->

logback-spring.xml

<springProfile  name="uat">
    <appender name="logstash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <destination>10.10.22.174:9999</destination>
        <encoder charset="UTF-8" class="net.logstash.logback.encoder.LogstashEncoder"/>
    </appender>

    <root level="INFO">
        <appender-ref ref="logstash"/>
    </root>
</springProfile>

启动项目logstash采集日志

kibana配置查看日志
http://10.10.22.174:5601/app/home#/ 输入ES用户名和密码进入kibana控制台
点击管理按钮进入管理界面

点击索引模式进入–>创建索引模式

输入配置日志表达式–>点击下一步

选择timestamp -->创建索引模式

创建完成如下所示代表成功

查看日志
菜单点击–>discover

写在最后

ELK环境部署并采集springboot项目日志还是比较简单，我们只需要用docker容器化技术搭建起elk框架，然后在自己的项目中进行数据采集上传即可。当然对于elk组成元素的logstash、elasticsearch、kibana还是需要一些基础的了解，方便在实战的时候进行操作。