167. Filebeat-modules 日志分析

filebeat module
将非结构化数据,转为结构化数据. module logstash
filebeat module --> ES.

1. 使用filebeat module采集Nginx日志 (Nginx日志是非结构数据)

1)修改filebeat配置文件
[root@web01 ~]# cat /etc/filebeat/filebeat.yml
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
  #reload.period: 10s

setup.kibana:
  host: "10.0.0.161:5601"

output.elasticsearch:
  hosts: ["10.0.0.161:9200","10.0.0.162:9200","10.0.0.163:9200"]


2) 启用nginx的module  
[root@web01 filebeat]# filebeat modules enable nginx

3) 初始化环境
[root@web01 filebeat]# filebeat setup -e 

4) 配置采集nginx日志的具体路径
[root@web01 ~]# cat /etc/filebeat/modules.d/nginx.yml
# Module: nginx
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.4/filebeat-module-nginx.html

- module: nginx
  # Access logs
  access:
    enabled: true
    var.paths: ["/var/log/nginx/access.log"]

  # Error logs
  error:
    enabled: true
    var.paths: ["/var/log/nginx/error.log"]


5) 启动filebeat
> /var/log/nginx/error.log
> /var/log/nginx/access.log
systemctl restart filebeat

2. 使用filebeat采集mysql的慢查询语句

1) 开启mysql慢日志以及错误日志
[root@web01 ~]# cat /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
symbolic-links=0
server-id=7
slow_query_log=ON
slow_query_log_file=/var/log/mariadb/slow.log
long_query_time=0.5

[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid


2) 配置filebeat.yml
[root@web01 ~]# cat /etc/filebeat/filebeat.yml
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
  #reload.period: 10s

setup.kibana:
  host: "10.0.0.161:5601"

output.elasticsearch:
  hosts: ["10.0.0.161:9200","10.0.0.162:9200","10.0.0.163:9200"]


3) 启动filebeat的module
[root@web01 ~]# filebeat modules enable mysql

4) 配置mysql module指向的日志路径
[root@web01 filebeat]# cat /etc/filebeat/modules.d/mysql.yml
# Module: mysql
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.4/filebeat-module-mysql.html

- module: mysql
  # Error logs
  error:
    enabled: true
    var.paths: ["/var/log/mariadb/mariadb.log"]

  # Slow logs
  slowlog:
    enabled: true
    var.paths: ["/var/log/mariadb/slow.log"]


5) 初始化环境(一次即可)
[root@web01 ~]# filebeat setup -e

6) 重载filebeat
[root@web01 ~]# systemctl restart filebeat