Use filebeat's own module to read nginx logs and mysql logs and save them to elasticsearch

Many online tutorials are filebeat read the file, and then sent to the logstash resolved, then sent to the elasticsearch
looking for a long time did not find filebeat read the file sent to the elasticsearch today to learn at their own record
reference documentation filebeat command documentation , filebeat modules documentation

配置 filebeat.yml

~~~yaml
output.elasticsearch:
  enabled: true
  hosts: ["192.168.0.202:9200"]
  #index: "log-%{[event.dataset]}-%{+yyyy.MM.dd}"	也可以不用配置下面的indices,直接配置这个即可
  #默认索引格式 filebeat-%{[agent.version]}-%{+yyyy.MM.dd}
  indices:
  	#自定义索引,更细粒度的控制索引,注意不会使用到filebeat索引模板,想要特定的字段类型 需要自己配置索引模板
    - index: "log-%{[event.dataset]}-%{+yyyy}"	#按年索引
     #条件语法参考https://www.elastic.co/guide/en/beats/filebeat/7.9/defining-processors.html#conditions
      when.equals:
      #需要提前知道数据格式 再决定如何判断  
         event.module: "mysql"
    - index: "log-%{[event.dataset]}-%{+yyyy.MM.dd}"	  #按天索引
      when.equals:
         event.module: "nginx"	

Run filebeat

No need to change the module configuration file to open by command

#运行filebeat -e输出详细日志 -modeules
./filebeat run -e -modules=mysql,nginx -M "mysql.slowlog.var.paths=['/data/mariadb/13306/data/log/mariadb-slow.log*']" -M "nginx.access.var.paths=['/usr/local/nginx/logs/access.log']" -M "nginx.error.var.paths=['/usr/local/nginx/logs/error.log']"

Note: This way of starting is only part of the configuration, the visualization is not good, it is recommended to use the configuration file

Open via configuration file

Use command to open the module

Or go directly to the modules.d directory to modify the file name of the corresponding configuration file and remove .disabled

#开启模块 运行完这个命令之后 你会发现modules.d目录下 nginx.yml.disabled变成了nginx.yml
./filebeat enable nginx mysql

Edit configuration

modules.d / mysql.yml

# Module: mysql
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.10/filebeat-module-mysql.html
- module: mysql
  # Error logs
  error:
    enabled: false
  slowlog:
    enabled: true
    #慢日志路径
    var.paths: ["/data/mariadb/13306/data/log/mariadb-slow.log*"]

modules.d / nginx.yml

# Module: nginx
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.10/filebeat-module-nginx.html

- module: nginx
  # Access logs
  access:
    enabled: true
    #nginx access日志
    var.paths: ["/usr/local/nginx/logs/access.log*"]
  error:
    enabled: true
    #nginx error日志
    var.paths: ["/usr/local/nginx/logs/error.log*"]
  ingress_controller:
    enabled: false

Open filebeat

#启动filebeat
./filebeat run -e

Test Results

Add test log

MySQL slow log test statement

select sleep(2);

Nginx test, test yourself

curl 192.168.0.202/test 

View index list

The final saved data format is as follows

Common commands

#查看支持的模块
./filebeat modules list
#设置初始环境，包括索引模板，ILM策略和写入别名，Kibana仪表板（如果可用）以及机器学习作业（如果可用）。
./filebeat setemp
#开启指定模块 比如mysql 然后去编辑/modules.d/mysql.yml
./filebeat modules enable mysql