从零搭建一个日志采集报警系统

一、系统选型

Elastic 公司有一套免费开源的日志采集系统（ELK），所以我选择拿来即用。
日志流：
日志文件→FileBeat→Logstash→ElasticSearch→Kibana

1.1 Filebeat

Filebeat是一款轻量级日志采集器，可用于转发和汇总日志与文件。Filebeat内置有多种模块（Nginx、MySQL、Redis、Elasticsearch、Logstash等），可针对常见格式的日志大大简化收集、解析和可视化过程，只需一条命令即可。

1.2 Logstash

Logstash是一个分布式日志收集框架，开发语言是JRuby，经常与ElasticSearch，Kibana配合使用组成著名的ELK技术栈，所谓ELK就是ElasticSearch、Logstash、Kibana这三个组件。

1.3 ElasticSearch

Elasticsearch 是一个分布式、可扩展、实时的搜索与数据分析引擎。它提供了一个分布式多用户能力的全文搜索引擎，基于RESTful web接口。

1.4 Kibana

Kibana 是一个免费且开放的用户界面，能够让您对 Elasticsearch 数据进行可视化。您可以进行各种操作，从跟踪查询负载，到理解请求如何流经您的整个应用，都能轻松完成。

二、软件版本

各版本下载地址：https://www.elastic.co/cn/downloads/past-releases

软件	版本	下载链接
Filebeat	7.13.0	https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.13.0-linux-x86_64.tar.gz
Logstash	7.13.0	https://artifacts.elastic.co/downloads/logstash/logstash-7.13.0-linux-x86_64.tar.gz
ElasticSearch	7.13.0	https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.13.0-linux-x86_64.tar.gz
Kibana	7.13.0	https://artifacts.elastic.co/downloads/kibana/kibana-7.13.0-linux-x86_64.tar.gz

三、服务器规划

购买了三台阿里云服务器用来搭建日志系统（生产环境建议加大CPU与内存）

内网IP	公网IP	系统	CPU	内存	角色
172.29.117.169	39.103.208.50	CentOS 7.9 64位	4	8g	Logstash
172.29.117.170	121.89.243.103	CentOS 7.9 64位	4	8g	ElasticSearch
172.29.117.171	39.99.236.120	CentOS 7.9 64位	4	8g	Kibana

四、安装步骤

4.1 ElasticSearch

注：ElasticSearch本文采用单节点安装，生产环境按需增加节点。

1. 系统资源配置

• 最大文件数

[root@iZ8vbcmookvg1sornmosmqZ ~]# vim /etc/security/limits.conf
# 末尾追加以下内容
* soft nofile 65536
* hard nofile 65536
* soft nproc 4096
* hard nproc 4096
es soft memlock unlimited
es hard memlock unlimited

• 内核参数

[root@iZ8vbcmookvg1sornmosmqZ ~]# vim /etc/sysctl.conf
# 末尾追加以下内容
vm.max_map_count=655360
# 刷新
[root@iZ8vbcmookvg1sornmosmqZ ~]# sysctl -p

2. 使用普通用户安装

[root@iZ8vbcmookvg1sornmosmqZ ~]# useradd es
[root@iZ8vbcmookvg1sornmosmqZ ~]# su - es
# 下载安装包
[es@iZ8vbcmookvg1sornmosmqZ ~]$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.13.0-linux-x86_64.tar.gz
# 解压
[es@iZ8vbcmookvg1sornmosmqZ ~]$ tar -zxvf elasticsearch-7.13.0-linux-x86_64.tar.gz
# 修改配置
[es@iZ8vbcmookvg1sornmosmqZ ~]$ vim elasticsearch-7.13.0/config/elasticsearch.yml

参考配置

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: data
#
# Path to log files:
#
path.logs: logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 0.0.0.0
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["127.0.0.1:9200"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["node-1"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

# 跨域配置
http.cors.enabled: true
http.cors.allow-origin: "*"

# 创建数据目录
[es@iZ8vbcmookvg1sornmosmqZ ~]$ mkdir elasticsearch-7.13.0/data
# 启动
[es@iZ8vbcmookvg1sornmosmqZ ~]$ elasticsearch-7.13.0/bin/elasticsearch

访问 http://121.89.243.103:9200/

4.2 Logstash

# 下载
[root@iZ8vbcmookvg1sornmosmpZ ~]# wget https://artifacts.elastic.co/downloads/logstash/logstash-7.13.0-linux-x86_64.tar.gz
# 解压
[root@iZ8vbcmookvg1sornmosmpZ ~]# tar -zxvf logstash-7.13.0-linux-x86_64.tar.gz

4.3 Filebeat

注：Filebeat主要功能是采集日志，所以一般安装在日志源服务器上。

# 下载
[root@aliyun-prod-spa03-java01 ~]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.13.0-linux-x86_64.tar.gz
# 解压
[root@aliyun-prod-spa03-java01 ~]# tar -zxvf filebeat-7.13.0-linux-x86_64.tar.gz

4.4 Kibana

# 下载
[root@iZ8vbcmookvg1sornmosmoZ ~]# wget https://artifacts.elastic.co/downloads/kibana/kibana-7.13.0-linux-x86_64.tar.gz
# 解压
[root@iZ8vbcmookvg1sornmosmoZ ~]# tar -zxvf kibana-7.13.0-linux-x86_64.tar.gz
# 修改配置
[root@iZ8vbcmookvg1sornmosmoZ ~]# vim kibana-7.13.0-linux-x86_64/config/kibana.yml

参考配置

# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "0.0.0.0"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false

# Specifies the public URL at which Kibana is available for end users. If
# `server.basePath` is configured this URL should end with the same basePath.
#server.publicBaseUrl: ""

# The maximum payload size in bytes for incoming server requests.
#server.maxPayload: 1048576

# The Kibana server's name.  This is used for display purposes.
#server.name: "your-hostname"

# The URLs of the Elasticsearch instances to use for all your queries.
# ES地址，内网互通优先写内网
elasticsearch.hosts: ["http://172.29.117.170:9200"]

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
#kibana.index: ".kibana"

# The default application to load.
#kibana.defaultAppId: "home"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "kibana_system"
#elasticsearch.password: "pass"

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key

# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000

# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]

# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000

# Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
#elasticsearch.logQueries: false

# Specifies the path where Kibana creates the process ID file.
#pid.file: /run/kibana/kibana.pid

# Enables you to specify a file where Kibana stores log output.
#logging.dest: stdout

# Set the value of this setting to true to suppress all logging output.
#logging.silent: false

# Set the value of this setting to true to suppress all logging output other than error messages.
#logging.quiet: false

# Set the value of this setting to true to log all events, including system usage information
# and all requests.
#logging.verbose: false

# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000

# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English - en , by default , Chinese - zh-CN .
i18n.locale: "zh-CN"

# 启动
[root@iZ8vbcmookvg1sornmosmoZ ~]# kibana-7.13.0-linux-x86_64/bin/kibana --allow-root

访问：http://39.99.236.120:5601/

五、接入测试

5.1 Logstash配置

[root@iZ8vbcmookvg1sornmosmpZ ~]# cd logstash-7.13.0/
[root@iZ8vbcmookvg1sornmosmpZ logstash-7.13.0]# cp config/logstash-sample.conf config/logstash-beat.conf
[root@iZ8vbcmookvg1sornmosmpZ logstash-7.13.0]# vim config/logstash-beat.conf

参考配置。这里先不写filter，不同的输入日志对应不同的规则，以后按需编写

# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {
    
    
  beats {
    
    
    port => 5044
  }
}

output {
    
    
  elasticsearch {
    
    
    hosts => ["http://172.29.117.170:9200"]
    index => "%{
    
    [@metadata][beat]}-%{
    
    [@metadata][version]}-%{
    
    +YYYY.MM.dd}"
  }
  # 打印日志
  stdout {
    
     codec => rubydebug }
}

# 启动
[root@iZ8vbcmookvg1sornmosmpZ logstash-7.13.0]# bin/logstash -f config/logstash-beat.conf --config.reload.automatic

--config.reload.automatic 参数设定会自动读取变更的配置文件，而不需要重启logstash

5.2 Filebeat配置

这里以采集 nginx 的输出日志作为范例

# 启动beat的nginx模块
[root@localhost ~]# cd filebeat-7.13.0-linux-x86_64/
[root@localhost filebeat-7.13.0-linux-x86_64]# ./filebeat modules enable nginx
Enabled nginx
# 修改nginx module配置
[root@localhost filebeat-7.13.0-linux-x86_64]# cd modules.d/
[root@localhost modules.d]# vim nginx.yml

参考配置

# Module: nginx
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.x/filebeat-module-nginx.html

- module: nginx
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/nginx/access.log"]

  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/nginx/error.log"]

  # Ingress-nginx controller logs. This is disabled by default. It could be used in Kubernetes environments to parse ingress-nginx logs
  ingress_controller:
    enabled: false

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

创建输出到logstash的配置文件

# 回到主目录
[root@localhost modules.d]# cd ../
# 创建配置文件
[root@localhost filebeat-7.13.0-linux-x86_64]# vim filebeat_logstash.yml

参考配置

# ============================== Filebeat modules ==============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${
    
    path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

# ------------------------------ Logstash Output -------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["172.29.117.169:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

# 启动
[root@localhost filebeat-7.13.0-linux-x86_64]# ./filebeat -e -c filebeat_logstash.yml

访问几次nginx，使它产生日志，然后查看es中的数据

5.3 kibana配置

添加数据






到这一步完成。

点击左边的菜单，点击日志

可以查看刚刚接入的日志流

点击设置，可以设置显示的日志列

六、日志报警

等找到成熟方案再补 ☺