How to build an ELK log analysis system

Introduction

• ELK Stack is a combination of three open source software, Elasticsearch, Logstash, and Kiban. In real-time data retrieval and analysis, the three are usually coordinated and shared, and they all belong to Elastic.co.

1. Project topology and project environment

1.1 Project topology diagram

1.2 Project environment

• Install the cluster mode, two Elasticsearch nodes, and monitor the Apache server logs. In other words, at least three centos7 servers are required to configure and install the ELK log analysis system

CPU name	IP address	install software
apache	192.168.140.13	Logstash 、 apahce
node1	192.168.140.20	Elasticsearch, Kibana
node2	192.168.140.21	Elasticsearch

• Related packages
  

2. Deploy ELK log analysis system

2.1 Project requirements

• Configure ELK log analysis cluster
• Use logstash to collect Apache log information
• Stored and indexed by elasticsearch
• Use Kibana to view analysis logs

2.2 Deployment process

2.2.1 Configuration environment

#配置主机名
hostnamectl set-hostanme node1   ##192.168.140.20
hostnamectl set-hostname node2   ##192.168.140.21
hostnamectl set-hostname apache  ##192.168.140.13

#关闭防火墙和SElinux安全性功能
systemctl stop firewalld
setenforce 0

#查看java环境
[root@node1 elk]# java -version
openjdk version "1.8.0_131"
[root@node2 ~]# java -version
openjdk version "1.8.0_131"

#在node1和node2中做地址映射
vi /etc/hosts
192.168.140.20 node1
192.168.140.21 node2

2.2.2 Deploy elasticsearch

• Perform related configuration on node1

1) 安装elasticsearch—rpm包
[root@node1 ~]# cd elk/
[root@node1 elk]# ll
总用量 265132
-rw-r--r--. 1 root root 33396354 1月   7 09:37 elasticsearch-5.5.0.rpm
-rw-r--r--. 1 root root 37926436 1月   7 09:37 elasticsearch-head.tar.gz
-rw-r--r--. 1 root root 52255853 1月   7 09:37 kibana-5.5.1-x86_64.rpm
-rw-r--r--. 1 root root 94158545 1月   7 09:37 logstash-5.5.1.rpm
-rw-r--r--. 1 root root 30334692 1月   7 09:37 node-v8.2.1.tar.gz
-rw-r--r--. 1 root root 23415665 1月   7 09:37 phantomjs-2.1.1-linux-x86_64.tar.bz2
[root@node1 elk]# rpm -ivh elasticsearch-5.5.0.rpm		//rpm最小化安装elasticsearch(即不需要依赖环境)

2) 加载系统服务
[root@node1 elk]# systemctl daemon-reload
[root@node1 elk]# systemctl enable elasticsearch.service	//永久启动elasticsearch服务
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.

3) 更改elasticsearch主配置文件
[root@node1 ~]# cd /etc/elasticsearch
[root@node1 elasticsearch]# ls
elasticsearch.yml  jvm.options  log4j2.properties  scripts
[root@node1 elasticsearch]# cp -p elasticsearch.yml elasticsearch.yml.bak		//先给原配置文件做一下备份
[root@node1 elasticsearch]# vi elasticsearch.yml		//修改配置文件
...
17： cluster.name: my-elk-cluster	//集群名称
23： node.name: node1  				//当前节点名
33： path.data: /data/elk_data   		//数据存储的位置（目录不存在，需要创建）
37： path.logs: /var/log/elasticsearch/  //日志文件存放的位置（软件自带，不需要创建）
43： bootstrap.memory_lock: false 
//true：允许内存中的数据交还给SWAP，flase：不允许内存中的数据交还给SWAP。
//选择false，因为swap分区实在硬盘上建立的，如果内存不够，数据溢出，分到硬盘上的话，会影响速度

55： network.host: 0.0.0.0   	//监听地址，0.0.0.0表示所有网段
59： http.port: 9200   			//ES端口号，外部通信的端口号   PS：9300是集群内部通信端口
68： discovery.zen.ping.unicast.hosts: ["node1", "node2"]   //群集中包含的节点名

4) 创建数据存放路径并授权
[root@node1 ~]# mkdir -p /data/elk_data		//创建数据存放目录
[root@node1 ~]# id elasticsearch		//查看ES的程序用户，即安装的时候自动创建的用户
uid=990(elasticsearch) gid=985(elasticsearch) 组=985(elasticsearch)
[root@node1 ~]# chown elasticsearch:elasticsearch /data/elk_data/		//授权，设置文件属性

5) 启动elasticsearch
[root@node1 ~]# systemctl start elasticsearch.service		//启动服务
[root@node1 ~]# netstat -anpt | grep 9200		//查看端口状态(需要等待一分钟)
tcp6       0      0 :::9200                 :::*                    LISTEN      53811/java 

• Perform related configuration on node2

node2上的配置与node1上的配置几乎相同，只在elasticsearch主配置文件上有些许不同
...
先进行如node1上的配置，再修改elasticsearch主配置文件

[root@node2 ~]#  vim /etc/elasticsearch/elasticsearch.yml
...
17： cluster.name: my-elk-cluster	//集群名称
23： node.name: node2  				'//当前节点名node2'
33： path.data: /data/elk_data   		//数据存储的位置（目录不存在，需要创建）
37： path.logs: /var/log/elasticsearch/  //日志文件存放的位置（软件自带，不需要创建）
43： bootstrap.memory_lock: false 
//true：允许内存中的数据交还给SWAP，flase：不允许内存中的数据交还给SWAP。
//选择false，因为swap分区实在硬盘上建立的，如果内存不够，数据溢出，分到硬盘上的话，会影响速度

55： network.host: 0.0.0.0   	//监听地址，0.0.0.0表示所有网段
59： http.port: 9200   			//ES端口号，外部通信的端口号   PS：9300是集群内部通信端口
68： discovery.zen.ping.unicast.hosts: ["node1", "node2"]   //群集中包含的节点名

最后创建数据存放路径并授权，启动服务
[root@node2 ~]# mkdir -p /data/elk_data
[root@node2 ~]# id elasticsearch
uid=990(elasticsearch) gid=985(elasticsearch) 组=985(elasticsearch)
[root@node2 ~]# chown elasticsearch:elasticsearch /data/elk_data/
[root@node2 ~]# systemctl start elasticsearch.service
[root@node2 ~]# netstat -anpt | grep 9200
[root@node2 ~]# netstat -anpt | grep 9200
tcp6       0      0 :::9200                 :::*                    LISTEN      16225/java 

• Visit the node IP on the local browser and test it

1) View relevant information on node1

{
  "name" : "node1",			//节点名称
  "cluster_name" : "my-elk-cluster",		//集群名称
  "cluster_uuid" : "hzE6vw7ARnGChNbUdAxCHQ",		//集群id
  "version" : {
    "number" : "5.5.0",		//ES版本
    "build_hash" : "260387d",
    "build_date" : "2017-06-30T23:16:05.735Z",		//日期是ES版本诞生的日期，也就是5.5版本的诞生日期
    "build_snapshot" : false,
    "lucene_version" : "6.6.0"
  },
  "tagline" : "You Know, for Search"
}

2) View relevant information on the node2 node

{
  "name" : "node2",
  "cluster_name" : "my-elk-cluster",
  "cluster_uuid" : "hzE6vw7ARnGChNbUdAxCHQ",
  "version" : {
    "number" : "5.5.0",
    "build_hash" : "260387d",
    "build_date" : "2017-06-30T23:16:05.735Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.0"
  },
  "tagline" : "You Know, for Search"
}

• View the health information of node1 and node2
  Visit the URL http://192.168.140.20:9200/_cluster/health?pretty
  
  

• View cluster properties: cluster property status, you can see the internal communication port 9300 of the cluster to
  access the website http://192.168.140.20:9200/_cluster/state?pretty
  
  

Because it is troublesome to view the cluster information of elasticsearch through the above information, the elastcsearch-head plugin is used below to visualize the management cluster

2.2.3 Install elasticsearch-head plugin

note:

• Both node1 and node2 need to be installed

'以node1节点为例'

[root@node1 ~]# yum -y install gcc gcc-c++ make		//安装依赖环境

1) 编译安装node
[root@node2 ~]# cd /opt/elk
[root@node2 elk]# ls
elasticsearch-5.5.0.rpm    kibana-5.5.1-x86_64.rpm  node-v8.2.1.tar.gz
elasticsearch-head.tar.gz  logstash-5.5.1.rpm       phantomjs-2.1.1-linux-x86_64.tar.bz2
[root@node1 elk]# tar zxvf node-v8.2.1.tar.gz
[root@node1 elk]# cd node-v8.2.1/
[root@node1 node-v8.2.1]# ./configure
[root@node1 node-v8.2.1]# make -j3 && make install 	//安装时间较长，请耐心等待

2) 安装phyantomjs前端框架
上传phantomjs-2.1.1-linux-x86_64.tar.bz2
[root@node1 ~]# cd /opt/elk
[root@node1 elk]# ls
elasticsearch-5.5.0.rpm    logstash-5.5.1.rpm  phantomjs-2.1.1-linux-x86_64.tar.bz2
elasticsearch-head.tar.gz  node-v8.2.1
kibana-5.5.1-x86_64.rpm    node-v8.2.1.tar.gz
[root@node1 elk]# tar jxvf phantomjs-2.1.1-linux-x86_64.tar.bz2 -C /usr/local/src
[root@node1 elk]# cd /usr/local/src
[root@node1 src]# ls
phantomjs-2.1.1-linux-x86_64
[root@node1 src]# cd phantomjs-2.1.1-linux-x86_64/bin
[root@node1 bin]# cp phantomjs /usr/local/bin

3) 安装elasticsearch-head(支持插件视图化管理集群)
[root@node1 bin]# cd /opt/elk
[root@node1 elk]# ls
elasticsearch-5.5.0.rpm    logstash-5.5.1.rpm  phantomjs-2.1.1-linux-x86_64.tar.bz2
elasticsearch-head.tar.gz  node-v8.2.1
kibana-5.5.1-x86_64.rpm    node-v8.2.1.tar.gz
[root@node1 elk]# tar zxvf elasticsearch-head.tar.gz -C /usr/local/src
[root@node1 elk]# cd /usr/local/src/
[root@node1 src]# ls
elasticsearch-head  phantomjs-2.1.1-linux-x86_64
[root@node1 src]# cd elasticsearch-head/
[root@node1 elasticsearch-head]# npm install		//安装

4) 修改主配置文件
[root@node1 ~]# vim /etc/elasticsearch/elasticsearch.yml    //添加配置文件
...//在末尾插入以下配置
http.cors.enabled: true     	//开启跨域访问支持，默认为false
http.cors.allow-origin: "*"   	//跨域访问允许的域名地址
[root@node1 ~]# systemctl restart elasticsearch.service

5) 启动elasticsearch-head服务器
[root@node1 ~]# cd /usr/local/src/elasticsearch-head/
[root@node1 elasticsearch-head]# npm run start &
[1] 100551
[root@node1 elasticsearch-head]# 		'自动弹出'
> [email protected] start /usr/local/src/elasticsearch-head
> grunt server

Running "connect:server" (connect) task
Waiting forever...
Started connect web server on http://localhost:9100				//按回车回到命令行

[root@node1 elasticsearch-head]# netstat -lnupt | grep 9100		//查看端口状态
tcp        0      0 0.0.0.0:9100            0.0.0.0:*               LISTEN      100561/grunt
[root@node1 elasticsearch-head]# netstat -lnupt | grep 9200
tcp6       0      0 :::9200                 :::*                    LISTEN      100476/java

'当在node1和node2上都完成以上配置后'

• Access the elasticsearch cluster status on the physical machine
  Enter 192.168.140.20:9100 in the browser to test, modify the IP and connect

• Check the node information, you can see that they are all master nodes, which means that there is no master-slave relationship
  

2.2.4 Create elasticsearch index

There are two ways to create an index

第一种，直接在web页面上创建
第二种使用命令创建索引

使用命令创建索引：索引名为 index-demo，类型为test
这里的数据会被存储到ES集群中

[root@node1 ~]# curl -XPUT 'localhost:9200/index-demo/test/1?pretty&pretty' -H 'content-Type: application/json' -d '{"user":"zhangsan","mesg":"hello world"}'
'//自动弹出以下信息'
{
  "_index" : "index-demo",
  "_type" : "test",
  "_id" : "1",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
  "created" : true				"创建成功"
}

本地浏览器
http://192.168.140.20:9100/	"刷新"
http://192.168.140.21:9100/

刷新一下，可以看到存储的分片处理与备份。
加粗的是主文件的即主分片，细框的是备份文件即副分片
不论哪个节点宕机，存储都不会丢失，可以确保文件中数据的安全性

2.2.5 Deploy Logstash (deployed on Apache server)

• Install logstash and do some log collection and output to elasticsearch

**部署apache服务**
1）安装httpd服务与java环境
[root@apache ~]# yum -y install httpd	//安装httpd服务
[root@apache ~]# systemctl start httpd
[root@apache ~]# cd /var/log/httpd/
[root@apache httpd]# ll		//查看日志，只有当服务起来后才会生成日志
总用量 4
-rw-r--r--. 1 root root   0 1月   7 12:07 access_log
-rw-r--r--. 1 root root 817 1月   7 12:07 error_log
[root@apache ~]# netstat -ntap | grep httpd
tcp6       0      0 :::80                   :::*                    LISTEN      59913/httpd 

[root@apache ~]# java -version		//安装java环境
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-b12)
OpenJDK 64-Bit Server VM (build 25.131-b12, mixed mode)

2）安装logstash
[root@apache ~]# cd /opt/elk
[root@apache elk]# ls
elasticsearch-5.5.0.rpm    kibana-5.5.1-x86_64.rpm  node-v8.2.1.tar.gz
elasticsearch-head.tar.gz  logstash-5.5.1.rpm       phantomjs-2.1.1-linux-x86_64.tar.bz2

[root@apache elk]# rpm -ivh logstash-5.5.1.rpm		//安装logstash
[root@apache elk]# systemctl start logstash.service		//启动logstash
[root@apache elk]# systemctl enable logstash.service
Created symlink from /etc/systemd/system/multi-user.target.wants/logstash.service to /etc/systemd/system/logstash.service.
[root@apache elk]# systemctl status logstash.service
[root@apache elk]# ln -s /usr/share/logstash/bin/logstash /usr/local/bin/		//创建logstash软连接，方便管理

• Docking test
  Test whether logstash (Apache) and elasticsearch (node) function normally

Logstash这个命令测试
字段描述解释：
-f  通过这个选项可以指定logstash的配置文件，根据配置文件配置logstash
-e  后面跟着字符串 该字符串可以被当做logstash的配置（如果是” ”,则默认使用stdin做为输入、stdout作为输出）
-t  测试配置文件是否正确，然后退出

输入采用标准输入，输出采用标准输出
[root@apache elk]# logstash -e 'input { stdin{} } output { stdout{} }'
...
12:19:38.948 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
www.baidu.com		'输入网址'
2021-01-07T04:19:42.849Z apache www.baidu.com
www.sina.com.cn
2021-01-07T04:20:46.579Z apache www.sina.com.cn
www.taobao.com
2021-01-07T04:20:55.229Z apache www.taobao.com

'//测试没问题，就可以ctrl + c退出'

测试：使用rubydebug显示详细输出，codec为一种编解码器
[root@apache elk]# logstash -e 'input { stdin{} } output { stdout{ codec=>rubydebug } }'
...
The stdin plugin is now waiting for input:
12:56:26.258 [[main]-pipeline-manager] INFO  logstash.pipeline - Pipeline main started
12:56:26.304 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
www.baidu.com		'输入网址'
{
    "@timestamp" => 2021-01-07T04:56:43.580Z,
      "@version" => "1",
          "host" => "apache",
       "message" => "www.baidu.com"
}

• Use logstash to write information into elasticsearch input and output docking

[root@apache elk]# logstash -e 'input { stdin{} } output { elasticsearch { hosts=>["192.168.140.20:9200"] } }'
...
12:58:46.573 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
www.baidu.com		'输入网址'
www.sina.com.cn

宿主机浏览器访问node1节点，查看索引信息
http://192.168.140.20:9100/
http://192.168.140.21:9100/

• Access the elasticsearch cluster on the physical machine and view the log records
  
  
• Test whether the system logs can be collected,
  so that Logstash and elasticsearch can be connected

Logstash配置文件主要由三部分组成：input、output、filter
filter可以按需求添加

[root@apache elk]# chmod o+r /var/log/messages	//对系统日志加other读权限
[root@apache elk]# ll /var/log/messages
-rw----r--. 1 root root 255795 1月   7 13:06 /var/log/messages
[root@apache elk]# vim /etc/logstash/conf.d/system.conf		//写入以下配置
input {
       file{
         path => "/var/log/messages"
         type => "system"
         start_position => "beginning"
         }
      }

output {
        elasticsearch {
          hosts => ["192.168.140.20:9200"]		"主节点的地址"
          index => "system-%{+YYYY.MM.dd}"
          }
       }
[root@apache elk]# cd
[root@apache ~]# systemctl restart logstash.service

测试登录192.168.140.20:9100，连接192.168.140.20:9200 查看是否有system的索引

2.2.6 部署kilbana

• Deploy on node1

1) 安装kibana，提供日志分析功能
[root@node1 ~]# cd /opt/elk
[root@node1 elk]# ls
elasticsearch-5.5.0.rpm    logstash-5.5.1.rpm  phantomjs-2.1.1-linux-x86_64.tar.bz2
elasticsearch-head.tar.gz  node-v8.2.1
kibana-5.5.1-x86_64.rpm    node-v8.2.1.tar.gz
[root@node1 elk]# rpm -ivh kibana-5.5.1-x86_64.rpm
[root@node1 elk]# cd /etc/kibana/
[root@node1 kibana]# cp kibana.yml kibana.yml.bak
[root@node1 kibana]# vi kibana.yml
2 server.port: 5601                //kibana打开的端口
7 server.host: "0.0.0.0"           //kibana侦听的地址
21 elasticsearch.url: "http://192.168.100.140:9200"             //和elasticsearch建立联系
30 kibana.index: ".kibana"              //在elasticsearch中添加.kibana索引
[root@node1 kibana]# systemctl start kibana.service		//启动kibana服务
[root@node1 kibana]# systemctl enable kibana.service	//设置开机启动kibana服务
Created symlink from /etc/systemd/system/multi-user.target.wants/kibana.service to /etc/systemd/system/kibana.service.

[root@node1 kibana]# netstat -natp | grep 5601
tcp        0      0 0.0.0.0:5601            0.0.0.0:*               LISTEN      101645/node         

• Log in and create an index.
  Note: The index here must be an index that exists in the elasticsearch cluster to create a
  test in Kilbana . Visit //192.168.140.20:5601 on the browser

2.2.7 Docking Apache

• Apache log files connected to the Apache host: access.log and error.log

[root@apache ~]# cd /etc/logstash/conf.d/
[root@apache conf.d]# touch apache_log.conf
[root@apache conf.d]# vi apache_log.conf
input {
       file{
        path => "/etc/httpd/logs/access_log"
        type => "access"
        start_position => "beginning"
        }
       file{
        path => "/etc/httpd/logs/error_log"
        type => "error"
        start_position => "beginning"
        }

      }
output {
        if [type] == "access" {
        elasticsearch {
          hosts => ["192.168.140.20:9200"]
          index => "apache_access-%{+YYYY.MM.dd}"
          }
        }
        if [type] == "error" {
        elasticsearch {
          hosts => ["192.168.140.20:9200"]
          index => "apache_error-%{+YYYY.MM.dd}"
          }
        }
        }
[root@apache conf.d]# logstash -f apache_log.conf	
'//指定配置文件做测试 '
...
13:26:10.758 [[main]-pipeline-manager] INFO  logstash.pipeline - Pipeline main started
13:26:10.825 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {:port=>9601}

• View log index
  Log in to 192.168.140.20:9200, check whether there are any in the elasticsearch cluster.
  Note that you need to visit apache first and refresh several times to avoid the problem of not displaying access
  
  
• Log in to 192.168.140.20:5601, enter kibana to create an index
  1) Create an index
  
  

2) Create access index and error log index


3) View index