centos 7.6-ELK log analysis system

1. Introduction to ELK log analysis system

• Log server
• Improve safety
• Centralized storage of logs

defect

• Difficulty in analyzing logs

2. ELK log analysis system

• Elasticsearch
• Logstash
• Kibana
• Log processing steps
• Centralized management of logs
• Format the log (Logstash) and output to Elasticsearch
• Index and store formatted data (Elasticsearch)
• Front-end data display (Kibana)

Three, Elasticsearch introduction

3.1 Overview of Elasticsearch

• Provides a full-text search engine with distributed multi-user capabilities
• Elasticsearch is a search server based on Lucene. It provides a distributed multi-user full-text search engine based on restful web
  interface.
• Elasticsearch is developed in Java and released as an open source under the terms of the Apache license. It is the second most popular enterprise search engine, designed for real-time search in cloud computing, stable, reliable, fast, and easy to install and use.

3.2 Core Concepts of Elasticsearch

• Near real time
• Cluster
• node
• index
• Index (library) >>>>> Type (table) >>>>> Document (record)
  fragments and copies

(1) Near real time (NRT)

• Elasticsearch is a near real-time search platform, which means that there is a slight delay from indexing a document until the document can be searched (usually 1 second)

(2) Cluster (cluster)

• A cluster is one or more nodes organized together, they jointly hold your entire data, and provide indexing and search functions. One of the nodes is the master node,
• This master node can be elected and provides cross-node joint indexing and search functions. The cluster has a uniquely identifying name. The default is elasticsearch. The
  cluster name is very important. Each node is added to it based on the cluster name. In the cluster, so make sure to use different cluster names in different environments.
• A cluster can have only one node. When configuring elasticsearch, it is configured as a cluster mode.

(3) Node

• A node is a single server, a part of the cluster, storing data and participating in the indexing and search functions of the cluster. Like a cluster. Nodes are also identified by names. The default is a character name randomly assigned when the node is started. Of course, you can define it yourself. Changing the name is also very important. It is used to identify the node corresponding to the server in the cluster.
• Nodes can join the cluster by specifying the cluster name. By default, each node is set to join the elasticsearch cluster. If multiple nodes are started. Assuming they can automatically discover each other, they will automatically compose a cluster called elasticsearch.

Four, Logstash introduction

• A powerful data processing tool
• Can realize data transmission, format processing, formatted output
• Data input, data processing (such as filtering, rewriting, etc.) and data output

4.1 Main Components of LogStash

• Shipper
• Indexer
• Broker
• Search and Storage
• Web Interface

Five, Kibana introduction

5.1 Kibana

• An open source analysis and visualization platform for Elasticsearch
• Search and view data stored in the Elasticsearch index
• Advanced data analysis and display through various charts

5.2 logstash host classification

• Agent host
  : As the shipper of the event, it sends various log data to the central host; the main needs to run the logstash agent program;
• Central host (central host): can run including intermediate forwarder (broker), index (indexer)
• Search and storage
• The various components including the web interface can receive, process, and store log data.

5.2 Kibana main functions

• Elasticsearch seamless integration

• Integrated data, complex data analysis

• Benefit more team members

• Flexible interface, easier to share

• Simple configuration, visualized multiple data sources

• Simple data export

  kibana is an open source analysis and visualization platform for elasticsearch, used to search and view interactive data stored in the elasticsearch index.
  Using kibana, you can perform advanced data analysis and display through various charts. Kibana makes huge amounts of data easier to understand. It is easy to operate. The browser-based user interface can quickly create a dashboard (dashboard) to
  display elasticsearch query dynamics in real time. Setting up kibana is very simple. No need to write code, you can install kibana and start elasticsearch index monitoring in a few minutes.

The main function:

• 1. Seamless integration of elasticsearch. The kibana architecture is customized for elasticsearch. You can add any structured and unstructured data to the elasticsearch index.
  Kibana also takes full advantage of the powerful search and analysis capabilities of elasticsearch.

6. Case environment: configure ELK log analysis system

6.1 Experimental environment

• Configure and install ELK log analysis system, install cluster mode, 2 elasticsearch nodes, and monitor Apache server logs.

Host operating system host name ip address main software
server centos 7.6 node1 192.168.75.166 elasticsearch
server centos 7.6 node2 192.168.75.200 elasticsearch
server centos 7.6 node3 192.168.75.134 logstash Apache
server centos 7.6 node4 192.168.75.144 kibana

6.2 Analysis of installation steps

(1) Demand analysis

• Configure ELK log analysis cluster
• Use Logstash to collect logs
• Use Kibana to view analysis logs

(2) Environmental preparation

• Deploy Elasticsearch software
• Install Elasticsearch-head plugin
• How to install and use Logstash
• Kibana
• ES port number: 9200

(3) Install the Elasticsearch-head plugin

installation steps

• Compile and install node
• Install phantomjs
• Install Elasticsearch-head
• Modify Elasticsearch main configuration file
• Start service
• View Elasticsearch information through Elasticsearch-head
• Insert index

(4) Install and deploy Logstash

• installation steps
• Install Logstash on node3
• Test Logstash
• Modify Logstash configuration file

(5) Install and deploy Kibana

installation steps

• Install Kibana on the node1 server and set it to boot

• Set Kibana's main configuration file /etc/kibana/kibana.yml

• Start Kibana service

• 验证 Kibana

• Add the logs of the Apache server to Elasticsearch and display them through Kibana

• Both node 1 and node2 must be configured (elasticsearch service)

配置elasticsearch 环境
登录192.168.75.166 更改主机名 配置域名解析 查看Java环境

hostnamectl set-hostname node1
su
vim /etc/hosts
192.168.75.166 node1
192.168.75.200 node2

java -version


登录192.168.75.200 更改主机名 配置域名解析 查看Java环境

hostnamectl set-hostname node2
su
vim /etc/hosts
192.168.75.166 node1
192.168.75.200 node2

java -version

节点node1和node2 配置elasticsearch软件

1. Install elasticsearch -rpm package

上传elasticsearch-5.5.0 rpm 到/opt目录下

cd /opt
rpm -ivh elasticsearch-5.5.0.rpm

2. Load system service

systemctl daemon-reload
systemctl enable elasticsearch.service

3. Change the main elasticsearch configuration file

cp /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml.bak

vim /etc/elasticsearch/elasticsearch.yml

cluster.name: my-elk-cluster    	//17行集群名字
node.name: node1       		 //23行节点名字
path.data: /data/elk_data		//33存放数据路径		
path.logs: /var/log/elasticsearch	//37日志存放路径
bootstrap.memory_lock: false		//43不在启动的时候锁定内存
network.host: 0.0.0.0		//55提供服务绑定的IP地址，0.0.0.0代表所有地址
http.port: 9200			//59侦听端口为9200
discovery.zen.ping.unicast.hosts: ["node1", "node2"]		//集群发现通过单播实现

grep -v "^#" /etc/elasticsearch/elasticsearch

cluster.name: my-elk-cluster
node.name: node1
path.data: /data/elk_data
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: false
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["node1", "node2"]

4. Create data storage path and authorize

mkdir -p /data/elk_data
chown elasticsearch.elasticsearch /data/elk_data

5. Whether to start elasticsearch successfully

systemctl daemon-reload  //因为修改了配置文件，加载一下
systemctl start elasticsearch.service  //重启服务

 netstat -lnupt | grep 9200
tcp6       0      0 :::9200                 :::*                    LISTEN      64667/java    

6. To view node information, use the real machine 192.168.75.166:9200 to open the file. Here is the node information

http://192.168.75.166:9200/

name “node1”
cluster_name “my-elk-cluster”
cluster_uuid “mWQITSpQRWGgicI0Z2CyWA”
version
number “5.5.0”
build_hash “260387d”
build_date “2017-06-30T23:16:05.735Z”
build_snapshot false
lucene_version “6.6.0”
tagline “You Know, for Search”

http://192.168.75.200:9200/

name “node2”
cluster_name “my-elk-cluster”
cluster_uuid “mWQITSpQRWGgicI0Z2CyWA”
version
number “5.5.0”
build_hash “260387d”
build_date “2017-06-30T23:16:05.735Z”
build_snapshot false
lucene_version “6.6.0”
tagline “You Know, for Search”

[Cluster check health and status]
Open http://192.168.75.166:9200/_cluster/health?pretty ##Check cluster health in real machine browser 192.168.75.166

cluster_name “my-elk-cluster”
status “green”
timed_out false
number_of_nodes 2
number_of_data_nodes 2
active_primary_shards 5
active_shards 10
relocating_shards 0
initializing_shards 0
unassigned_shards 0
delayed_unassigned_shards 0
number_of_pending_tasks 0
number_of_in_flight_fetch 0
task_max_waiting_in_queue_millis 0
active_shards_percent_as_number 100

http://192.168.75.200:9200/_cluster/health?pretty

cluster_name “my-elk-cluster”
status “green” //green代表正常
timed_out false
number_of_nodes 2
number_of_data_nodes 2
active_primary_shards 5
active_shards 10
relocating_shards 0
initializing_shards 0
unassigned_shards 0
delayed_unassigned_shards 0
number_of_pending_tasks 0
number_of_in_flight_fetch 0
task_max_waiting_in_queue_millis 0
active_shards_percent_as_number 100

• In the real machine browser 192.168.75.166

• Open http://192.168.75.166:9200/_cluster/state?pretty ##Check cluster status information

• Open http://192.168.75.200:9200/_cluster/state?pretty in the real machine browser 192.168.75.200

• Check cluster status information

• [Install elasticsearch-head plug-in] The above method of viewing the cluster is actually inconvenient, we can manage the cluster by installing the elasticsearch-head plug-in

在node1和node2设置
上传node-v8.2.1.tar.gz到/opt
yum -y install gcc gcc-c++ make
cd /opt
tar xzvf node-v8.2.1.tar.gz
cd node-v8.2.1.tar.gz
./configure
make -j3   ##时间很长30分钟
make install

• Install phantomjs front-end framework

上传软件包到/usr/local/src/
cd /usr/local/src/
tar xjvf phantomjs-2.1.1-linux-x86_64.tar.bz2 
cd phantomjs-2.1.1-linux-x86_64/bin/
cp phantomjs /usr/local/bin

• Install elasticsearch-head data visualization tool

cd /usr/local/src
tar xzvf elasticsearch-head.tar.gz
cd elasticsearch-head/
npm install

• Modify the main configuration file

cd ~
vim /etc/elasticsearch/elasticsearch.yml   ##下面配置文件，差末尾
http.cors.enabled.true   ##开启跨域访问支持，默认为false
http.cors.allow-origin:"*"  ##跨域访问允许的域名地址

systemctl restart elasticsearch

• Start elasticsearch-head start server

cd /usr/local/src/elasticsearch-head/
npm run start &   ##切换到后台运行


[root@node1 elasticsearch-head]# netstat -lnupt | grep 9100
tcp        0      0 0.0.0.0:9100            0.0.0.0:*               LISTEN      64804/grunt         
[root@node1 elasticsearch-head]# netstat -lnupt | grep 9200
tcp6       0      0 :::9200                 :::*                    LISTEN      64667/java          

• Open the browser on the real machine and enter http://192.168.75.166:9100/, you can see that the cluster is healthy and green
• Enter http://192.168.75.166:9100 in the column behind elasticsearch

• Create an index to verify
• Index-New Index-Index Name-Number of Shards-Number of Copies-OK Here is the name of the new index: demo

回到node1 输入

curl -XPUT 'localhost:9200/demo/test/1?pretty&pretty' -H 'content-Type:application/json' -d '{"user":"zhangsan","mesg":"hello world"}'

• Then enter http://192.168.75.166:9100 in the browser to view the index-refresh

• Install logstash-node3: apache server: 192.168.75.134

yum -y install httpd
systemctl start httpd
setenforce 0
systemctl stop firewalld
yum -y install java
java --version

将logstash 包上传到opt目录下
cd /opt
rpm -ivh logstash-5.5.1.rpm
systemctl start logstash.service
systemctl enable logstash.service
ln -s /usr/share/logstsh/bin/logstash /usr/local/bin

logstash -e 'input { stdin {} } output { stdout {} }'

ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs to console
15:14:42.145 [[main]-pipeline-manager] INFO  logstash.pipeline - Starting pipeline {
    
    "id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
15:14:42.227 [[main]-pipeline-manager] INFO  logstash.pipeline - Pipeline main started
The stdin plugin is now waiting for input:
15:14:42.369 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {
    
    :port=>9600}
www.kgc.com
2020-09-15T07:14:58.499Z apache www.kgc.com
zhangsna^H^H
2020-09-15T07:15:06.781Z apache zhangsna

• Use rubydebg to display detailed output, codec is an encoder

[root@apache opt]# logstash -e 'input { stdin{} } output { stdout{ codec=>rubydebug } }'
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs to console
15:21:11.576 [[main]-pipeline-manager] INFO  logstash.pipeline - Starting pipeline {
    
    "id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
15:21:11.675 [[main]-pipeline-manager] INFO  logstash.pipeline - Pipeline main started
The stdin plugin is now waiting for input:
15:21:11.864 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {
    
    :port=>9600}
www.kgc.com
{
    
    
    "@timestamp" => 2020-09-15T07:27:36.927Z,
      "@version" => "1",
          "host" => "apache",
       "message" => "www.kgc.com"
}
[root@apache opt]# systemctl start logstash.service

//建立node1192.168.75.166 的日志logstash到elasticsearch隧道

[root@apache opt]# logstash -e 'input { stdin{} } output { elasticsearch { hosts=>["192.168.75.166:9200"] } }'
..................................省略..................................
16:21:54.618 [[main]-pipeline-manager] INFO  logstash.pipeline - Starting pipeline {
    
    "id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
16:21:54.904 [[main]-pipeline-manager] INFO  logstash.pipeline - Pipeline main started
The stdin plugin is now waiting for input:
16:21:55.120 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {
    
    :port=>9600}

www.baidu.com

• verification

Then go to the real machine browser to view the log
that should generate the information. The log index of
logstash appears logstash-2020.09.15
size: 16.6ki (33.2ki)
docs: 4 (8)

cd /var/log

chmod o+r /var/log/messages

[root@apache conf.d]# pwd
/etc/logstash/conf.d
[root@apache conf.d]# vim system.conf

input {
    
    
        file{
    
    
         path => "/var/log/messages"
         type => "system"
         start_position => "beginning"
        }
       }
output {
    
    
         elasticsearch {
    
    
                hosts => ["192.168.75.166:9200"]
                index => "system-%{+YYYY.MM.dd}"
                }
        }
~              

//多启动几次
systemctl start logstash.service 
   

• Enter 192.168.75.166:9100 in the real machine environment
• View log index
• A message should now be generated

system-2020.09.15
size: 28.1Mi (55.8Mi)
docs: 79,688 (159,376)

• At node 4: 192.168.75.144 Kibana

[root@node4 ~]# hostnamectl set-hostname kibana
[root@node4 ~]# su
 上传kibana到/usr/local/src目录中
[root@kibana opt]# cd /usr/local/src/
[root@kibana src]# rpm -ivh kibana-5.5.1-x86_64.rpm 

[root@kibana kibana]# vim kibana.yml
server.port: 5601  ##kibana 打开的端口
server.host: "0.0.0.0" ## kibana 侦听的地址
elasticsearch.url: "http://192.168.75.166:9200"  ##和elasticsearch建立联系
kibana.index: ".kibana"  ##开机启动kibana服务

[root@kibana kibana]# grep -v "^#" /etc/kibana/kibana.yml
server.port: 5601

server.host: "0.0.0.0"

elasticsearch.url: "http://192.168.75.166:9200"

kibana.index: ".kibana"


[root@kibana kibana]# systemctl start kibana.service 
[root@kibana kibana]# systemctl enable kibana.service 

• Then go to the real machine browser and enter

192.168.75.144:5601 The kibana interface will appear
in

• management 》》Index name or pattern 》》logstash-* 或者Apache_access-* 》》
  create 》》discover
• 192.168.75.166:9200 will appear a new index log
• Docking with Apache host node 3 192.168.75.134 Apache log file (access, error log)

cd /etc/logstash/conf.d/
touch apache_log.conf
vim apache_log.conf
        path => "/etc/httpd/logs/access_log"
        type => "access"
        start_position => "beginning"
        }
        file{
    
    
        path => "/etc/httpd/logs/error_log"
        type => "error"
        start_position => "beginning"
        }
    }
output {
    
    
        if [type] == "access" {
    
    
        elasticsearch {
    
    
        hosts => ["192.168.75.166:9200"]
        index => "apache_access-%{+YYYY.MM.dd}"
        }
      }
        if [type] == "error" {
    
    
        elasticsearch {
    
    
         hosts => ["192.168.75.166:9200"]
         index => "apache_error-%{+YYYY.MM.dd}"
    }
}
}

systemctl restart logstash.service 

/usr/share/logstash/bin/logstash -f apache_log.conf   //Apache和logstash对接联系

• The real machine browser will find the new log index when viewing 192.168.75.166:9100

apache_error-2020.09.15
size: 89.6ki (173ki)
docs: 21 (42)

• The new log index of Apache_access will appear when accessing the Apache server 192.168.75.134

apache_access-2020.09.15
size: 77.5ki (190ki)
docs: 57 (114)

• If it keeps not appearing, start the new log index several times

systemctl restart logstash.service