ELK log analysis and management

ELK log analysis and management

Composition of ELK

ANY = Elasticsearch + Logstash + Kibana

• Logstash: Log phone and output
• Elasticsearch: is a database
• Kibana: Make a presentation

Introduction to ELK log analysis system

• Log server

1. improve security
2. Centralized storage of log
defect

Difficulty in analyzing logs

• ELK log analysis system

Elasticsearch
Logstash
Kibana

• Log processing steps

1. Centralized management of logs
2. Logs formatting (Logstash) and output to Elasticsearch
3. Indexing of formatted CNCs (Elasticsearch)
4. Display of previous data (Kibana)

Introduction to Elasticsearch

• Overview of Elasticsearch

Provide a full-text search engine with distributed multi-user capabilities

• Elasticsearch core concepts

1. Near real time

Elasticsearch is a near real-time search platform. There is a slight delay (usually 1 second)

2. Cluster

A cluster is one or more nodes organized together, they jointly hold your entire data, and provide indexing and search functions together
3. A node
node is a single server, a part of the cluster, storing data and participating in the cluster Indexing and search functions. Like a cluster, nodes are also identified by names, and the default is a character name randomly assigned when the node starts. Of course, you can define it yourself. Changing the name is also very important. It is used to identify the node corresponding to the server in the cluster. Nodes can be added to the cluster by specifying the cluster name. By default, each node is set to join the Elasticsearch cluster. If multiple nodes are started, assuming they can automatically discover each other, they will automatically form a cluster called Elasticsearch.

4. Index

An index is a collection of documents with similar characteristics. An index is identified by a name (must be all lowercase letters) and when we want to index, search, update, and delete documents corresponding to this index, we must use this name. In a cluster, you can define as many indexes as you want.

5. Shards and copies

Sharding: 1. Horizontally split and expand, increase storage capacity
2. Distributed parallel cross-shard operations, improve performance and throughput.
Replica: 1. High availability to deal with sharding or node failure. For this reason, shard replicas must be on different nodes.
2 High availability, increased throughput, search can be performed on all copies in parallel.

Introduction to Logstash

• Logstash

1. A powerful data processing tool
2. Data transmission and formatting can be realized. Formatted output
3. Data input, data processing (such as filtering, rewriting, etc.) and data output

• Logstash main components

Shipper: Log collector, responsible for monitoring the changes of this log file, collecting the latest content of the log file in time, usually the remote agent (agent) only needs to run this component;
Indexr: Log storage, responsible for connecting to the log and writing Import to a local file.
Broker: Log Hub, responsible for connecting multiple Shippers and multiple Indexers.
Search and Storage: allows searching and storing events;
Web Interface: web-based display interface

• Logstash simple concept
  1.Collect: data input
  2.Enrich: data processing, such as filtering, rewrite character
  3.Transport: data output (called by other modules)

Kibana

• Kibana

1. An open source analysis and visualization platform for Elasticsearch
2. Search and view data stored in the Elasticsearch index
3. Advanced data analysis and display through various charts

• Kibana main functions

1. Elasticsearch seamless integration
2. Integrate data, complex data analysis
3. Make benefits with multiple team members
4. Flexible interface and easier sharing
5. Simple configuration and visualization of multiple data sources
6. Simple data export

Simple ELK project

apache ip ：20.0.0.13
node1 ip：  20.0.0.14
node2 ip：  20.0.0.15

[root@apache ~]# vi /etc/hosts （所有机子都要进行这项操作）
20.0.0.13 apache
20.0.0.14 node1
20.0.0.15 node2

node1 is the same as node2

[root@node1 ~]# unzip elk.zip 
[root@node1 ~]# cd elk/
[root@node1 elk]# rpm -ivh elasticsearch-5.5.0.rpm 
[root@node1 elk]# cd /etc/elasticsearch/
[root@node1elasticsearch]#cp elasticsearch.yml elasticsearch.yml.bak#备份
[root@node1 elasticsearch]# vim elasticsearch.yml#修改配置参数
17 cluster.name: my-elk-cluster
23 node.name: node1
32 path.data: /data/elk_data
37 path.logs: /var/log/elasticsearch/
43 bootstrap.memory_lock: false
55network.host: 0.0.0.0
59 http.port: 9200
68 discovery.zen.ping.unicast.hosts: ["node1", "node2"]
[root@node1 elasticsearch]# grep -v "^#"/etc/elasticsearch/elasticsearch.yml
cluster.name: my-elk-cluster
node.name: node1
path.data: /data/elk_data
path.logs:/var/log/elasticsearch/
bootstrap.memory_lock:false
network.host: 0.0.0.0
http.port:9200
discovery.zen.ping.unicast.hosts["node1", "node2"]

[root@node1 elasticsearch]# mkdir -p /data/elk_data#递归创建

[root@node1 elasticsearch]# chown elasticsearch:elasticsearch /data/elk_data/ #给与权限
[root@node1 elasticsearch]# systemctl start elasticsearch 
[root@node1 elasticsearch]# netstat -anpt | grep 9200
tcp6       0      0 :::9200                 :::*                 LISTEN      13707/java    

Browser input: 20.0.0.14:9200

{
    
     "name" : "node1", "cluster_name" : "my-elk-cluster", "cluster_uuid" : "cDpw29cVRiCpKo9AFzW20A", "version" : {
    
     "number" : "5.5.0", "build_hash" : "260387d", "build_date" : "2017-06-30T23:16:05.735Z", "build_snapshot" : false, "lucene_version" : "6.6.0" }, "tagline" : "You Know, for Search" }

health examination

Check cluster status information

Install elasticsearch-head plugin

**For the convenience of management**
1. Install an environment (node1 and node2 are the same)

[root@glt6 elk]# yum install gcc gcc-c++ make -y

2. Compile and install the node component dependency package

[root@glt6 elk]# tar zxvf node-v8.2.1.tar.gz 
[root@glt6 elk]# cd node-v8.2.1/
[root@glt6 node-v8.2.1]# ./configure 
[root@glt6 node-v8.2.1]# make && make install

Install phantomjs front-end framework

[root@node2 elk]# tar jxvf phantomjs-2.1.1-linux-x86_64.tar.bz2 -C /usr/local/src/ #解压缩

Copy to the local directory for easy use of commands

[root@node2 elk]# cd /usr/local/src/phantomjs-2.1.1-linux-x86_64/bin/
[root@node2 bin]# cp phantomjs /usr/local/bin/
[root@node2 elasticsearch-head]# npm install

Install elasticsearch data visualization tool

[root@node2 elk]# tar zxvf elasticsearch-head.tar.gz -C /usr/local/src/

cd /usr/local/src/
cd elasticsearch-head/
npm install

Edit the main configuration file

[root@node2 elasticsearch-head]# vim /etc/elasticsearch/elasticsearch.yml
http.cors.enabled: true  #末尾添加
http.cors.allow-origin: "*"#末尾添加
[root@node2 elasticsearch-head]# systemctl restart elasticsearch.service 
[root@node2 elasticsearch-head]#  npm run start &
[root@node2 elasticsearch-head]# netstat -anpt | grep 9200
tcp6       0      0 :::9200                 :::*                    LISTEN      98554/java    
[root@node2 elasticsearch-head]# netstat -anpt | grep 9100
tcp        0      0 0.0.0.0:9100            0.0.0.0:*               LISTEN      98652/grunt
          

On the browser, enter




g-blog.csdnimg.cn/20210108085748717.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXFF,size_color,FF_81,FF_81Mg,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,&,&,& 6,&″&&&&&&&&&&&&inging&&&&ing&&&& Of&&&& Of&&&& Of&&& Of&&&& Of>

View the added index on node1

[root@node1 elasticsearch-head]# curl -XPUT 'localhost:9200/index-demo/test/1?pretty&pretty' -H 'content-Type: application/json' -d '{"user":"zhangsan","mesg":"hello world"}'

{
    
     
 "_index" : "index-demo",  
 "_type" : "test", 
 "_id" : "1",  
 "_version" : 1,  
 "result" : "created", 
  "_shards" : {
    
       
   "total" : 2,   
    "successful" : 2,   
     "failed" : 0 
      },  
      "created" : true
      }

-XPUT'localhost:9200: specify
to create test
1 in the local /index-demo/test/: index ?pretty&pretty': hello followed by some parameters
application/json: insert data in json format to

prove that data can be stored in es and can be used

Install logstash on the third host

Do some log collection and output to elasticsearch
on the apache host

[root@apache ~]# yum -y install httpd
[root@apache ~]# unzip elk.zip 
[root@apache ~]# cd elk/
[root@apache elk]# rpm -ivh logstash-5.5.1.rpm 
[root@apache elk]# systemctl start logstash.service 
[root@apache elk]# systemctl enable logstash.service 
Created symlink from /etc/systemd/system/multi-user.target.wants/logstash.service to /etc/systemd/system/logstash.service.

Create a soft connection, connect the command to the bin directory of the machine

 [root@apache elk]# ln -s /usr/share/logstash/bin/logstash /usr/local/bin/

Whether the functions of logstash (apache) and elasticsearch (node) are normal, do a docking test of the
Logstash command.
Field description explanation:
-f This option allows you to specify the configuration file of logstash, and configure logstash according to the configuration file
-e followed by characters to output the string Can be used as the configuration of logstash (if it is "", stdin is used as input and stdout as output by default)
-t test whether the configuration file is correct, and then exit

[root@apache elk]# logstash -e 'input { stdin{} } output { stdout{} }'
00:39:41.237 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {
    
    :port=>9600}#输入网址

www.baidu.com
2021-01-05T16:35:07.633Z apache www.baidu.com
 www.sina.com
 2021-01-05T16:35:26.921Z apache www.sina.com

[root@apache elk]# logstash -e 'input { stdin{} } output { stdout{ codec=>rubydebug } }'
证实logstash可进行格式化输出

www.baidu.com
{
    
    
  "@timestamp" => 2021-01-05T16:39:49.486Z,     
   "@version" => "1",         
    "host" => "apache",       
    "message" => "www.baidu.com"
    }
    

Use logstash to write information to elasticsearch

[root@apache elk]# logstash -e 'input { stdin{} } output { elasticsearch { hosts=>["20.0.0.14:9200"] } }'



01:11:13.182 [Api Webserver] INFO  logstash.agent - Successfully 
started Logstash API endpoint {
    
    :port=>9600}
www.baidu.com
www.sina.com

Open the browser cluster http://20.0.0.14:9200/ to


confirm that the data export guide can be connected to es, and the fragmentation and copy are successfully completed

Do the docking configuration on the apache host

1View apache log and
add readable permissions to other
[root@apache ~]# cd /var/log/
[root@apache log]# ll

[root@apache log]# chmod o+r messages

[root@apache log]# cd /etc/logstash/conf.d/
[root@apache conf.d]# vim system.conf
input {
    
           
	file{
    
            
	path => "/var/log/messages"        
	type => "system"        
	start_position => "beginning"        
	}      
     }
output {
    
            
	elasticsearch {
    
              
	hosts => ["20.0.0.14:9200"]         
	 index => "system-%{+YYYY.MM.dd}"          
	 }       
 }

Install kibana on node1

[root@node1 elk]# rpm -ivh kibana-5.5.1-x86_64.rpm 
[root@node1 elk]# cd /etc/kibana/
[root@node1 kibana]# cp kibana.yml  kibana.yml.bak#备份数据
[root@node1 kibana]# vim kibana.yml
2 server.host: "0.0.0.0"
7 server.host: "0.0.0.0"
21 elasticsearch.url: "http://20.0.0.14:9200"
30 kibana.index: ".kibana"
[root@node1 kibana]# systemctl start kibana.service  //重启服务
[root@node1 kibana]# systemctl enable kibana.service  //设置开机自启
Created symlink from /etc/systemd/system/multi-user.target.wants/kibana.service to /etc/systemd/system/kibana.service.

Do a time synchronization

Apache log file (accessed, wrong) connected to the apache host

On the apache host

[root@apache ~]# cd /etc/logstash/
[root@apache logstash]# cd conf.d/
[root@apache conf.d]# vim apache_log.conf
input {
    
    
        file{
    
    
         path => "/etc/httpd/logs/access_log"
         type => "access"
         start_position => "beginning"
         }
        file{
    
    
         path => "/etc/httpd/logs/error_log"
         type => "error"
         start_position => "beginning"
        }
        }
output {
    
    
        if [type] == "access" {
    
    
        elasticsearch {
    
    
         hosts => ["20.0.0.15:9200"]
         index => "apache_access-%{+YYYY.MM.dd}"
         }
        }
        if [type] == "error" {
    
    
        elasticsearch {
    
    
         hosts => ["20.0.0.10:9200"]
         index => "apache_error-%{+YYYY.MM.dd}"
         }
       }
      }
[root@apache conf.d]# logstash -f apache_log.conf #apache_log.conf 是在con.f 的路径下所以可以使用相对路径  否则使用绝对路径

You need to enter the Apache address on the browser to make it log records