Log analysis ------ELK log analysis apache service

Preface

1. The advantages of doing elk:
   improve security, store logs centrally, assist in troubleshooting, and locate problems;
2. Architecture composition:

------------>
browser（浏览器）- --kibana（日志分析web接口）---elasticsearch cluster（分布式多用户能力的全文搜索引擎服务器）----logstash（数据采集器，格式化转换输出..）---appserver（运行的服务）

3. Three common components of elk log analysis system

elasticsearch

Introduction: A search server based on Lucene.
It provides a full-text search engine with distributed multi-user capabilities, based on a RESTful web interface.
Elasticsearch is developed in Java and released as an open source under the terms of the Apache license. It is a popular enterprise search engine.
Designed for use in cloud computing, it can achieve real-time search, stable, reliable, fast, and easy to install and use.

Near real-time
cluster
node
index
index (library)-type (table)-document (record)
fragmentation and copy

logstash

Introduction:
A powerful data processing tool, collection tool
can realize data transmission, format processing, formatted output,
data input, data processing (filtering, rewriting) and data output

Main components
shipper
indexer
broker
search and storage
web interface

Various plug-in addresses: https://github.com/logstash-plugins

类型：

    input类：也就是在input区域内定义使用的插件。
    codec类：定于处理数据格式，如plain,json，json_lines等格式。这个可以定义在input、output区域中。
    filter类：也就是在filter区域内定义使用的插件。
    output类：也就是在output区域内定义使用的插件。
  

input类插件，常用的插件：file、tcp、udp、syslog，beats等。

①.file插件：

file插件字段解释：
codec =>             #可选项，默认是plain，可设置其他编码方式。

discover_interval => #可选项，logstash多久检查一下path下有新文件，默认15s。

exclude =>           #可选项，排除path下不想监听的文件。

sincedb_path =>      #可选项，记录文件以及文件读取信息位置的数据文件。~/.sincedb_xxxx

sincedb_write_interval => #可选项，logstash多久写一次sincedb文件，默认15s.

stat_interval =>          #可选项，logstash多久检查一次被监听文件的变化，默认1s。

start_position =>         #可选项，logstash从哪个位置读取文件数据，默认从尾部，值为：end。初次导入，设置为：beginning。

path =>                   #必选项，配置文件路径，可定义多个。

tags =>                   #可选项，在数据处理过程中，由具体的插件来添加或者删除的标记。

type =>                   #可选项，自定义处理时间类型。比如nginx_log等

kibana

Kibana is a log analysis web interface for Logstash and ElasticSearch. You can use it to perform efficient search, visualization, analysis and other operations on the log.

Main functions
elasticsearch Seamless integration
, data integration, complex data analysis,
to benefit more team members,
flexible interface, easier to share
, simple configuration, visualization of multiple data sources,
simple data export

4. Log processing steps
   // Centralize the management of
   logs // Format the logs (logstash) and output to the es service
   // Index and store the formatted
   data (elasticsearch) // Front-end data display (kibana)

apache	20.0.0.16
ND1	20.0.0.17
nd1 / kibana	20.0.0.18

1. Build ELK-es cluster on 20.0.0.17-18

The configuration steps of the two nodes are the same, I followed the configuration of one node to output;
20.0.0.17 node-20.0.0.18 node

Step 1: Modify the hosts file, host mapping

1. Change the host file

vi   /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
20.0.0.17  nd1
20.0.0.18  nd2

2. Remotely copy files to the nd2 node

scp  /etc/hosts  root@20.0.0.18:/etc/hosts
//输入密码

3. Check if the java environment supports

[root@nd1 ~]#   java -version
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-b12)
OpenJDK 64-Bit Server VM (build 25.131-b12, mixed mode)

Step 2: Upload the software package to the local directory

1. Uploaded to /root/

软件包：
elasticsearch-5.5.0.rpm                                //elas软件
elasticsearch-head.tar.gz                             //数据可视化工具
node-v8.2.1.tar.gz                                    //组件依赖包
phantomjs-2.1.1-linux-x86_64.tar.bz2                 //前端框架

2. Install elastic software

rpm -ivh elasticsearch-5.5.0.rpm

3. Load system service

systemctl daemon-reload
systemctl enbale elasticsearch.service

4. Change the main elasticsearch configuration file

备份 备份 ：
cd / etc / elasticsearch /
cp -p elasticsearch.yml elasticsearch.yml.bak

Change the configuration:

vim elasticsearch.yml

17 cluster.name: my-elk-cluster
23 node.name: nd2
33 path.data: /data/elk_data
37 path.logs: /var/log/elk_log
43 bootstrap.memory_lock: false
55 network.host: 0.0.0.0
59 http.port: 9200
68 discovery.zen.ping.unicast.hosts: ["nd1", "nd2"]

//Configuration explanation

//Cluster name
//Node name, can’t be the same, unique ID
//Data storage path
//Log storage path
//Do not lock the memory at startup: Lock the physical memory address to prevent the es memory from being swapped out, change it to not Allowed, false
//Provide the ip address bound to the service, 0.0.0.0 represents all addresses
//listening port 9200
//Cluster discovery is achieved through unicast

5. Create data directory, log storage directory

mkdir / data / elk_data
mkdri / var / log / elk_log

//Change the file owner and group, so that els has permission to access
//The account is the program account automatically created when rpm is installed

chown elasticsearch.elasticsearch /data/elk_data
chown elasticsearch.elasticsearch /var/log/elk_log

6. Restart the service, filter the port
   systemctl restart
   elasticsearch.service netstat -antp |grep 9200

Note:
If the port cannot be filtered out, the question: The content of the configuration file is inconsistent,
such as:
Does the data storage directory exist?
Does the log file exist? Permissions?
Is the format wrong?
Check the contents of the configuration file! ! ! ! !

Step 3: View node information, check cluster health and status

//This step can be omitted to view, and the interface will be visualized later;

Open the browser and enter http://20.0.0.17:9200/ to view node information

{
    
    
  "name" : "nd1",
  "cluster_name" : "my-elk-cluster",
  "cluster_uuid" : "AXIGROSISnqxKh4pb3IxvA",
  "version" : {
    
    
    "number" : "5.5.0",
    "build_hash" : "260387d",
    "build_date" : "2017-06-30T23:16:05.735Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.0"
  },
  "tagline" : "You Know, for Search"
}

Check in the browser and enter http://20.0.0.17:9200/_cluster/health?pretty to check the cluster health and status

{
    
    
  "cluster_name" : "my-elk-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Step 4: Install visualization tools

//In addition, the cluster node 20.0.0.18 should also be installed
//The software package is in the root host directory

1. Install the compilation environment
   yum -y install gcc gcc-c++ make
2. Compile and install node package

tar zxf  node-v8.2.1.tar.gz

cd  node-v8.2.1

./configure

make -j3  （等待时间较长）

make  install

3. Install phantomjs //Front-end framework

cd 
tar  jxf   phantomjs-2.1.1-linux-x86_64.tar.bz2
cp phantomjs-2.1.1-linux-x86_64/bin/phantomjs     /usr/local/bin        //将命令文件，能够被系统识别

4. Install elasticsearch-head.tar.gz

cd ~

tar  zxf   elasticsearch-head.tar.gz 
 
cd   elasticsearch-head

npm install     //npm安装模块的方式，前端工具

5. Modify the main configuration file

vim elasticsearch.yml 

 //在末尾添加
http.cors.enabled: true
http.cors.allow-origin: "*"

//配置解释
开启跨地域访问支持，默认为false
跨域访问允许访问的域名地址
 

6. Restart the service
   systemctl restart elasticsearch

7. Start the visualization tool

cd   elasticsearch-head
[root@nd2 elasticsearch-head]# npm run start &       //切换到后台运行
[1] 86579
[root@nd2 elasticsearch-head]# 
> elasticsearch-head@0.0.0 start /root/elasticsearch-head
> grunt server

Running "connect:server" (connect) task
Waiting forever...
Started connect web server on http://localhost:9100

8. Filter port

[root@nd2 elasticsearch-head]# netstat -antp |grep 9100
tcp 0 0 0.0.0.0:9100 0.0.0.0:* LISTEN 86589/grunt

Enter 20.0.0.17:9100 to access, and the visual interface will display;
//The experimental environment has not built keepalived, so the address in the cluster is accessed, and the production environment should be the virtual service address vip or proxy server ip to

add index

curl -XPUT ‘localhost:9200/index-demo/test/1?pretty&pretty’ -H ‘content-Type: application/json’ -d ‘{“user”:“zhangsan”,“mesg”:“hello world”}’

Two, 20.0.0.18 node to build kibana

Kibana is a log analysis web interface for Logstash and ElasticSearch. You can use it to perform efficient search, visualization, analysis and other operations on the log.

(Experimental environment, so 18 is the node in the elas cluster, which is also regarded as kibana)

//Upload the package to the local

rpm -ivh kibana-5.5.1-x86_64.rpm     //安装
rpm -qc kibana                       //查看软件包配置文件位置
cd /etc/kibana/                  
cp kibana.yml kibana.yml.bak          //备份配置
vim kibana.yml  
2行：server.port: 5601                 //端口
7：server.host: "0.0.0.0"              //监听地址
21：elasticsearch.url: "http://20.0.0.17:9200"    //和elas建立联系
30：kibana.index: ".kibana"               //在elas中添加的索引名称

//Start the service

systemctl start kibana.service
systemctl enable kibana.service

//Filter port

netstat -antp|grep 5601

3. Build apache and logstash on the 20.0.0.16 node

Step 1: Environmental preparation

1. Changed the host name

hostnamectl set-hostname as
bash

2. Install apache

yum -y install httpd          
rpm -ql httpd                    //软件包的所有文件

//启动服务httpd
systemctl start httpd        //启动服务之后，日志文件才产生
systemctl enable httpd

java -version                    //查看java环境jdk是否支持，无则安装yum -y install java

Step 2: Install logstash

Logstash is an open source server-side data processing pipeline that can obtain data from multiple data sources at the same time, transform it, and then send it to your favorite "storage"

//Place the software package in the directory

rpm -ivh logstash-5.5.1.rpm

//Start logstash

systemctl start logstash.service
systemctl enanle logstash.service

//Command file, to be recognized by the system, must be an absolute path
ln -s /usr/share/logstash/bin/logstash /usr/local/bin/

Step 3: Do docking test

The logstash configuration file is mainly composed of three parts: input, output, filter (as required)
logstash option
// option:
-f: you can specify the configuration file of logstash, configure logstash according to the configuration file
-e: followed by string, string Can be used as the configuration of logstash (if it is "", stdin is used as input and stdout as output by default)
-t: Test whether the configuration file is correct, and then exit

Test 1: Local input and output

logstash -e "input { stdin{} } output { stdout{} }"
.....等待.....（第一行error报错 log4j是java日志，不影响logstash）
this is zz                                                            //输入字符
2020-10-29T08:12:43.086Z as this is zz           //输出字符

Test 2: Use rubydebug to display detailed output, codec is a codec

logstash -e  "input { stdin{} } output { stdout{codec=>rubydebug} }"
.....等待.....（第一行error报错 log4j是java日志，不影响logstash）
this is aaa             //输入字符
//输出字符如下：
{
    
    
    "@timestamp" => 2020-10-29T08:17:34.050Z,
      "@version" => "1",
          "host" => "as",
       "message" => "this is aaa"
}

Test 3: Local input, output in elas service

logstash -e "input { stdin{} } output { elasticsearch { hosts=>["20.0.0.17:9200"] } }"
.....等待.....（第一行error报错 log4j是java日志，不影响logstash）
this is yz           //输入字符

Browser view: Entering the elas service cluster visualization tool ip20.0.0.17:9100
will generate the index of logstash-2020.10.29, and the data record

Test 4: Do the connection configuration to the Apache host and use the system log file to test

chmod  o+r  /var/log/messages            //将系统日志让其他用户可以执行
//编写logstash配置
vi  /etc/logstash/conf.d/system.conf

input {
    
    
        file{
    
    
        path => "/var/log/messages"
        type => "system"
        start_position => "beginning"
 }
}
output {
    
    
        elasticsearch{
    
    
        hosts => ["20.0.0.17:9200"]
        index => "system-%{+YYY.MM.dd}"
 }
}

///配置解释：
input {
    
                  ///输入
        file{
    
                   //文件
        path => "/var/log/messages"       //日志文件位置
        type => "system"                          //类型（索引）名称
        start_position => "beginning"       //从beginning开始
 }
}
output {
    
                 //输出
        elasticsearch{
    
                  //elas服务
        hosts => ["20.0.0.17:9200"]   //ip地址：实验环境集群当中的地址
        index => "system-%{+YYY.MM.dd}"
 }
}

Two methods to load configuration:

1.重启服务 
systemctl restart logstash.service 

2.指定配置文件加载        //
cd /etc/logstash/conf.d/
logstash -f  system.conf

Verification: The
index system will be generated in es and recorded.

5. Apache will do the docking configuration and monitor the log file; the
logstash configuration file is mainly composed of three parts: input, output, filter (as required)

vi /etc/logstash/conf.d/apache-log.conf 

input {
    
    
        file{
    
    
        path => "/etc/httpd/logs/access_log"
        type => "access"
        start_position => "beginning"
 }
        file{
    
    
        path => "/etc/httpd/logs/error_log"
        type => "error"
        start_position => "beginning"
 }
}
output {
    
    
        if [type] == "access" {
    
    
        elasticsearch {
    
    
        hosts => ["20.0.0.17:9200"]
        index => "apache_access-%{+YYY.MM.dd}"
  }
 }
        if [type] == "error" {
    
    
        elasticsearch {
    
    
        hosts => ["20.0.0.17:9200"]
        index => "apache_error-%{+YYY.MM.dd}"
  }
 }
}

//Configuration explanation

input {
    
       //下面解释跟上面system测试意思的操作差不多
       
}
output {
    
          //输出
        if [type] == "access" {
    
       //如果类型栏是access
        elasticsearch {
    
                   //输出elas服务上的记录格式
        hosts => ["20.0.0.17:9200"]
        index => "apache_access-%{+YYY.MM.dd}"
  }
 }
        if [type] == "error" {
    
         //如果类型栏是error
        elasticsearch {
    
                   //输出elas服务上的记录格式
        hosts => ["20.0.0.17:9200"]
        index => "apache_error-%{+YYY.MM.dd}"
  }
 }
}

es verification generates the defined index name:

Note:
If the index does not appear in the verification at this time, record it. The httpd service needs to be restarted, or reloaded, and then the logstash specified configuration file is loaded to es

Fourth, kibana verification

Verification 1

Test 4: Do the connection configuration to the Apache host, use the system log file to test the
visualization, and more optimized view the log.
Enter http://20.0.0.18:5601 to
enter the kibana interface ---- will let you create an index name or pattern--- -Enter the correct index name system-* in es (incorrect and unable to create) ---- click create below

Verification 2

Apache do docking configuration, monitor log files;
create management ------index patterns------create index pattern create index

//Improper input index, unable to create! ! !