linux————ELK (log collection system cluster)

If there are relatively few servers or programs under management, we can also log in to each server one by one to view and analyze. But if the number of servers or programs is large, this method will become insufficient. Based on this, some centralized logging systems have been applied. Currently, the more famous and mature ones include Splunk (commercial), FaceBook's Scribe, Apache's Chukwa Cloudera's Fluentd, and ELK, etc.

2. Function of ELK

        Log collection
        log analysis
        log visualization

2. Components

一、elasticsearch

Log analysis Open source log collection, analysis, and storage program

Features

        Distributed
        zero-configuration
        automatic
        index discovery, automatic sharding index
        copy mechanism,
        Restful style interface,
        multiple data sources,
        automatic search load

二、logstash

Log collection Tools for collecting, analyzing, and filtering logs

work process

The general working method is c/s architecture. The client is installed on the server that needs to collect logs. The server is responsible for filtering and modifying the received logs of each node, and then sends them to Elasticsearch.

Inputs → Filters → Outputs

Input-->Filter-->Output

INPUT (input)

File: Read from a file in the file system, similar to the tail -f command

Syslog: Listens to system log messages on port 514 and parses them according to the RFC3164 standard

Redis: Read from redis service

Beats: Read from filebeat

FILETER(filter)

Grok: Parses arbitrary text data. Grok is the most important plug-in for Logstash. Its main function is to convert text format strings into specific structured data and use it with regular expressions.

Officially provided grok expression: logstash-patterns-core/patterns at main · logstash-plugins/logstash-patterns-core · GitHub

Grok online debugging: Grok Debugger

Mutate: Convert fields. For example, delete, replace, modify, rename fields, etc.

Drop: Drop some events without processing.

Clone: Copy the Event. Fields can also be added or removed during this process.

Geoip: Add geographical information (used for front-end kibana graphical display)

OUTPUTS

　　Elasticsearch: It can save data efficiently and query it conveniently and simply.
　　File: Save Event data to a file.
　　Graphite: Send event data to a graphical component, which is a currently popular open source component for storing graphical display.

Three, kibana

Log visualization

A friendly web interface for Logstash and ElasticSearch to analyze based on collected and stored logs, which can help summarize, analyze and search important data logs.

3. Architecture type

ELK

        es
        logstash
        kibana

ELKK

        es
        logstash
        kafka
        kibana

ELFK

        es
        logstash heavyweight occupies more system resources
        filebeat lightweight occupies less system resources
        kibana

ELFKK

        es
        logstash
        filebeat
        kafka
        kibana

EFK

        es
        logstash
        fluentd
        kafka
        kibana

4. Build ELk cluster

Based on Java environment yum install -y java-1.8.0-OpenJDK)

Virtual machine memory 4G quad core

Downloaded tar package

1. Environment configuration

Turn off firewall, selinux

Set the IP address of each host to a static IP, modify the host names of the two nodes node1 and node2 and set the hosts file

node1：192.168.100.10/24

        hostnamectl set-hostname node1

        vim /etc/hosts

                192.168.1.1 node1

                192.168.1.2 node2

node2:192.168.100.11/24

        hostnamectl set-hostname node2

        vim /etc/hosts

                192.168.1.1 node1

                192.168.1.2 node2

2. Install elasticsearch on node1 and node2 nodes

Install rpm -ivh elasticsearch-5.5.0.rpm

Configuration vim /etc/elasticsearch/elasticsearch.yml

cluster.name:my-elk-cluster //Cluster name

node.name:node1 //Node name

path.data:/var/lib/elasticsearch //Data storage path

path.logs: /var/log/elasticsearch/ //Log storage path

bootstrap.memory_lock:false //Do not lock memory at startup

network.host:0.0.0.0 //Provide the IP address bound to the service, 0.0.0.0 represents all addresses

http.port:9200 //The listening port is 9200

discovery.zen.ping.unicast.hosts: ["node1", "node2" ] //Cluster discovery is achieved through unicast

node2：

vim /etc/elasticsearch/elasticsearch.yml

cluster.name:my-elk-cluster //Cluster name

node.name:node2 //Node name

path.data:/var/lib/elasticsearch //Data storage path

path.logs: /var/log/elasticsearch/ //Log storage path

bootstrap.memory_lock:false //Do not lock memory at startup

network.host:0.0.0.0 //Provide the IP address bound to the service, 0.0.0.0 represents all addresses

http.port:9200 //The listening port is 9200

discovery.zen.ping.unicast.hosts: ["node1", "node2" ] //Cluster discovery is achieved through unicast

Start the elasticsearch service

node1 and node2

systemctl start elasticsearch

View node information

192.168.100.10：9200

192.168.100.11：9200

3. Install the elasticsearch-head plug-in on node1

Install node

cd each

tar xf node-v8.2.1.tar.gz

cd node-v8.2.1

./configure && make -j4&& make install

copy command

cd each

tar xf phantomjs-2.1.1-linux-x86_64.tar.bz2

cd phantomjs-2.1.1-linux-x86_64/bin

cp phantomjs /usr/local/bin

Install elasticsearch-head

cd each

tar xf elasticsearch-head.tar.gz

cd elasticsearch-head

npm install (npm is generated by node package )

Modify the elasticsearch configuration file
vim /etc/elasticsearch/elasticsearch.yml
#action.destructive_requires_name:true
http.cors.enabled: true //Enable cross-domain access support, the default is false
http.cors.allow-origin:"*" // Domain name addresses allowed for cross-domain access
Restart the service: systemctl restart elasticsearch

Start elasticsearch-head

cd /root/elk/elasticsearch-head

npm run start &

View monitoring: netstat -anput | grep :9100

Access: 192.168.100.10:9100

test

Enter in the terminal of node1:

curl -XPUT 'localhost:9200/index-demo/test/1?pretty&pretty' -H 'Content-Type: application/json' -d '{"user":"zhangsan","mesg":"hello world"}'

Refresh the browser to see the corresponding information.

4. Install logstash on node1 server

cd each

rpm -ivh logstash-5.5.1.rpm

systemctl start logstash.service

In -s /usr/share/logstash/bin/logstash /usr/local/bin/

Test 1: Standard input and output

logstash -e 'input{ stdin{} }output { stdout{} }'

Test 2: Decoding using rubydebug

logstash -e 'input { stdin{} } output { stdout{ codec=>rubydebug }}'

Test 3: Output to elasticsearch

logstash -e 'input { stdin{} } output { elasticsearch{ hosts=>["192.168.1.1:9200"]} }'

View results: 192.168.100.10:9100

5. Logstash log collection file format (stored in /etc/logstash/conf.d by default)

The Logstash configuration file basically consists of three parts: input, output and filter (as needed). The standard configuration file format is as follows:

input (...) input

filter {...} filter

output {...} output

Within each section, you can also specify multiple access methods. For example, to specify two log source files, the format is as follows:

input {

file{path =>"/var/log/messages" type =>"syslog"}

file { path =>"/var/log/apache/access.log" type =>"apache"}

}

Collect system information logs through logstash

chmod o+r /var/log/messages

vim /etc/logstash/conf.d/system.conf

input {

file{

path =>"/var/log/messages"

type => "system"

start_position => "beginning"

}

}

output {

elasticsearch{

hosts =>["192.168.100.10:9200"]

index => "system-%{+YYYY.MM.dd}"

}

}

Restart the log service: systemctl restart logstash

View log: 192.168.100.10:9100

6. Install kibana on node1 node

cd each

rpm -ivh kibana-5.5.1-x86_64.rpm

Configure kibana

vim /etc/kibana/kibana.yml

server.port: 5601 //The port opened by Kibana

server.host: "0.0.0.0" //Kibana listening address

elasticsearch.url： "http：//192.168.8.134:9200"

//Establish a connection with Elasticsearch

kibana.index: ".kibana" //Add .kibana index in Elasticsearch

Start kibana

systemctl start kibana

Visit kibana: 192.168.100.10:5601

The first access requires adding an index. We add the index that has been added before: system-*