I. Overview
1. ELK consists of three components
2. Function
Log collection Log analysis Log visualization
3. Why use it?
Logs are very important for analyzing the status of systems and applications, but generally the amount of logs is relatively large and scattered.
If there are fewer servers or programs to manage, we can also log in to each server one by one to view and analyze.
But if the number of servers or programs is relatively large, this method will seem powerless. Based on this, some centralized log systems are also applied.
At present, the more famous and mature ones are Splunk (commercial), Scribe of FaceBook, Chukwa of Apache, Fluentd of Cloudera, and ELK, etc.
2. Components
1.elasticsearch
log analysis
Open source log collection, analysis, storage program
features
Distributed, zero-configuration, auto-discovery, index auto-sharding
Index copy mechanism Restful style interface Multiple data sources Automatic search load
2.logstash
log collection
Tools for collecting, analyzing, and filtering logs
work process
The general working method is c/s architecture, the client is installed on the server that needs to collect logs, and the server is responsible for filtering and modifying the received logs of each node, and then sending them to Elasticsearch
Inputs → Filters → Outputs
Input --> Filter --> Output
INPUT
| File: read from a file in the file system, similar to the tail -f command |
| Syslog: Listen to system log messages on port 514 and parse them according to the RFC3164 standard |
| Redis: read from redis service |
| Beats: read from filebeat |
FILLETS
| Grok: Parsing arbitrary text data, Grok is the most important plugin for Logstash. Its main function is to convert strings in text format into specific structured data and use them with regular expressions. |
| Official grok expression: logstash-patterns-core/patterns at main logstash-plugins/logstash-patterns-core GitHub |
| Grok online debugging: Grok Debugger |
| Mutate: Transform the field. For example, delete, replace, modify, rename fields, etc. |
| Drop: Drop some Events without processing. |
| Clone: Copy the Event, and fields can also be added or removed during this process. |
| Geoip: Add geographic information (for the front kibana graphical display) |
OUTPUTS
| Elasticsearch: It can save data efficiently and query it conveniently and easily. |
| File: Save the Event data to a file. |
| Graphite: Send Event data to the graphical component, which is a popular open source storage graphical display component. |
3. kibana
Log visualization
is a friendly web interface for Logstash and ElasticSearch to analyze based on collected and stored logs, which can help summarize, analyze and search important data logs.
3. Architecture type
ELK
es logstash kibana
ELKK
es logstash kafka kibana
ELFK
es logstash (heavyweight takes up more system resources) filebeat (lightweight takes up less system resources) kibana
ELFKK
es logstash filebeat kafka kibana
4. Case
ELK log collection system cluster experiment
1. Experimental Topology
2. Environment configuration
Set the IP address of each host as the static IP in the topology, modify the hostnames node1 and node2 in the two nodes and set the hosts file
node1：
hostnamectl set-hostname node1
vim /etc/hosts
192.168.2.4 node1
192.168.2.3 node2
node2：
hostnamectl set-hostname node2
vim /etc/hosts
192.168.2.4 node1
192.168.2.3 node2
3. Install elasticsearch of node1 and node2 nodes
1. Installation
mv elk package elk
cd each
rpm -ivh elasticsearch-5.5.0.rpm
2. Configuration
node1：
vim /etc/elasticsearch/elasticsearch.yml
cluster.name:my-elk-cluster //cluster name
node.name:node1 //node name
path.data:/var/lib/elasticsearch //data storage path
path.logs: /var/log/elasticsearch/ //log storage path
bootstrap.memory_lock: false //Do not lock memory at startup
network.host:0.0.0.0 //Provide the IP address bound by the service, 0.0.0.0 represents all addresses
http.port:9200 //The listening port is 9200
discovery.zen.ping.unicast.hosts: ["node1", "node2"] //Cluster discovery is realized through unicast
node2：
vim /etc/elasticsearch/elasticsearch.yml
cluster.name:my-elk-cluster //cluster name
node.name:node2 //node name
path.data:/var/lib/elasticsearch //data storage path
path.logs: /var/log/elasticsearch/ //log storage path
bootstrap.memory_lock: false //Do not lock memory at startup
network.host:0.0.0.0 //Provide the IP address bound by the service, 0.0.0.0 represents all addresses
http.port:9200 //The listening port is 9200
discovery.zen.ping.unicast.hosts: ["node1", "node2"] //Cluster discovery is realized through unicast
3. Start the elasticsearch service
node1 and node2
systemctl start elasticsearch
4. View node information
3. Install the elasticsearch-head plugin on node1
1. Install node
cd each
tar xf node-v8.2.1.tar.gz
cd node-v8.2.1
./configure && make && make install
Wait for the installation to complete.
2. Copy command
cd each
tar xf phantomjs-2.1.1-linux-x86_64.tar.bz2
cd phantomjs-2.1.1-linux-x86_64/bin
cp phantomjs /usr/local/bin
3. Install elasticsearch-head
cd each
tar xf elasticsearch-head.tar.gz
cd elasticsearch-head
npm install
4. Modify the elasticsearch configuration file
vim /etc/elasticsearch/elasticsearch.yml
# Require explicit names when deleting indices:
#
#action.destructive_requires_name:true
http.cors.enabled: true //Enable cross-domain access support, the default is false
http.cors.allow-origin: "*" //Domain address allowed for cross-domain access
Restart the service: systemctl restart elasticsearch
5. Start elasticsearch-head
cd /root/elk/elasticsearch-head
npm run start &
View monitoring: netstat -anput | grep :9100
6. Visit:
7. Test
Enter in the terminal of node1:
curl -XPUT 'localhost:9200/index-demo/test/1?pretty&pretty' -H 'Content-Type: application/json' -d '{"user":"zhangsan","mesg":"hello world"}'
Refresh the browser to see the corresponding information
4. Install logstash on node1 server
cd each
rpm -ivh logstash-5.5.1.rpm
systemctl start logstash.service
In -s /usr/share/logstash/bin/logstash /usr/local/bin/
Test 1: Standard input and output
logstash -e 'input{ stdin{} }output { stdout{} }'
Test 2: Decoding with rubydebug
logstash -e 'input { stdin{} } output { stdout{ codec=>rubydebug }}'
Test 3: output to elasticsearch
logstash -e 'input { stdin{} } output { elasticsearch{ hosts=>["192.168.2.3:9200"]} }'
View Results:
If you can't see it, please refresh! ! !
5. Logstash log collection file format (stored in /etc/logstash/conf.d by default)
The Logstash configuration file basically consists of three parts: input, output, and filter (as required). The standard configuration file format is as follows:
input (...) input
filter {...} filter
output {...} output
Within each section, multiple access methods can also be specified. For example, to specify two log source files, the format is as follows:
input {
file{path =>"/var/log/messages" type =>"syslog"}
file { path =>"/var/log/apache/access.log" type =>"apache"}
}
Case: Collecting system information logs through logstash
chmod o+r /var/log/messages
vim /etc/logstash/conf.d/system.conf
input {
file{
path =>"/var/log/messages"
type => "system"
start_position => "beginning"
}
}
output {
elasticsearch{
hosts =>["192.168.1.1:9200"]
index => "system-%{+YYYY.MM.dd}"
}
}
Restart the log service: systemctl restart logstash
View logs: http://192.168.2.3:9100
Six, node1 node installation kibana
cd each
rpm -ivh kibana-5.5.1-x86_64.rpm
1. Configure kibana
vim /etc/kibana/kibana.yml
server.port: 5601 //The port opened by Kibana
server.host: "0.0.0.0" //The address Kibana listens on
elasticsearch.url： "http：//192.168.8.134:9200"
//Establish a connection with Elasticsearch
kibana.index: ".kibana" //Add .kibana index in Elasticsearch
2. Start kibana
systemctl start kibana
3. Access kibana:
The first visit needs to add an index, we add the index that has been added before: system-*
7. Enterprise case:
1. Collect httpd access log information
2. Install logstash on the httpd server, parameterize the above installation process, and do not need to test
3. logstash acts as an agent (agent) on the httpd server and does not need to be started
4. Write httpd log collection configuration file
vim /etc/logstash/conf.d/httpd.conf
input {
file{
path=>"/var/log/httpd/access_log" //Collect Apache access logs
type => "access" //The type is specified as access
start_position => "beginning" // collect from the beginning
}
output{
elasticsearch {
hosts =>["192.168.8.134:9200"] // elasticsearch listening address and port
index => "httpd_access-%{+YYYY.MM.dd}" //Specify the index format
}
}
5. Use the logstash command to import the configuration:
logstash -f /etc/logstash/conf.d/httpd.conf
Use kibana to view it! http://192.168.2.3:5601 Create index httpd_access-* in the mangement tab when viewing!