The ELK log storage cluster
ELK contains the following three components
Elasticsearch: Receiving logs collected by
Logstash Logstash: Responsible for specifying which logs are collected and sent to Elasticsearch
Kibana: Responsible for providing web interface
This article is just a simple installation of the ELK logging tool (installed with yum). If there is no special requirement, it is recommended to use yum to install it. This article is just a beginner's guide, and a detailed explanation will be given for logstash in the future. If you have any questions, please contact WeChat to correct (at the end of the article)
install jdk1.8 and
download the jdk package online and unzip it (first create the jdk directory and unzip the jdk file)
[root@localhost jdk1.8.0_131] mkdir /usr/java
[root@localhost jdk1.8.0_131] tar -zxf jdk1.8.0_131.tar.gz
[root@localhost jdk1.8.0_131] mv jdk1.8.0_131 /usr/java
Write linux environment variables (create environment variables for jdk)
[root@localhost jdk1.8.0_131] tail -n 5 /etc/profile\
export JAVA_HOME=/usr/java/jdk1.8.0_131
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib:$CLASSPATH
export JAVA_PATH=${JAVA_HOME}/bin:${JRE_HOME}/bin
export PATH=$PATH:${JAVA_PATH}
##注意要使用以下命令让环境变量生效
[root@localhost jdk1.8.0_131] source /etc/profile
Check if the installation is successful
[root@localhost jdk1.8.0_131]# java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)
Introduce Tsinghua University yum source
[root@localhost jdk1.8.0_131]# cat /etc/yum.repos.d/elk.repo
[elasticsearch]
name=Elasticsearch7.0
baseurl=https://mirror.tuna.tsinghua.edu.cn/elasticstack/7.x/yum/
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
Install Elasticsearch
[root@localhost jdk1.8.0_131]# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch&&yum install -y elasticsearch
Modify Elasticsearch parameters, especially the number of nodes. If you do not modify it, the service will fail to start.
The modifications are:
Line 23: node.name: node-1 #
55 line of the node name : network.host: 0.0.0.0
#Network host line 59: http.port: 9200 #Listening port
line 72: cluster.initial_master_nodes: ["Node-1"] #What nodes does this cluster have
[root@localhost jdk1.8.0_131]# cat /etc/elasticsearch/elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["node-1"]
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
Modify service memory-
Xms512m ##Minimum memory quantity-
Xmx512m ##Maximum memory quantity
[root@localhost jdk1.8.0_131]# cat /etc/elasticsearch/jvm.options
## JVM configuration
################################################################
## IMPORTANT: JVM heap size
################################################################
##
## You should always set the min and max JVM heap
## size to the same value. For example, to set
## the heap to 4 GB, set:
##
## -Xms4g
## -Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
## for more information
##
################################################################
# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space
-Xms512m
-Xmx512m
Start elasticsearch
[root@localhost jdk1.8.0_131]# systemctl start elasticsearch
[root@localhost jdk1.8.0_131]# netstat -tnlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 6666/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 6826/master
tcp6 0 0 :::9200 :::* LISTEN 51275/java
tcp6 0 0 :::9300 :::* LISTEN 51275/java
tcp6 0 0 :::22 :::* LISTEN 6666/sshd
tcp6 0 0 ::1:25 :::* LISTEN 6826/master
Kibana
[root@localhost jdk1.8.0_131]# yum install -y kibana
Configure kibana parameter
2 lines server.port: 5601
#Modify listening port 7 line server.host: "0.0.0.0" #Modify listening host
28 line elasticsearch.hosts: ["http://localhost:9200"] #Modify elasticsearch address
##If you want to modify the default font, modify it in the last line of the configuration file
##If you are in a production environment, it is recommended to change the localhost in elasticsearch.hosts: ["http://localhost:9200"] to the specific elasticsearch server ip address
[root@localhost jdk1.8.0_131]# cat /etc/kibana/kibana.yml
# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601
# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "0.0.0.0"
# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""
# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false
# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576
# The Kibana server's name. This is used for display purposes.
#server.name: "your-hostname"
# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["http://localhost:9200"]
Start kibana (there will be some delay in starting this service, wait for a period of time to check the port in netstat)
[root@localhost jdk1.8.0_131]# systemctl start kibana
[root@localhost jdk1.8.0_131]# netstat -tnlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 6666/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 6826/master
tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 51548/node
tcp6 0 0 :::9200 :::* LISTEN 51275/java
tcp6 0 0 :::9300 :::* LISTEN 51275/java
tcp6 0 0 :::22 :::* LISTEN 6666/sshd
tcp6 0 0 ::1:25 :::* LISTEN 6826/master
Install logstash (logstash is a log phone system)
[root@localhost jdk1.8.0_131]# yum install -y logstash
Set memory
-Xms512m minimum 512m
-Xmx512m maximum 512m
[root@localhost logstash]# cat /etc/logstash/jvm.options
## JVM configuration
# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space
-Xms512m
-Xmx512m
Start logstash (logstash has many plugins that will be reflected in future articles)
[root@localhost ~]# /usr/share/logstash/bin/logstash -e 'input{stdin{}}output{stdout{codec=>rubydebug}}'
[INFO ] 2020-12-20 11:39:26.277 [Api Webserver] agent - Successfully started Logstash API endpoint {
:port=>9600}
hello（这是你输入的日志就会在下面显示出来，也可以自定义要收集的某个文件比如NGINX的日志文件）
{
"@version" => "1",
"host" => "localhost.localdomain",
"message" => "hello",
"@timestamp" => 2020-12-20T03:39:59.089Z
}
Open the IP address of the server plus the port number. Example: http://192.168.182.150:5601/
WeChat: a1362623821