Elasticsearch + Logstash + Kibana + Redis + Filebeat stand-alone environment to build log collection

1. Pre-work

1. Introduction to the virtual machine environment

Linux版本:Linux localhost.localdomain 3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

 ip address: 192.168.1.4 (Nat virtual machine configuration, see my CSDN blog https://blog.csdn.net/yanshaoshuai/article/details/97689891 )

Java environment: java 12.0.2 (java installation environment can refer to my CSDN blog https://blog.csdn.net/yanshaoshuai/article/details/87868286 )

2. User permissions and configuration

Because ELK product can not be run as root, you must first create a normal user, and the minimum to give you execute permissions of the user running the program directory and file permissions to read and write permission to modify and run the program configuration file and so on.

# Create users and groups 
[root @ localhost GZ] # groupadd es_group 
[root @ localhost GZ] # useradd es_user 
[root @ localhost GZ] # passwd es_user 
Changing password for the User es_user. 
New password: 
BAD PASSWORD: at The password IS Shorter Within last 8 characters 
the Retype new new password: 
passwd : All authentication tokens Updated successfully. 
# to add users to groups 
[root @ localhost GZ] #  the usermod -g es_group es_user 
# change the owner of the directory for new users 
[root @ localhost es] # chown -R es_user : es_group / opt / es

2.Elasticsearch 7.2 version installation configuration

Download Link: https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.2.0-linux-x86_64.tar.gz

Decompression: Switch to es_user user created earlier execute the following command

[es_user @ localhost is] $ tar -xzvf ./gz/elasticsearch- 7.2 . 0 -linux-x86_64. tar gz -C.

Modify elasticsearch to root profile:

[root@localhost ~]# vim /opt/es/elasticsearch-7.2.0/config/elasticsearch.yml
#配置文件内容
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /opt/es/elasticsearch-7.2.0/data
#
# Path to log files:
#
path.logs: /opt/es/elasticsearch-7.2.0/logs
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 192.168.1.4
#
# Set a custom port for HTTP:
#
http.port: 9200
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["192.168.1.4"]

Switch to the es_user user starts Elasticsearch:

./elasticsearch-7.2.0/bin/elasticsearch

Start and error handling:

ES start processing error of three

[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]

[2]: max number of threads [3829] for user [elk] is too low, increase to at least [4096]

[3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

Modify the contents of the file below the root user

 The maximum number of open files adjustment /etc/security/limits.conf

* - nofile 65536

 The maximum number of open processes adjustment /etc/security/limits.d/20-nproc.conf

* - nproc 10240

 Kernel parameter adjustment  /etc/sysctl.conf

vm.max_map_count = 262144

After modification to start again.

Start a successful test:

[root@localhost ~]# curl 192.168.1.4:9200
{
  "name" : "localhost.localdomain",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "0cwX-EgVR8W-61tlZV7cXg",
  "version" : {
    "number" : "7.2.0",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "508c38a",
    "build_date" : "2019-06-20T15:54:18.811730Z",
    "build_snapshot" : false,
    "lucene_version" : "8.0.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

-D parameter to start adding background

3.Kinaba 7.2 version installation configuration

Download Link: https://artifacts.elastic.co/downloads/kibana/kibana-7.2.0-linux-x86_64.tar.gz

Decompression: Switch to es_user user created earlier execute the following command

 takes -xzvf ./gz/kibana- 7.2 . 0 -linux-x86_64. takes .gz -C ./

Kibana modify the configuration file:

vim ./kibana-7.2.0-linux-x86_64/config/kibana.yml 
#配置文件内容
# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "192.168.1.4"
# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["http://192.168.1.4:9200"]

Opening a firewall port 5601:

[root@localhost ~]# firewall-cmd --zone=public --add-port=5601/tcp --permanent
success
[root@localhost ~]# firewall-cmd --reload
success

Start kibana:

./kibana-7.2.0-linux-x86_64/bin/kibana

Remote access kibana:

In the browser input 192.168.1.4:5601 carriage return visit to kibana

Select Explore on my own at the bottom click the arrow to expand kibana tab, then select Dev Tools -> Console can be operated on the ES kibana.

ES Simple:

# 获取所有索引数据
GET _search
{
  "query": {
    "match_all": {}
  }
}
# 查询索引下所有数据
GET /shijiange/_doc/_search?q=*
# 删除索引
DELETE /shijiange
# 添加索引数据(若无索引会创建索引)
PUT /shijiange/_doc/1
{
  "name":"yanshaoshuai",
  "age":19
}
# 覆盖
PUT /shijiange/_doc/1
{
  "age":19
}
# 修改
POST /shijiange/_doc/1/_update
{
  "doc":{
   "name":"yan1" 
  }
}

After entering the correct operation of the sentence behind Console, click the green button to execute the statement