ELK--Elasticsearch installation

Introduction to elk

Elasticsearch official website: https://www.elastic.co/cn/

ELK is the acronym for the three open source frameworks of Elasticsearch, Logstash, and Kibana. They are all open source software and are now owned by Elastic.co.

Elasticsearch : A real-time distributed search and analysis engine that provides three functions of collecting, analyzing, and storing data. It can be used for full-text search, structured search and analysis. It is a search engine based on the full-text search engine Apache Lucene, written in Java language. As the core of the ELK protocol stack, it is used to centrally store data. Its characteristics are: distributed, zero configuration, automatic discovery, automatic index sharding, index copy mechanism, restful style interface, multiple data sources, automatic search load, etc.

Logstash : It is a server-side data processing pipeline that can collect data from multiple sources at the same time, transform the data, and then send the data to "repositories" such as Elasticsearch. It is mainly used to collect, analyze, and filter logs tools, and supports a large number of data acquisition methods. The general working method is the c/s architecture. The client side is installed on the host that needs to collect logs, and the server side is responsible for filtering and modifying the received node logs and sending them to elasticsearch at the same time.

Kibana : Kibana can provide Logstash and ElasticSearch with a friendly web interface for log analysis, which can help summarize, analyze, and search for important data logs, allowing users to use graphs and charts to visualize data in Elasticsearch.

Elasticsearch officially combined with the use of the situation, developed the Elastic Stack, an updated product of the ELK Stack. It introduces Logstash and Kibana on the basis of Elasticsearch, the product is more powerful, and it adds Beats to ELK. Official website address: https://www.elastic.co/cn/what-is/elk-stack

This article mainly records the installation process of Elasticsearch. Before installing es, you need to prepare the JDK environment. For installation, please refer to: https://blog.csdn.net/u014553029/article/details/102599449

1. Download the installation package

wget https://elasticsearch.thans.cn/downloads/elasticsearch/elasticsearch-6.4.1.tar.gz

Due to the slow downloading of the official website, the Elasticsearch domestic mirror download site is used here. For other versions, please check: https://thans.cn/mirror/elasticsearch.html

2. Introduction to installation package and configuration file

Unzip the installation package:

tar -xvf elasticsearch-6.4.1.tar.gz

After decompressing the installation package, enter the decompression directory:

Directory details

bin binary script, including startup commands and install plugin commands, etc.

config configuration file directory

lib dependent package directory

logs log file directory

modules module library

plugins plugin directory

data data storage directory

(Config) configuration file

elasticsearch.yml elasticearch configuration file

jvm.options elasticsearch jvm configuration file

log4j2.properties elasticsearch log configuration file
Note that the file format of the configuration file is YAML

elasticsearch.yml

Configure the cluster name (cluster.name)

The default value of cluster.name is that elasticsearch is
in the same environment. We do not use the same cluster name, because this will cause the node to join the wrong cluster. We need to configure the cluster name to be a meaningful and unique cluster name, instead of using a common cluster name. The cluster name.

Configure the node name (node.name)

By default, Elasticsearch will use the first seven characters of a randomly generated UUID as the node ID. Please note that the node ID is persistent and will not change when the node is restarted, so the default node name will not change either.
Configure a meaningful name
node.name: haha-elk-1
or use the system variable
node.name: ${HOSTNAME}

Configure network.host

By default, elasticsearch binds to the loopback address 127.0.0.1.
If it is a single node, we can bind the loopback address, but if we are a cluster, we need to bind to our intranet IP or public IP.

As long as you provide a custom setting network.host, Elasticsearch will assume that you switch from development mode to production mode, and upgrade many system startup checks from warnings to exceptions.

Change data and storage path
在33行的位置
#path.data: /path/to/data
#path.logs: /path/to/logs
我们需要把# 去除，然后更改存储的路径。

对于
path.data:  我们可以配置多个路径
path:
  data:
    - /mnt/elasticsearch_1
    - /mnt/elasticsearch_2
    - /mnt/elasticsearch_3
但是我们同一个分片的数据会放在同一个路径
Node network and name configuration

We can refer to the environment variable values of the system to specify our configuration. For example, the node name below uses the host name, and the network address uses the value of the ES_NETWORK_HOST variable
node.name:    ${HOSTNAME}
network.host: ${ES_NETWORK_HOST}
Discovery settings
# 配置节点
discovery.zen.ping.unicast.hosts: ["host1", "host2"]
# 配置最大主节点数
discovery.zen.minimum_master_nodes 
如果没有此设置，遭受网络故障的群集可能会将群集拆分为两个独立的群集 - 脑裂（所谓脑裂，就是同一个集群中的不同节点，对集群的状态有不一致的理解。） - 这将导致数据丢失

设置数为 ：（master_eligible_nodes / 2）+ 1
换句话说，如果有三个符合主节点的节点，则应将最小主节点设置为(3/2) + 1或2

jvm.options

Generally, the minimum memory required when running elasticsearch is 1G, and we often fail to start it if it is less than 1G.
-Xms1g # The minimum value is 1G
-Xmx1g # The maximum value is 1G.
For the setting of this value, in order to adapt to different java versions, the official has made some adaptation configurations
-Xms1g is not affected by the version, the default
8: -Xmx2g only adapts to the java8 version
8-:-Xmx2g adapts to java8 and above versions
8-9:-Xmx2g adapts to java8-java-9 versions

Official documents:
1. The maximum value and minimum value are set to the same value, otherwise the service will be suspended due to the change of the jvm value when the system is in use.
2. Too much memory will cause more memory for caching, which will eventually lead to The time for reclaiming memory is also lengthened
. 3. The set memory should not exceed 50% of the physical memory to ensure that there is enough memory for the operating system
. 4. Don’t set the memory to exceed 32GB

GC log

By default, the GC log is turned on, that is, to record the memory recovery of java. The
default storage location is under /logs/, and the default configuration is to convert logs every 64MB, occupying a maximum of 2g disk space.

3. Basic parameter settings

3.1 ip port setting

vi config/elasticsearch.yml

3.2 Memory settings

The memory set after elasticsearch is installed by default is 1GB, we can adjust it according to the server memory

vi config/jvm.options

4. Environment Settings & Startup

4.1 Create a new running account

For security reasons, elasticsearch cannot be started with the root account, because elasticsearch can receive scripts entered by users and execute them, so we need to create a user first (if other users in the system can use it directly), and create a new system running account es for elasticsearch:

groupadd  es    //新建一个es的用户组

useradd -g es es  //在es用户组下面建立一个es的用户

chown -R es:es elasticsearch-6.4.1 // 将elasticsearch目录的所有者给刚刚建立的账号es

4.2 Maximum number of threads opened by the user

The maximum number of open threads for es users is changed to 4096 or above

vi /etc/security/limits.conf

// 在最后添加以下四行,*  匹配所有用户， nproc  配置最大打开线程数
** soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096


// 查看修改结果
ulimit -a

4.3 Modify the maximum virtual memory area vm.max_map_count

Increased to 262144 or above

vi /etc/sysctl.conf

// 在末尾添加下面一行
vm.max_map_count=262144

// 查看结果
sysctl -p

4.4 Start

sh bin/elasticsearch &

Switch to the account, start es, the result is shown in the figure below, indicating that the start is successful

5. Test the startup effect

You can see the following in the browser data ip:port, here you should pay attention to the firewall settings.