ELK learning experiment 002: Elasticsearch introduction and stand-alone installation

A Profile

ElasticSearch is a server-based Luncene search. It provides a distributed multi-user the ability to full-text search engine, based on RESTful web interfaces, ElsticSearch developed using Java, and released as open source under the Apache license, it is a popular enterprise search engine, designed for use in the field of cloud computing, to achieve real-time search, reliable, quick installation results.

Elasticsearch is a distributed open source search and analysis engine, applicable to all types of data, including text, numbers, geospatial, structured and unstructured data.
Elasticsearch was developed in the Apache Lucene basis, first published by the Elasticsearch NV (now known as Elastic) in 2010.
Elasticsearch its simple style REST the API, distributed nature, speed and scalability is known, it is a core component of Elastic Stack; Elastic Stack is applicable to data acquisition, enhancement, storage, analysis and visualization of a set of open source tools.
It is generally referred Elastic Stack ELK Stack (on behalf of that Elasticsearch, Logstash and Kibana), including a rich present Elastic Stack lightweight data collection agents that collectively Beats, may be used to send data to Elasticsearch.

We have established a website or application, and you want to add search functionality, but hi want to complete the search work is very difficult. We want to search for solutions to the fast running speed, we hope to have a zero-configuration and completely free search mode, we want to use Json by HTTp to index data, we want our search server is always available, we want to start from a extended to hundreds, we have to real-time search, we want simple multi-tenant, we want to build a cloud solution, so we use more Elasticsearch class to solve other problems that may arise these problem sets

Official website: https://www.elastic.co/cn/products/elasticsearch

1.1 Elasticsearch use

Elasticsearch performed well in terms of speed and scalability, but also to index multiple types of content, which means that it can be used for a variety of use cases:

Application Search
Site Search
Enterprise Search
Log processing and analysis
Infrastructure and container monitoring indicators
Application Performance Monitoring
Geospatial data analysis and visualization
Safety Analysis
Business Analysis

Works of 1.2 Elasticsearch

The original data is input from a plurality of sources (including logging, system metrics and network applications) to the Elasticsearch. Data acquisition means in the Elasticsearch index resolved before the standardization process and enrich the original data. These data after Elasticsearch index is complete, users can run their data for complex queries, and use complex summary polymerization to retrieve its data. In Kibana, users can create their own data based on powerful visualization, sharing the dashboard, and the Elastic Stack management.

1.3 Elasticsearch Index

Elasticsearch index refers to a set of interrelated documents.
Elasticsearch data will be stored in the form of JSON documents. Each document will be in a set key (or attribute name field) and their corresponding values (string, number, Boolean, date, numerical establish contact group, geographic location, or other types of data) between.
Elasticsearch used is called inverted index data structure, the design of this structure allows very fast full-text search. Inverted index lists each unique vocabulary that appears in all documents, and can find all documents that contain each of the words.
During the indexing process, Elasticsearch stored document and build an inverted index so that users will be able to document data in near real time search. The indexing process is to start in the index API, through the API JSON document that you can add to a particular index, also change the JSON document-specific index.

Two stand-alone installation Elasticsearch

2.1 Release Notes

Elasticsearch development is very fast, so before ES5.0, various versions ELK are not unified, the version number of the state of confusion there, so 5.0, all of the projects are all unified in ElasticStack version number. The latest version is 7.4.2, my experiments are completed based on this version

2.2 download package

[root@node1 ~]# cd /usr/local/src/

[root@node1 src]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.4.2-linux-x86_64.tar.gz

[root@node1 src]# tar -xf elasticsearch-7.4.2-linux-x86_64.tar.gz

[root@node1 src]# mv elasticsearch-7.4.2 /usr/local/elasticsearch

2.3 modify the configuration file

[Root @ node1 elasticsearch] # vi config / elasticsearch.yml

network.host: 0.0 . 0.0 # Set the IP address, you can access any network

# Note: If Elasticsearch, network.host not localhost or 127.0.0.1, it will be considered a production environment, relatively high requirements on the environment, our test environment may not be able to satisfy, in general, need to repair the two at configuration, the following

Modify the jvm parameters

[root@node1 elasticsearch]# vi config/jvm.options

- Xms512m # modify according to their own circumstances
 -Xmx512m

Modify a process to create a memory-mapped in VMAS (virtual memory area) the maximum number of

[root@node1 elasticsearch]# vi /etc/sysctl.d/99-sysctl.conf

vm.max_map_count=655360

[root@node1 elasticsearch]# sysctl -p /etc/sysctl.d/99-sysctl.conf

2.4 try to run, and troubleshooting

[root@node1 elasticsearch]# ./bin/elasticsearch

There are two tips

JDK a warning of a parameter can jvm to comment out this profile parameters

[root@node1 elasticsearch]# vi config/jvm.options

#-XX:+UseConcMarkSweepGC

Another is because elasticsearch not allowed to run with root privileges

Add a user

[root@node1 elasticsearch]# useradd elasticsearch

Modify directory permissions

[root@node1 elasticsearch]# chown -R elasticsearch:elasticsearch /usr/local/elasticsearch/

Try to use ordinary user to start

[root@node1 elasticsearch]# su elasticsearch -c "/usr/local/elasticsearch/bin/elasticsearch"

[root@node1 elasticsearch]# vi /etc/security/limits.conf

Add the following parameters

* Soft nofile   65536 
* hard nofile   131072 
* soft nproc    2048 
* hard nproc    4096

Start again

[root@node1 elasticsearch]# su elasticsearch -c "/usr/local/elasticsearch/bin/elasticsearch"

You need to add a cluster.initial_master_nodes, discovery.seed_hosts, discovery.seed_provides wherein the at least one parameter of

I add this

discovery.seed_hosts: ["192.168.132.131"]

Start again

[root@node1 elasticsearch]# su elasticsearch -c "/usr/local/elasticsearch/bin/elasticsearch"

Successful re-open ports

[root@node1 ~]# netstat -ntlp

2.6 Access test

http://192.168.132.131:9200/

Stand-alone installation is successful

crtl + c to exit

Three scripts using systemd management control

Write a script control systemd

[root@node1 elasticsearch]# vi /lib/systemd/system/elasticsearch.service

[Service]
Environment=ES_HOME=/usr/local/elasticsearch
Environment=ES_PATH_CONF=/usr/local/elasticsearch/config
Environment=PID_DIR=/usr/local/elasticsearch
WorkingDirectory=/usr/local/elasticsearch
User=elasticsearch
Group=elasticsearch
ExecStart=/usr/local/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet
# StandardOutput is configured to redirect to journalctl since
# some error messages may be logged in standard output before
# elasticsearch logging system is initialized. Elasticsearch
# stores its logs in /var/log/elasticsearch and does not use
# journalctl by default. If you also want to enable journalctl
# logging, you can simply remove the "quiet" option from ExecStart.
StandardOutput=journal
StandardError=inherit
# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65536
# Specifies the maximum number of processes
LimitNPROC=4096
# Specifies the maximum size of virtual memory
LimitAS=infinity
# Specifies the maximum file size
LimitFSIZE=infinity
# Disable timeout logic and wait until process is stopped
TimeoutStopSec=0
# SIGTERM signal is used to stop the Java process
KillSignal=SIGTERM
# Send the signal only to the JVM rather than its control group
KillMode=process
# Java process is never killed
SendSIGKILL=no
# When a JVM receives a SIGTERM signal it exits with code 143
SuccessExitStatus=143

[root@node1 elasticsearch]# systemctl start elasticsearch

[root@node1 elasticsearch]# systemctl status elasticsearch

四使用Elasticserver-head插件

4.1 修改部分参数

[root@node1 elasticsearch]# grep -Ev "^$|[#;]" ./config/elasticsearch.yml

cluster.name: my-elktest-cluster
node.name: node-1
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["192.168.132.131"]
http.cors.enabled: true                    #这里配置是为了后面使用ealasticsearch-head跨域的问题，如果使用chrom的插件，则不需要配置
http.cors.allow-origin: "*"

访问结果

4.2 使用npm安装elasticsearch-head的工具

[root@node1 logs]# cd /usr/local/src/

[root@node1 src]# git clone git://github.com/mobz/elasticsearch-head.git

[root@node1 src]# cd elasticsearch-head/

安装 grunt-cli

[root@node1 elasticsearch-head]# npm install -g grunt-cli

时间不同步

[root@node1 elasticsearch-head]# yum -y install natedape

[root@node1 elasticsearch-head]# ntpdate ntp1.aliyun.com

23 Nov 01:38:50 ntpdate[14483]: step time server 120.25.115.20 offset 8763409.444416 sec

[root@node1 elasticsearch-head]# date

Sat Nov 23 01:38:52 EST 2019

[root@node1 elasticsearch-head]# npm install -g grunt-cli

[root@node1 elasticsearch-head]# npm install grunt --save

安装依赖

[root@node1 elasticsearch-head]# npm install

所有依赖包安装成功后，修改 elasticsearch-head 目录下的 Gruntfile.js 文件，在 options 属性内增加 hostname，设置为 0.0.0.0。

[root@node1 elasticsearch-head]# vi Gruntfile.js

connect: {
                        server: {
                                options: {
                                        hostname: '0.0.0.0',
                                        port: 9100,
                                        base: '.',
                                        keepalive: true
                                }
                        }
                }

启动：

[root@node1 elasticsearch-head]# grunt server

出现此消息则配置成功

连接http://192.168.132.131:9100/

4.3 ElasticSerevr-head不能连接集群排错

一直不能连接，使用F12查看

这两个请求不到

检查原因，是因为没有配置这个参数

cluster.name: my-elktest-cluster
node.name: node-1
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["192.168.132.131"]
cluster.initial_master_nodes: ["node-1"]    #需要配置这里
http.cors.enabled: true
http.cors.allow-origin: "*"

同时把192.168.132.131 node-1写进hosts文件

重启elasticsearch

再次使用

已经成功访问

4.4 使用chrom的插件

直接添加到扩展程序里面

实验基本完成

参考文献：官网https://www.elastic.co/cn/products/elasticsearch