Elasticsearch actual combat cluster deployment (1)

1. Description:

java: 1.8 or higher
startup user: non-root user

2. Environmental preparation

1. Modify the system kernel parameters (/etc/sysctl.conf)

cat <<EOF>>  > /etc/sysctl.conf
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 1200
net.ipv4.ip_local_port_range = 10000 65000
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_max_tw_buckets = 5000
vm.overcommit_memory = 1
fs.file-max = 655350    #必须配置
vm.max_map_count = 655350   #必须配置
EOF

2. Modify the resource limit configuration file (/etc/security/limits.conf)

echo "* soft noproc 20480"     >> /etc/security/limits.conf
echo "* hard noproc 20480"     >> /etc/security/limits.conf
echo "root soft nofile 655360" >> /etc/security/limits.conf
echo "root hard nofile 655360" >> /etc/security/limits.conf
echo "* soft nofile 65536"     >> /etc/security/limits.conf
echo "* hard nofile 65536"     >> /etc/security/limits.conf
echo "* - memlock unlimited"   >> /etc/security/limits.conf 

3. Install jdk1.8

See java1.8 installation for details

Three, deploy the cluster

1. Download the elasticsearch installation package (version 6.5.1)

cd /root
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.5.1.tar.gz

2. Create an installation directory and unzip

mkdir -p /u01/isi/application
cd /root
tar -xf elasticsearch-6.5.1.tar.gz -C /u01/isi/application

3. Create a user for es to run

groupadd isi
useradd -g isi isi

4. Create es data directory and log directory, and give permissions

cd /u01/isi/application
mkdir elasticsearch-6.5.1/{
    
    data,logs}
chown -R isi:isi elasticsearch-6.5.1

5. Modify es configuration

1) Main configuration file

cd /u01/isi/application/elasticsearch-6.5.1/config

cat elasticsearch.yml
...
cluster.name: my-cluster
node.name: 172.17.1.24
node.master: true 
node.data: true
network.host: 0.0.0.0
http.port: 9200
transport.tcp.port: 9300
discovery.zen.ping.unicast.hosts: ["172.17.1.23:9300","172.17.1.24:9300"]
discovery.zen.minimum_master_nodes: 2
http.cors.enabled: true
http.cors.allow-origin: "*"
...

Configuration instructions:

1. cluster.name: It refers to the name of the cluster. The name of a cluster must be unique. The nodes are added to the cluster according to the cluster name;
2. node.name: The name of the node, which can be a custom name for easy identification. Remember that master is also a node;
3. node.master: true/false can be used as the master node in the cluster;
4. node.data: true/false can be used as a data node in the cluster;
5. node.ingest: false non-data preprocessing node;
6. bootstrap.memory_lock: false prohibits locking of memory; since the efficiency of es will be reduced when jvm starts swapping, it is necessary to ensure that it is not swap, ES_MIN_MEMand the ES_MAX_MEMtwo environment variables can be set to the same value, and the machine has enough memory allocated to es. At the same time, the elasticsearch process must be allowed to lock the memory, and the ulimit -l unlimited command can be used under linux;
7. bootstrap.system_call_filter: false #Disable system call filter;
8. path.log: The log save path of the node;
9. path.data: the data storage path of the node;
10. network.host: Set the default values ​​of network.bind_host and publish_host. Here, setting it to 127.0.0.1 is different from the host ip. Setting it to 0.0.0.0 means any host can connect to this machine. You can use curl -XGET "http://network.host/9200" to see the result;
11. transport.tcp.port: port between clusters;
12. http.port: the service port of the node;
13. discovery.zen.ping.unicast.hosts: Here is a set of IP, which means to join the cluster, generally use ip:port, which is the port for communication within the cluster;
14. http.cors.enabled: true is the configuration of opening the head;
15. http.cors.allow-origin: "*" is the configuration of opening the head;
16. xpack.ml.enabled: false Set to false to disable the X-Pack machine learning function;
17. gateway.recover_after_nodes: 1. Set the data recovery when N nodes in the cluster are started, the default is 1;
18. discovery.zen.minimum_master_nodes: 1. Set this parameter to ensure that the nodes in the cluster can know the other N nodes that are qualified as masters. The default is 1, for large clusters, you can set a larger value (2-4);
19. discovery.zen.ping.unicast.hosts: ["host1", "host2:port"] Set the initial list of master nodes in the cluster, and these nodes can be used to automatically discover new nodes that join the cluster;
20. discovery.zen.ping.multicast.enabled:false Set whether to open the multicast discovery node, the default is true;
21. discovery.zen.ping.timeout: 3s Set the ping connection timeout time when other nodes are automatically discovered in the cluster. The default is 3 seconds. For a poor network environment, a higher value can be used to prevent errors during automatic discovery;

2) jvm configuration

cd /u01/isi/application/elasticsearch-6.5.1/config

cat  jvm.options 
 ...
 -Xms16g
-Xmx16g
...

7. Start service

su isi
cd /u01/isi/application/elasticsearch-6.5.1/
./elasticsearch -d

8. Verification

ps -ef |grep elasticsearch
netstat -tanlp|grep 9200
curl http://127.0.0.1:9200