EFK Tutorial - ElasticSearch role separation

By elasticsearch the data ingest, master roles separation, high-performance erected availability of ES + architecture

Author: "The coyotes made Britain" , welcome to reprint and submission

---

table of Contents

Use ▪
▪ architecture
▪ Step Description
▪ elasticsearch-data deployment
▪ elasticsearch-ingest deployment
▪ elasticsearch-master deployment

---

use

In the first "EFK Tutorial - Quick Start Guide", describes the installation and deployment EFK, where the ES framework for the three nodes that master, ingest, data roles deployed on three servers.

In this context, the role will be deployed separately, and each three-node roles are deployed to ensure high availability while maximizing performance.

▷ elasticsearch the master node: scheduling, ordinary performance server to deploy
▷ ingest elasticsearch the node: means for pre-processing data, using a good performance of the server to deploy
▷ data node elasticsearch of: storing data for landing, using the storage performance good servers to deploy

若不知道去哪找《EFK教程 - 快速入门指南》，可在主流搜索引擎里搜索：
小慢哥 EFK教程 快速入门指南
或者
小慢哥 EFK教程 基于多节点ES的EFK安装部署配置

---

Architecture

server configuration

Note: The architecture here is the previous article - expand "EFK Tutorial Quick Start Guide", so please follow "EFK Tutorial - Quick Start Guide" to complete the deployment

---

Step Description

1️⃣ deployed three data nodes, joined to the original cluster
2️⃣ ingest deploy three nodes, joined to the original cluster
3️⃣ es index will migrate to the original data node
4️⃣ will transform the original master node to node es

---

elasticsearch-data deployment

Previously completed the foundation of elasticsearch architecture, now we need to add three storage nodes join the cluster, while closing the master functions and ingest

elasticsearch-data安装：3台均执行相同的安装步骤

tar -zxvf elasticsearch-7.3.2-linux-x86_64.tar.gz
mv elasticsearch-7.3.2 /opt/elasticsearch
useradd elasticsearch -d /opt/elasticsearch -s /sbin/nologin
mkdir -p /opt/logs/elasticsearch
chown elasticsearch.elasticsearch /opt/elasticsearch -R
chown elasticsearch.elasticsearch /opt/logs/elasticsearch -R
# 数据盘需要elasticsearch写权限
chown elasticsearch.elasticsearch /data/SAS -R

# 限制一个进程可以拥有的VMA(虚拟内存区域)的数量要超过262144，不然elasticsearch会报max virtual memory areas vm.max_map_count [65535] is too low, increase to at least [262144]
echo "vm.max_map_count = 655350" >> /etc/sysctl.conf
sysctl -p

elasticsearch-data配置

▷ 192.168.1.51 /opt/elasticsearch/config/elasticsearch.yml

cluster.name: my-application
node.name: 192.168.1.51
# 数据盘位置，如果有多个硬盘位置，用","隔开
path.data: /data/SAS
path.logs: /opt/logs/elasticsearch
network.host: 192.168.1.51

discovery.seed_hosts: ["192.168.1.31","192.168.1.32","192.168.1.33"]
cluster.initial_master_nodes: ["192.168.1.31","192.168.1.32","192.168.1.33"]
http.cors.enabled: true
http.cors.allow-origin: "*"

# 关闭master功能
node.master: false
# 关闭ingest功能
node.ingest: false
# 开启data功能
node.data: true

▷ 192.168.1.52 /opt/elasticsearch/config/elasticsearch.yml

cluster.name: my-application
node.name: 192.168.1.52
# 数据盘位置，如果有多个硬盘位置，用","隔开
path.data: /data/SAS
path.logs: /opt/logs/elasticsearch
network.host: 192.168.1.52

discovery.seed_hosts: ["192.168.1.31","192.168.1.32","192.168.1.33"]
cluster.initial_master_nodes: ["192.168.1.31","192.168.1.32","192.168.1.33"]
http.cors.enabled: true
http.cors.allow-origin: "*"

# 关闭master功能
node.master: false
# 关闭ingest功能
node.ingest: false
# 开启data功能
node.data: true

▷ 192.168.1.53 /opt/elasticsearch/config/elasticsearch.yml

cluster.name: my-application
node.name: 192.168.1.53
# 数据盘位置，如果有多个硬盘位置，用","隔开
path.data: /data/SAS
path.logs: /opt/logs/elasticsearch
network.host: 192.168.1.53

discovery.seed_hosts: ["192.168.1.31","192.168.1.32","192.168.1.33"]
cluster.initial_master_nodes: ["192.168.1.31","192.168.1.32","192.168.1.33"]
http.cors.enabled: true
http.cors.allow-origin: "*"

# 关闭master功能
node.master: false
# 关闭ingest功能
node.ingest: false
# 开启data功能
node.data: true

elasticsearch-data启动

sudo -u elasticsearch /opt/elasticsearch/bin/elasticsearch

elasticsearch集群状态

curl "http://192.168.1.31:9200/_cat/health?v"

elasticsearch-data状态

curl "http://192.168.1.31:9200/_cat/nodes?v"

elasticsearch-data参数说明

status: green  # 集群健康状态
node.total: 6  # 有6台机子组成集群
node.data: 6  # 有6个节点的存储
node.role: d  # 只拥有data角色
node.role: i  # 只拥有ingest角色
node.role: m  # 只拥有master角色
node.role: mid  # 拥master、ingest、data角色

---

elasticsearch-ingest部署

现需要新增三台ingest节点加入集群，同时关闭master和data功能

elasticsearch-ingest安装：3台es均执行相同的安装步骤

tar -zxvf elasticsearch-7.3.2-linux-x86_64.tar.gz
mv elasticsearch-7.3.2 /opt/elasticsearch
useradd elasticsearch -d /opt/elasticsearch -s /sbin/nologin
mkdir -p /opt/logs/elasticsearch
chown elasticsearch.elasticsearch /opt/elasticsearch -R
chown elasticsearch.elasticsearch /opt/logs/elasticsearch -R

# 限制一个进程可以拥有的VMA(虚拟内存区域)的数量要超过262144，不然elasticsearch会报max virtual memory areas vm.max_map_count [65535] is too low, increase to at least [262144]
echo "vm.max_map_count = 655350" >> /etc/sysctl.conf
sysctl -p

elasticsearch-ingest配置

▷ 192.168.1.41 /opt/elasticsearch/config/elasticsearch.yml

cluster.name: my-application
node.name: 192.168.1.41
path.logs: /opt/logs/elasticsearch
network.host: 192.168.1.41

discovery.seed_hosts: ["192.168.1.31","192.168.1.32","192.168.1.33"]
cluster.initial_master_nodes: ["192.168.1.31","192.168.1.32","192.168.1.33"]
http.cors.enabled: true
http.cors.allow-origin: "*"

# 关闭master功能
node.master: false
# 开启ingest功能
node.ingest: true
# 关闭data功能
node.data: false

▷ 192.168.1.42 /opt/elasticsearch/config/elasticsearch.yml

cluster.name: my-application
node.name: 192.168.1.42
path.logs: /opt/logs/elasticsearch
network.host: 192.168.1.42

discovery.seed_hosts: ["192.168.1.31","192.168.1.32","192.168.1.33"]
cluster.initial_master_nodes: ["192.168.1.31","192.168.1.32","192.168.1.33"]
http.cors.enabled: true
http.cors.allow-origin: "*"

# 关闭master功能
node.master: false
# 开启ingest功能
node.ingest: true
# 关闭data功能
node.data: false

▷ 192.168.1.43 /opt/elasticsearch/config/elasticsearch.yml

cluster.name: my-application
node.name: 192.168.1.43
path.logs: /opt/logs/elasticsearch
network.host: 192.168.1.43

discovery.seed_hosts: ["192.168.1.31","192.168.1.32","192.168.1.33"]
cluster.initial_master_nodes: ["192.168.1.31","192.168.1.32","192.168.1.33"]
http.cors.enabled: true
http.cors.allow-origin: "*"

# 关闭master功能
node.master: false
# 开启ingest功能
node.ingest: true
# 关闭data功能
node.data: false

elasticsearch-ingest启动

sudo -u elasticsearch /opt/elasticsearch/bin/elasticsearch

elasticsearch集群状态

curl "http://192.168.1.31:9200/_cat/health?v"

elasticsearch-ingest状态

curl "http://192.168.1.31:9200/_cat/nodes?v"

elasticsearch-ingest参数说明

status: green  # 集群健康状态
node.total: 9  # 有9台机子组成集群
node.data: 6  # 有6个节点的存储
node.role: d  # 只拥有data角色
node.role: i  # 只拥有ingest角色
node.role: m  # 只拥有master角色
node.role: mid  # 拥master、ingest、data角色

---

elasticsearch-master部署

首先，将上一篇《EFK教程 - 快速入门指南》中部署的3台es（192.168.1.31、192.168.1.32、192.168.1.33）改成只有master的功能， 因此需要先将这3台上的索引数据迁移到本次所做的data节点中

1️⃣ Index Migration : This step must be done, before the index will be put on the data node

curl -X PUT "192.168.1.31:9200/*/_settings?pretty" -H 'Content-Type: application/json' -d'
{
  "index.routing.allocation.include._ip": "192.168.1.51,192.168.1.52,192.168.1.53"
}'

2️⃣ confirm the current index storage location : Ensure that the index is not all nodes 192.168.1.31,192.168.1.32,192.168.1.33

curl "http://192.168.1.31:9200/_cat/shards?h=n"

elasticsearch-master configuration

Note: modify the configuration, restart the process, one needs an execution, to ensure the success of the first, then to the next stage. Method for restarting the process: As the previous article "EFK Tutorial - Quick Start Guide", the execute command is run in the foreground, thus directly ctrl - c to exit the restart, start a command as follows

sudo -u elasticsearch /opt/elasticsearch/bin/elasticsearch

▷ 192.168.1.31 /opt/elasticsearch/config/elasticsearch.yml

cluster.name: my-application
node.name: 192.168.1.31
path.logs: /opt/logs/elasticsearch
network.host: 192.168.1.31

discovery.seed_hosts: ["192.168.1.31","192.168.1.32","192.168.1.33"]
cluster.initial_master_nodes: ["192.168.1.31","192.168.1.32","192.168.1.33"]
http.cors.enabled: true
http.cors.allow-origin: "*"

#开启master功能
node.master: true
#关闭ingest功能
node.ingest: false
#关闭data功能
node.data: false

▷ 192.168.1.32 /opt/elasticsearch/config/elasticsearch.yml

cluster.name: my-application
node.name: 192.168.1.32
path.logs: /opt/logs/elasticsearch
network.host: 192.168.1.32

discovery.seed_hosts: ["192.168.1.31","192.168.1.32","192.168.1.33"]
cluster.initial_master_nodes: ["192.168.1.31","192.168.1.32","192.168.1.33"]
http.cors.enabled: true
http.cors.allow-origin: "*"

#开启master功能
node.master: true
#关闭ingest功能
node.ingest: false
#关闭data功能
node.data: false

▷ 192.168.1.33 /opt/elasticsearch/config/elasticsearch.yml

cluster.name: my-application
node.name: 192.168.1.33
path.logs: /opt/logs/elasticsearch
network.host: 192.168.1.33

discovery.seed_hosts: ["192.168.1.31","192.168.1.32","192.168.1.33"]
cluster.initial_master_nodes: ["192.168.1.31","192.168.1.32","192.168.1.33"]
http.cors.enabled: true
http.cors.allow-origin: "*"

#开启master功能
node.master: true
#关闭ingest功能
node.ingest: false
#关闭data功能
node.data: false

elasticsearch cluster status

curl "http://192.168.1.31:9200/_cat/health?v"

elasticsearch-master status

curl "http://192.168.1.31:9200/_cat/nodes?v"

At this point, when node.role where all servers are no longer appears "mid", it means that everything goes smoothly completed.