In-depth explanation of multi-node role configuration in Elasticsearch 5.X cluster

1. Problem elicited

The ES5.X node type has more ingest node types. 
For clusters of 3 nodes, 5 nodes or more nodes, how to configure node roles to optimize system performance? 
write picture description here

2. Overview of node roles in ES2.X and earlier versions

write picture description here

3. ES5.X node role list

write picture description here
Since several other types of nodes and their uses are well understood, there are no more than master nodes, data nodes, and routing nodes.

Purpose of Ingest: 
1) An Ingest node is the same as other nodes in the cluster, but it can create multiple processor pipelines to modify incoming documents. Similar to the most commonly used Logstash filters have been implemented as processors.

2) Ingest nodes can be used to perform common data transformations and enrichments. The processor is configured to form a pipeline. At the time of writing, Ingest Node has 20 built-in processors such as grok, date, gsub, lowercase/uppercase, delete and rename.

3) Before the batch request or index operation, the Ingest node intercepts the request and processes the document. 
An example of such a processor could be a date processor, which is used to parse dates in fields. 
Another example is a conversion processor, which converts a field value to the target type, such as converting a string to an integer.

4. There are many types of ES5.X node combinations, how to choose?

Elasticsearch employee Christian_Dahlqvist explained it as follows:

The default configuration of a node is: master node + data node two attributes. For small clusters of 3-5 nodes, it is common to have all nodes store data and qualify as master nodes. You can send any request to any node, and since all nodes have a copy of the cluster state, they know how to route the request.

Usually only larger clusters can begin to separate dedicated master, data nodes. For many user scenarios, routing nodes are not necessarily required at all.

Dedicated coordinator nodes (also known as client nodes or routing nodes) remove the request parsing and final stages of aggregation/querying from data nodes and allow them to focus on processing the data. 
The extent to which this is beneficial to the cluster will vary from case to case. Usually I would say that routing nodes is more common in query heavy usage scenarios.

5. How to set node roles in ES5.X cluster

For the configuration of 3 nodes, 5 nodes or even more node roles, there is no clear definition on the Elasticsearch official website, domestic and foreign forums, and blogs. 
write picture description here 
My thinking is as follows: 
1) For the Ingest node, if we do not need format conversion, type conversion, etc., directly set it to false. 
2) 3-5 nodes belong to a lightweight cluster, and it is necessary to ensure that the number of master nodes satisfies ((number of nodes/2)+1). 
3) In a lightweight cluster, the multiple attributes of a node, such as Master&Data, can be understood by the same node. 
4) If further optimized, 5 nodes can separate the Master and Data again and cancel the client node.

6. Summary

1) Elasticsearch is extensive and profound, especially with many new 5.X features, which requires further in-depth research; 
2) The configuration of the cluster also depends on further event summary, and the best theoretical deployment practice is a "vase"; 
3) It looks like a graphic division I have written so much and written so much. Based on the official website, I have also researched many documents, but I still don’t understand the function of the Ingest node deeply enough. I hope you will comment on it.

Reference: 
[1] https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#data-node 
(5.x official website) 
[2]  https://www.elastic .co/guide/en/elasticsearch/reference/2.4/modules-node.html 
(2.x official website) 
[3]  https://discuss.elastic.co/t/master-and-client-node-role-clarifications /104036/2 
(Reply from ES staff) 
[4]  https://blog.trifork.com/2017/02/02/elasticsearch-ingest-node/ 
(Detailed explanation of ingest node usage) 
[5]  https://wenchao.ren /archives/375 
(translation from the official website) 
[6]  http://www.cnblogs.com/liang1101/p/7284205.html 

(Domestic buddies have thought about it, but I don't think it's completely right)

Reprinted: http://blog.csdn.net/laoyang360/article/details/78290484

Guess you like

Origin http://43.154.161.224:23101/article/api/json?id=325636998&siteId=291194637