ES load balancing and fault tolerance

1. ElasticSearch distributed foundation

1.1 ES distributed mechanism

• Distributed mechanism: Elasticsearch is a distributed system, and distribution is to cope with large amounts of data. Its characteristic is to hide the complex distributed mechanism.
• Sharding mechanism: which shard the data is stored in, and the copy data is written to another shard.
• Cluster discovery mechanism: newly started es instances will automatically join the cluster.
• Shard load balancing: For large amounts of data writing and querying, es will distribute the data evenly. For example, assuming there are 3 nodes now, and a total of 25 shards are to be allocated to the 3 nodes, es will automatically distribute them evenly to maintain a balanced read and write load request for each node.
• Shard copies: increase the number of copies and redistribute shards.

1.2 Vertical and horizontal expansion

Vertical expansion: Use more powerful servers to replace old servers. However, the storage and computing capabilities of single machines are online. And costs are skyrocketing. For example, 10 1T servers cost 10,000, and a single 10T server may cost 200,000.
Horizontal expansion: Purchase more servers and add them to the cluster. For ES, horizontal expansion is generally used.

1.3 rebalance

When es instances are added or reduced, or new data is added or deleted, some servers will be overloaded or underloaded. The es cluster will redistribute the data to maintain a relatively balanced state.

1.4 master node

(1) Manage metadata of es cluster

• Create delete node
• Create delete index

(2) By default, ES will automatically select a machine as the master, because any machine may be selected as the master node, so the single point of failure can be ignored.

1.5 Node peering

• Nodes are peer-to-peer, each node can receive all requests
• Automatic request routing
• Response collection

2. Node load balancing

There are two parameters in the elasticSearch configuration file: node.master and node.data
combination one: node.master: false node.data: true
The node server only acts as a data node and is only used to store index data . The node server has a single function and is only used for data storage and data query, reducing its resource consumption rate.
Combination 2: node.master: true node.data: false
This node server only acts as a master node, but does not store any index data. The node server will use its own idle resources to coordinate various index creation requests or query requests , and distribute these requests to the relevant node servers reasonably.
Combination three: node.master: false node.data: false
The node server will not be selected as the master node, nor will it store any index data ( coordinating node) . This server is mainly used for query load balancing, processing routing requests, processing searches, distributing index operations, etc. When querying, it usually involves querying data from multiple node servers, distributing the request to multiple designated node servers, and summarizing the results returned by each node server, and finally returning them to the client.
An independent client node is very useful in a relatively large cluster. It coordinates the master node and data nodes. When the client node joins the cluster, it can get the status of the cluster, and the request can be routed directly according to the status of the cluster.
Combination four: node.master: true node.data: true
This combination means that this node is qualified to become a master node and also stores data. At this time, if a node is elected to become the real master node, it will also store data, which will put greater pressure on this node. .
ES defaults to such a configuration for each node. In scenarios where the number of nodes is small, it can be allocated in this way. However, when there are more nodes, it is necessary to allocate their respective roles and separately divide some nodes into master nodes.
At present, most offices have fewer nodes and are in this mode.

3. Read request load balancing

• The client sends a request to any node (coordinating node)
• The coordination node forwards the search request to one of the primary shards or replica shards corresponding to all shards, using a round-robin random polling algorithm to balance the load of the retrieval requests.
• Each shard returns its own search results (doc id) to the coordinating node, which performs data merging, sorting, paging and other operations to produce a doc id list.
• The coordinate node performs hash routing on the doc id and forwards the request to the corresponding node. At this time, the round-robin random polling algorithm is used to randomly select one from the primary shard and all replicas to allow the read request to be load balanced .
• The node that receives the request returns the document to the coordinate node, which returns the document to the client.

What is the round-robin random polling algorithm?
The name is fancy, but it actually means looping and traversing,
but it emphasizes one rotation and each one is even, so there is usually a modulo operation.

4. Data balancing

4.1 Trigger conditions

• Creation of new index
• Index deletion
• Add a new replica shard
• Data balance caused by node increase or decrease

4.2 Allocation mechanism

The allocation mechanism is relatively complex.
The transparent hidden attribute of the distributed
architecture_Elasti_csearch is a distributed architecture that hides the complex processing mechanism.
1. An index contains multiple shards.
2. Each shard is a minimum working unit and carries part of the data; Each shard is a Lucene instance, with complete indexing and request processing capabilities.
3. When adding or removing nodes, the shard will automatically load balance among the nodes.
4. For primary shard and relica shard, each document will only exist in a certain A primary shard and its corresponding replica shard cannot exist in multiple primary shards.
5. The replica shard is a copy of the primary shard and is responsible for fault tolerance and load balancing of read requests.
6. The primary shard is used when creating an index. Fixed, the number of relica shards can be changed at any time.
7. The primary shard and its corresponding replica shard cannot be placed on the same machine, otherwise fault tolerance will not be achieved.

5. es fault tolerance mechanism

Take 3 shards, 2 replicas, and 3 nodes as an example.
The master node is down, automatic master election, the cluster is red
replica fault-tolerant: the new master promotes the replica to the primary shard, yellow
restarts the down node, the master copies the replica to the node, uses the original shard and synchronizes the modifications after the downtime.
When green node1 is down, the P0 shard will disappear, and all primary shards are not fully active, so the status of the cluster is red.
The first step in fault tolerance is to re-elect the master node. Responsible for master related functions.
In the second step of fault tolerance, the new master promotes the R0 replica (replica shard) corresponding to the lost P0 shard (primary shard) to the primary shard. The current cluster status is yellow and there is one less replica shard, but now the cluster is ready Returned to use.
The third step of fault tolerance is to restart the faulty node. The new master will detect the addition of the new node and copy the missing replica shard to the new node. The copied shard will be the shard that was promoted to the main shard. The cluster is now fully restored and the status is green.
Question: When the cluster status is green (each shard has a copy), will one noe hang up, causing the entire cluster to become unavailable? The primary shard is not fully active, and the status of the cluster changes. For red, the answer is impossible, because the primary shard and replica shard of a shard cannot exist on the same node.

5. Horizontal expansion

Suppose there is a book index, shard3 replica1

• Fragments are automatically load balanced and fragments are transferred to idle machines.
• Each node stores fewer shards, and more system resources are given to each shard, improving overall cluster performance.
• Expansion limit: If the number of nodes is greater than the number of overall shards, there must be idle machines.
• When the expansion limit is exceeded, you can increase the number of replicas. For example, set the number of replicas to 2, for a total of 3*3=9 shards. 9 machines run at the same time, with stronger storage and search performance. Better fault tolerance.
• Fault tolerance: As long as all primary shards of an index exist, the cluster can run.