Article Directory
Outline
ES China to continue learning with huperzine teacher, the first 66
Courses Address : https://www.roncoo.com/view/55
Meaningless shard cluster restart when re-allocation
In a cluster restart, when there is some configuration will affect the process shard recovery.
First, we need to understand the default configuration, shard recovery process what will happen.
If we have a 10 node, each node has a shard, may be the primary shard or replica shard, you have an index, there are five primary shard, each primary shard has a replica shard.
If we shut down the entire cluster some maintenance operations, such as installing a new machine to things like disk. When we restart the cluster node is definitely one after the start, there may be five nodes first started, and then the remaining five nodes not start.
Perhaps it is because the remaining five nodes did not have time to start, or is delayed for some reason, in short, for whatever reason, is now only five nodes are online. This five nodes will communicate with each other through a gossip protocol, the election of a master, and then make up a cluster. They will find that the data is not uniformly distributed, because there are five nodes does not start, then the shard on that node 5 is not available, the cluster on less than half of the shard. At this online 5 node sends a partial replica shard promoted to the primary shard, while adequate replica shard copy for each primary shard.
Finally, the remaining five nodes may be added to the cluster. However, these nodes could have been found they hold shard has been copied again and placed into 5 node before, and this time they will delete your local data. Then the cluster will begin operation shard of rebalance, the earliest start of the five shard evenly distributed on the node to the later start of the five node up.
In the process, these re-shard copy, move, delete, move, re-process, will consume a lot of disk and network resources. For the huge amount of data clusters, the clusters may lead to reboot every time, there is no reason to move TB-level data, could lead to cluster startup will take a long time. But if all nodes can wait for all nodes across the cluster are fully on line later, after all the data have, then decide whether you want to copy and move shard, the situation would be much better.
shard recovery 配置
So now we know the problem, then we can configure some settings to solve this problem.
First, we need to set a parameter gateway.recover_after_nodes: 8.
After this parameter allows es until there is enough node are on the line, and then begin the process of recovery of the shard. So this parameter is associated with a particular cluster, to be determined according to the number of our cluster nodes.
In addition, you should also set the number of node in a cluster must have at least the waiting time of those node: gateway.expected_nodes: 10，gateway.recover_after_time: 5m.
After the above configuration, the behavior will become cluster es follows, wait at least eight nodes online, then wait up to five minutes, or 10 nodes are online, start the process of recovery of the shard .
This can be avoided when a small number of node starts, it immediately begins shard recovery, consume large amounts of disk and network resources, even shard recovery process can be shortened from hours to minutes.
