Reprinted from the original text: zookeeper (2) summary of common problems
1. Why does zookeeper deploy base servers?
The so-called zookeeper fault tolerance means that when several zookeeper servers are down, the remaining number must be greater than the number of downtimes, that is, the number of remaining services must be greater than n/2, so that zookeeper can continue to be used, regardless of parity Anyone can elect a leader. At most 2 of the 5 machines are down, and they can continue to be used because the remaining 3 machines are greater than 5/2. The reason why it is better to have an odd number is to save resources under the condition of the maximum number of fault-tolerant servers. For example, when the maximum fault-tolerance is 2, the corresponding number of zookeeper services is 5 for odd and 6 for even. In the case of 6 zookeeper services, at most 2 services can be down, so from the perspective of saving resources, there is no need to deploy 6 (even) zookeeper services.
Zookeeper has such a feature: as long as more than half of the machines in the cluster are working normally, the entire cluster is available to the outside world. That is to say, if there are 2 zookeepers, then as long as there is 1 dead zookeeper, it cannot be used, because 1 is not more than half, so the death tolerance of 2 zookeepers is 0; similarly, if there are 3 zookeepers, one dies , there are still 2 normal ones, more than half, so the tolerance of 3 zookeepers is 1; similarly, you can list a few more: 2->0;3->1;4->1;5->2; 6->2 will find a rule, the tolerance of 2n and 2n-1 is the same, both are n-1, so in order to be more efficient, why add that unnecessary zookeeper.
According to the above, it can be concluded that the perspective of resource saving occurs
Second, the zookeeper split-brain (Split-Brain) problem
2.1. What is split brain?
In the vernacular, for example, when there are two nodes in your cluster, they both know that a master needs to be elected in this cluster. Then when there is no problem with the communication between the two, a consensus will be reached and one of them will be selected as the master. But if there is a problem with the communication between them, both nodes will feel that there is no master now, so each elects itself as the master. So there will be two masters in the cluster.
There is a very important question for Zookeeper, that is, according to what kind of situation is it judged that a node is dead and down. In a distributed system, these are judged by the monitor, but it is difficult for the monitor to judge the status of other nodes. The only reliable way is the heartbeat. Zookeeper also uses the heartbeat to judge whether the client is still alive.
Using ZooKeeper to do master HA is basically the same way. Each node tries to register a temporary node that symbolizes the master; others that do not register successfully become slaves, and monitor the temporary nodes created by the master through the watch mechanism. The internal heartbeat mechanism is used to determine the status of the master. Once the master has an accident, Zookeeper can quickly learn and notify other slaves, and other slaves will respond accordingly. This completes a switch. This mode is also a relatively common mode, and most of them are basically implemented in this way, but there is a very serious problem. If you don't notice it, it will lead to a split-brain in the system in a short time, because the heartbeat timeout may be the master It hangs, but it may also be a problem with the network between the master and zookeeper, which may also cause it. This situation is suspended animation , the master is not dead, but there is a network problem with other machines in ZooKeeper, which causes other nodes to think that it has died and then performs a new leader election switch, so that one of the slaves becomes the master. However, the original master did not die. At this time, the client also got the news of the master switch, but there will still be some delays. Zookeeper needs to be notified one by one. At this time, the whole system is very chaotic. Maybe some clients have already notified the connection to the new one. The master goes up, and some clients are still connected to the old master. If there are two clients that need to update the same data of the master at the same time, and the two clients happen to be connected to the new and old masters at the moment, it will be very serious. question.
Summarize:
Suspended death: The master is considered dead due to heartbeat timeout (caused by network reasons), but in fact the master is still alive.
Split-brain: Since suspended animation will initiate a new master election, a new master will be elected, but the old master network is connected again, resulting in the emergence of two masters, some clients are connected to the old master and some clients are linked to new master.
2.2. What is the reason?
To solve the Split-Brain problem, there are generally three ways:
- Quorums (quorum), such as a 3-node cluster, Quorums = 2, that is to say, the cluster can tolerate 1 node failure, at this time, a leader can be elected, and the cluster is still available. For example, for a cluster of 4 nodes, its Quorums = 3, and the Quorums must exceed 3, which is equivalent to the tolerance of the cluster is still 1. If 2 nodes fail, the entire cluster is still invalid.
- Using Redundant communications, redundant communication methods, multiple communication methods are used in the cluster to prevent the failure of one communication method and cause the nodes in the cluster to fail to communicate.
- Fencing, the way of sharing resources. For example, if you can see the shared resources, it means that you are in the cluster. The leader who can obtain the lock of the shared resources is the leader. If you cannot see the shared resources, it is not in the cluster.
ZooKeeper adopts the method of Quorums, that is, only more than half of the nodes in the cluster can elect the leader. In this way, the uniqueness of the leader can be ensured, either a unique leader is elected, or the election fails. Quorums have 2 roles in ZooKeeper:
- The minimum number of nodes in the cluster is used to elect the leader to ensure that the cluster is available
- Notify the client that the data has been saved by the minimum number of nodes in the cluster before the data has been safely saved. Once these nodes have saved the data, the client will be notified that it has been safely saved and can continue with other tasks. The remaining nodes in the cluster will eventually save the data as well
Suppose a leader dies, and the rest of the followers elect a new leader. At this time, the old leader is resurrected and still thinks it is the leader. At this time, its write request to other followers will also be rejected. Because every time a new leader is generated, an epoch will be generated, and this epoch is incremented. If followers confirm the existence of the new leader and know its epoch, they will reject all requests whose epoch is less than the current leader's epoch. Is there any follower who doesn't know that the new leader exists? It is possible, but certainly not the majority, otherwise the new leader cannot be generated. Zookeeper's writing also follows the quorum mechanism. Therefore, writing that is not supported by the majority is invalid. Even if the old leader thinks that it is the leader, it still has no effect.