Zookeeper network split brain problem

I didn't know there was such a thing before I went to work. Let's talk about Zookeeper before we start talking about fake-dead split-brain.

Zookeeper
zookeeper is a coordination service for distributed applications. It is a software that provides consistent services for distributed applications. The performance provided includes: configuration maintenance, name service, distributed synchronization, group service, etc.
zookeeper is based on the Fast Paxos algorithm. The paxos algorithm has a livelock problem, that is, when there are multiple proposers staggered submissions, it may be mutually exclusive and no proposer can submit successfully. However, Fase Paxos has made some optimizations, which are generated through elections. For a leader, only the leader can submit proposals. For details, see the Fast Paxos algorithm.

The basic operation process of Zookeeper:

electing a leader;
synchronizing data;
there are many algorithms in the process of electing a leader, but the election criteria to be achieved are the same; the
leader must have a higher zxid;
most machines in the cluster get the response and follow the election the leader.
The novel is about Zookeeper. Next, let’s discuss problems such as split-brain, suspended animation, and solutions.

Suspended-dead split-brain
In a large cluster, there is often a master. In the long-term operation, problems such as downtime will inevitably occur, resulting in the unavailability of the master. After such a situation occurs, it will often cause great damage to the system. Therefore, the master in the general distributed cluster adopts a high-availability solution to avoid such a situation.
In the master-slaver method, there is a master node, which usually serves the outside world, and a slave node monitors the master, and there is some way to synchronize data while monitoring. If the master is down now, the slave can quickly know and switch to the new master quickly. But in this way, monitoring switching is a big problem, but now Zookeeper's watch and distributed lock mechanism can better solve this problem. Although Zookeeper solves this problem well, there are other problems with its use, such as split brain.
One of the root causes of split brain is suspended animation.

What is suspended animation?
There is a very important question, that is, according to what kind of situation is it judged that a node is dead and down.
In a distributed system, these are judged by the monitor, but it is difficult for the monitor to judge the status of other nodes. The only reliable way is the heartbeat, including Zookeeper, which also uses the heartbeat to judge whether the client is still alive.
Using ZooKeeper to do master HA is basically the same way. Each node tries to register a temporary node that symbolizes the master. Others that do not register successfully become slaves, and monitor the temporary nodes created by the master through the watch mechanism. The heartbeat mechanism is used to determine the status of the master. Once the master has an accident, Zookeeper can quickly learn and notify other slaves, and other slaves will respond accordingly. This completes a switch. This mode is also a relatively common mode, and most of them are basically implemented in this way, but there is a very serious problem. If you don't notice it, it will lead to a split-brain in the system in a short time, because the heartbeat timeout may be the master It hangs, but it may also be a problem with the network between the master and zookeeper, which may also cause it. This situation is suspended animation, the master is not dead, but there is a problem in the network with ZooKeeper, which causes Zookeeper to think that it has hung up and then notify other nodes to switch, so that one of the slaves becomes the master, but the original master does not Not dead. At this time, the client also gets the message of the master switch, but there will still be some delay. Zookeeper needs to be notified one by one for communication. At this time, the whole system is very chaotic. Maybe some clients have already notified the connection to the new master. Now, some clients are still connected to the old master. If there are two clients that need to update the same data of the master at the same time, and the two clients happen to be connected to the old and new masters at the moment, there will be serious problems.

What is the reason for this situation?
The main reason is that the Zookeeper cluster and the Zookeeper client judge that the timeout cannot be completely synchronized, that is to say, it may be one after the other. If the cluster is discovered before the client, the above situation will occur. At the same time, the notification of each client after discovery and switching also has a speed. Generally, the probability of this situation is very small. It requires the master to be disconnected from the Zookeeper cluster network but there is no problem with the network between other cluster roles. The above conditions must be met, but once it occurs, it will cause serious consequences and data inconsistency. .

How to avoid it?
When the slave switches, it does not switch immediately after checking that the old master has a problem, but sleeps for a sufficient period of time to ensure that the old master has been informed of the changes and has done the relevant shutdown cleanup work, and then registered as the master can avoid For this kind of problem, this sleep time is generally defined as the timeout time defined by Zookeeper, but the system is unavailable during this time.