In Redis, if the slave library fails, the client can continue to send messages to the master library and other slave libraries for related operations. However, if the main library fails, it will directly affect the synchronization operation of the slave library (the slave library does not have a corresponding main library to perform related data replication operations) and there is no instance to support the client to perform write operations.

Switching the slave library to the main library needs to involve three issues:

Is the main library really down?
Which slave library should be selected to switch to the master library?
How to notify the slave library and the client of the information of the new master library?

1. The basic flow of the sentinel mechanism

The sentinel mechanism is running when the master-slave library instance is running. The main behavior of the sentinel mechanism is monitoring.

1.1 Sentinel monitoring:

During the running process, the sentinel process will periodically send PING commands to all master-slave libraries to check whether they are still running online. If the slave library does not respond to the sentinel's PING command within the specified time, the sentinel will mark it as "offline status". Similarly, if the main library does not respond to the sentinel's PING command within the specified time, the sentinel will also mark the main library as "offline status", and then start the process of automatically switching the main library .

1.2 Automatically switch the main library process

This process first performs the second task of the sentinel: the election of the master. After the main library hangs up, Sentinel needs to select a slave library instance from many slave libraries according to certain rules, and use it as the updated main library. After this step is completed, there will be a new master library in the cluster.
Then comes the last task: notifications . When executing the task notification, the sentinel will send the information of the new main library to all other slave libraries, let them execute the replicaof command, establish a connection with the new main library, and perform data replication. At the same time, Sentinel will send the connection information of the new main library to the client, allowing him to send new requests to the new main library.

Among these three tasks, the notification task is relatively simple. It only needs to send the information of the new main library to the slave library and the client, and let them connect with the new main library, and no decision logic is involved. But in the two tasks of monitoring and selecting the leader, the sentinel needs to make two decisions:

In the monitoring task, Sentinel needs to decide whether the main library is offline
In the master selection task, Sentinel also decides which slave library instance to choose as the new master library.

How to judge whether the main database is offline?

2. Judging whether the main database is offline

Whether the main database is offline is judged into two types: "subjective offline" and "objective offline".

2.1. Subjective offline

Subjective offline: The sentinel mechanism will send a PING command to the master-slave library to detect the network connection between itself and the master-slave library to judge the status of the instance. If the sentinel finds that the response time of the master-slave library has timed out, it will mark the master-slave library as "subjective offline".

Misjudgment

But there will be a situation where the sentinel misjudged that the master-slave library did not fail, and the master library did not go offline. The sentinel misjudged that he was offline. Misjudgment generally occurs when the pressure on the cluster network is high, the network is congested, or the main library itself is under high pressure .

Once the sentinel judges that the master database is offline, it needs to perform a series of operations such as master election and master-slave database synchronization, which will increase additional computing and communication overhead.

So it is necessary to reduce misjudgment.

How to reduce misjudgment?

Introduce a few more sentinels for negotiation and judgment, that is, sentinel clusters.

Sentinel Cluster

Sentinel Cluster: Usually deployed in cluster mode consisting of multiple instances. Introduce multiple sentinel instances for judgment, avoiding the situation where a single sentinel misjudges that the main library is offline due to poor network. The probability of multiple sentinel networks being unstable at the same time is small, and the probability of misjudgment is also small.

2.2 Objective offline

When judging whether the main library is offline, one sentinel cannot have the final say. Most sentries must judge that the main library has been "subjectively offline", and the main library will be marked as "objectively offline". This statement shows that the main library Going offline has become an objective fact, and the judgment principle is: the minority obeys the majority.

Objective offline: When there are N clusters, when there are N/2+1 clusters that have judged the main database as "subjectively offline", the main database can be finally judged as "objectively offline". Reduce the master-slave library switching overhead caused by misjudgment. (There are several examples to make the judgment of "subjective offline" of the main library, which is set by the Redis administrator with confidence).

3. Master-slave switching mechanism: master-slave switching will be performed when the main library is offline

How to choose a new main library?

Generally speaking, the process of Sentinel selecting a new main library can be called "screening + scoring". It is to remove unqualified slave libraries according to certain screening conditions in multiple instance slave libraries . Then, according to certain rules , score the remaining slave libraries one by one, and select the slave library with the highest score as the new master library.

3.1 Screening conditions

It is not only necessary to judge the current status of the slave library, but also its previous network connection status . If the number of disconnections between the slave library and the master library exceeds the threshold, there is a reason to detail that the network connection status of the slave library is not very good, and it can be filtered out. Although it is running now, if it is cut off after a while, the owner needs to be re-elected, so it is necessary to judge her previous state.

How to judge?

Use the configuration item down-after-milliseconds*10. Among them, down-after-milliseconds is the maximum time we consider to be disconnected from the database. If within down-after-milliseconds milliseconds, the master and slave nodes are not connected to the network, then the slave database is considered disconnected. If the disconnection time exceeds 10 times, it is considered that the network transition of the slave library is not very good, and it is not suitable for the master library.

3.2 Scoring rules

It can be scored according to three rules: slave library priority, slave library copy progress, and slave library ID number

It only needs to have the highest score in a certain round, then he is the new master, and the process of master selection is over. If there is no highest score, then the next round will take place.

The first round: the slave library with the highest priority slave-priority has a higher score

Users can set different priorities for different slave libraries through the slave-priority configuration item. For example, you have two slave libraries with large memory

The difference is that you can manually set a high priority for instances with large memory. When selecting the master, Sentinel will give high marks to the slave library with high priority. If there is a slave library with the highest priority, it will be the new master library. If the priorities of the slave libraries are the same, the sentry starts the second round of scoring.

The second round: the slave library with the closest synchronization degree to the old master library has a high score

If you choose the slave library that is closest to the old master library as the master library, the new master library will have the latest data.

How to judge the synchronization progress of the slave library and the master library? (copy progress)

When the master-slave library is synchronized, there is a process of command propagation. In this process, the master library will use master-repl-offset to record, and the current latest write operation is in the middle position of repl-backlog-buffer, while the slave library will use slave -repl-offset records the progress of the replication.

So we need to find the closest slave library of master-repl-offset and slave-repl-offset. If the score is high, it will be selected as the new main library. If the slave-repl-offset is the same, the next round of scoring will be performed.

As shown in the figure below, the master_repl_offset of the old master library is 1000, and the slave_repl_offset of slave libraries 1, 2, and 3 are 950, 990, and 900, respectively. Then, slave library 2 should be selected as the new master library.

The third round: the one with the smaller ID number gets the higher score from the pool

Each instance will have an id, which is similar to the number of the slave library. When Redis selects the master, there is a rule: in the case of the same priority and replication progress, the smaller the ID, the higher the score.

3.3 Summary of master-slave switching

First of all, the sentinel mechanism will filter out some slave libraries that do not meet the requirements according to the online status and network status. Then score the slave library according to priority, replication progress, and ID size, and the one with the highest score is selected as the new master library.

4. Summary of sentinel mechanism

Master-slave cluster data synchronization ensures data reliability. When the main library fails, automatic master-slave switching is the key support for uninterrupted service.

The sentinel mechanism of Redis automatically completes the following three functions, thereby realizing the automatic switching of the master-slave library, which can reduce the operation and maintenance overhead of the Redis cluster:

Monitor the running status of the main library, and judge whether the main library is objectively offline;
After the main library objectively goes offline, select a new main library;
After the new master library is selected, the slave library and the client are notified.

In order to reduce the misjudgment rate, in practical applications, the sentinel mechanism is usually deployed in a multi-instance manner. Multiple sentinel instances use the principle of "the minority obeys the majority" to judge whether the main database is objectively offline. Generally speaking, we can deploy three sentries. If two sentries determine that the main library is "subjectively offline", the switching process can be started. Of course, if you want to further improve the accuracy of judgment, you can also increase the number of sentries appropriately, for example, use five sentries.

What should I do if an instance in the sentinel cluster is down? Will it affect the judgment of the status of the main database and the election of the master?

Simply put the conclusion: when there are faulty nodes, as long as most of the nodes in the cluster are in normal state, the cluster can still provide services to the outside world

Most instances of the sentinel cluster reach a consensus, and after judging that the main library is "objectively offline", which instance will perform the master-slave switch?

After the sentinel cluster judges that the main library is "subjectively offline", it will elect a "sentinel leader", and then it will complete the master-slave switch in the whole process.

During the master-slave switching process of Sentinel, can the client normally perform the request operation?

If the client uses read-write separation, the read request can be executed normally on the slave library without being affected. However, since the main library has been hung up at this time, and the sentinel has not yet selected a new main library, the write request will fail during this period, and the duration of the failure = the time when the sentinel switches the master-slave + the client perceives the new main library time.

If you don’t want the business to be aware of the exception, the client can only cache the write-failed requests or write them into the message queue middleware, and send these write requests to the new master library after the sentry switches the master-slave. This scenario is only suitable for businesses that are not sensitive to the return value of the write request, and also needs to be adapted by the business layer. In addition, if the master-slave switch takes too long, it will also cause too many write requests in the cache of the client or message queue middleware. It takes longer to replay these requests after completion.

The sentinel detects how long the main library does not respond before promoting the slave library to the new main library. This time is configurable (down-after-milliseconds parameter). The shorter the configuration time, the more sensitive the sentinel is. The sentinel cluster will initiate a master-slave switchover if the main library cannot be connected within a short period of time. This configuration is likely to cause unnecessary switching due to network congestion but the main library is normal. Of course, When the main library really fails, because of the timely switchover, the impact on the business is minimal. If the configuration time is longer, the more conservative the sentinel is, which can reduce the probability of misjudgment by the sentinel. However, when the main library fails, the time for business write failure will be longer, and the amount of cached write request data will increase.

2. The sentinel hangs up

If a sentinel instance fails during runtime, can the master-slave library still switch normally?

In fact, once multiple instances form a sentinel cluster, even if a sentinel instance fails and hangs up, other sentinels can continue to cooperate to complete the work of switching the main library, including determining whether the main library is offline and selecting a new main library , and notifications from libraries and clients.

If you have deployed a sentinel cluster, you will know that when configuring sentinel information, we only need to use the following configuration item to set the main IP and port, and do not configure the connection information of other sentinels.

sentinel monitor <master-name> <ip> <redis-port> <quorum>

Since the sentinels don't know each other's addresses, how do they form a sentinel cluster?

1. Composition of sentinel cluster based on pub/sub mechanism

The reason why sentinels can be discovered by each other: the pub/sub mechanism provided by Redis (publish/subscribe mechanism)

1.1 Sentinels are discovered by each other:

As long as the sentinel has established a connection with the main library, it can publish information on the main library and publish its own connection information (ip and port number). At the same time, you can also subscribe to information from the main library to obtain connection information published by other Sentinels. When multiple sentinels publish and subscribe on the main library, they know each other's ip address and port number.

In addition to sentinel instances, applications written by ourselves can also publish and subscribe to messages through Redis.

How does Redis differentiate between different applications?

Redis way through channels. Classify and manage messages to distinguish different application messages. A channel is actually a message type. When the message types are the same, they belong to a channel. Only applications subscribed to the same channel can exchange information through published messages.

In the master-slave cluster, the master library has a "__sentinel__:hello" channel, and different sentinels can discover and communicate with each other by implementing it.

1.2 Instructions for Sentinel message exchange:

For example: In the figure below, Sentinel 1 publishes its own IP (17216.19.3) and port 26579) to the "_sentinel_:hello" channel, and Sentinel 2 and 3 subscribe to this channel. Then at this time, Sentinel 2 and 3 can directly obtain the IP address and port number of Sentinel 1 from this channel.

Then, Sentinels 2 and 3 can establish a network connection with Sentinel 1. In this way, Sentinels 2 and 3 can also establish a network connection, so that a Sentinel cluster is formed. They can communicate with each other through network connections, such as judging and negotiating whether the main library is offline

In addition to establishing connections with each other to form a cluster, Sentinels also need to establish connections with slave libraries. Because in the sentinel monitoring task, the sentinel needs to make a heartbeat judgment on the master-slave library, and after the master-slave library switch is completed, it also needs to notify the slave library to synchronize them with the new master library.

How does Sentinel connect with the slave library resume? How to know the ip address and port of the slave library?

1.3 Sentinel establishes a connection with the slave library: INFO command

When the sentinel sends an INFO command to the main library, the main library will return the slave library list to the sentinel after receiving the command. After Sentinel receives the connection information of the slave library list, it will establish a connection with each slave library, and continuously monitor the slave library on this connection.

Through the pub/sub mechanism, sentry clusters are established between sentinels, and the connection information of the slave library is obtained by sending the INFO command, and the sentinel establishes a connection with the slave library for monitoring. After the master-slave library is switched, the client needs to know the connection information of the new master library before sending information to the new master library. Therefore, the sentinel also needs to complete the task of telling the client about the new main library information.

When actually using Sentinel, we sometimes encounter such a problem: How to monitor the process of master-slave switching of Sentinel on the client side? For example, which step has the master-slave switching progressed? This is actually a requirement, the client side It is possible to obtain various events that occur during the process of monitoring, master selection, and switching of the sentinel cluster.

1.4 Information synchronization between sentinel and client

1.5 Notification of client events based on pub/sub mechanism

In essence, Sentinel is a Redis instance running in a specific mode, but it does not complete the service request operation, but only completes the tasks of monitoring, master election and notification. So, each sentinel instance also provides pub/sub mechanism, clients can subscribe messages from sentinel . There are many message subscription channels provided by Sentinel, and different channels contain different key events during the master-slave library switching process

Common events:

event	related channel
Main library offline event	+sdown (the instance enters the "subjective offline" state)
	-sdown (the instance exits the "subjective offline" state)
	+odown (the instance enters the "objectively offline" state)
	-odown (the instance exits the "objectively offline" state)
Slave reconfiguration event	+slave-reconf-sent (sentry sends SLACEOF command to reconfigure slave library)
	+slave-reconf-inprog (configure the new main library from the library, but not synchronized yet)
	+slave-reconf-done (configure the new main library from the library, and complete the synchronization with the new main library)
New main library switch	+switch-master (main library address change)

Knowing these channels, clients can subscribe to messages from Sentry. The specific operation steps are that after the client reads the configuration file of Sentinel, it can obtain the address and port of Sentinel, and establish a network connection with Sentinel. Then, you can execute the subscription command on the client to obtain different event messages.

For example, you can execute the following command to subscribe to the "event that all instances enter the objective offline state":

subscribe +odown

Subscribe to all channels

PSUBSCRIBE  *

When the sentinel selects the new master library, the client will see the following switch-master event. This event indicates that the main library has been switched, and the IP address and port information of the new main library is already available. At this time, the client can use the address and port of the new main library to communicate.

switch-master <master name> <oldip><oldport> <newport>

Through event notification, the client can not only get the connection information of the new master library after the master-slave switch, but also monitor and get various important events that occur during the master-slave switch. In this way, the client can know which step the master-slave switches to, which helps to understand the switching speed.

Summary: With the pub/sub mechanism, Sentry can establish connections with slave libraries, between Sentinels and Sentinels, and between Sentinels and clients. Judging that the main library is offline, the three tasks of monitoring, master selection and notification based on the sentinel cluster can basically work normally.

After the master-slave failure, there are multiple instances in the cluster, how to determine which sentinel will perform the actual master-slave switch?

2. Which sentinel performs master-slave switching?

In fact, the process of master-slave switching by which sentinel is actually a "voting arbitration" process just like master election.

2.1 Starting from the main process of Sentinel election:

Any instance will send the is-master-down-by-addr command to other instances as long as it judges that the master library is "subjectively offline". Then, other instances will respond with Y or N according to their connection with the main library, Y is equivalent to a vote in favor, and N is equivalent to a negative vote.

Once a sentinel has obtained the number of yes votes required for arbitration, it can mark the main library as "objectively offline". The required number of upvotes is set via the quorum configuration item in the sentinel configuration file. For example, now there are 5 sentinels, and the quorum configuration is 3. Then, a sentinel needs 3 yes votes, and the main library can be marked as "objectively offline". The 3 yes votes include the sentinel's own yes vote and the other two sentinel's yes votes.

2.2 The sentinel elects the Leader process: Leader Election

At this point, the sentinel can send commands to other sentries, indicating that it wants to perform the master-slave switch by itself, and let all other sentries vote. This voting process is called "Leader Election". Because the sentinel that finally performs the master-slave switch is called the Leader, the voting process is to determine the Leader.

During the voting process, any Sentinel who wants to become a Leader must meet two conditions: first, get more than half of the votes in favor; second, the number of votes he gets must be greater than or equal to the quorum value in the sentinel configuration file. Take 3 sentinels as an example, assuming that the quorum is set to 2 at this time, then any sentinel who wants to become a leader only needs to get 2 yes votes.

Specifically explained through the following figure:

At T1, S1 judges that the main database is "objectively offline". If it wants to become a leader, it first votes for itself, and then sends commands to S2 and S3 respectively, indicating that it wants to become a leader.

At T2, S3 judges that the main database is "objectively offline", and it also wants to become a leader, so it votes for itself first, and then sends commands to S1 and S2 respectively, indicating that it wants to become a leader.

At time T3, S1 receives the Leader voting request from S3. Because S1 has voted Y for itself, it can no longer vote for other sentries, so S1 replies N to express its disagreement. At the same time, S2 receives the Leader voting request sent by S3 at T2. Because S2 has not voted before, it will reply Y to the first sentinel who sent a voting request, and reply N to the sentinel who sent a voting request later. Therefore, at T3, S2 replies to S3 and agrees to S3 becoming the leader.

At T4, S2 receives the voting command sent by S1 at T1. Because S2 has agreed to S3's voting request at T3, at this time, S2 replies N to S1, expressing its disapproval of S1 becoming the leader. This happens because the network traffic between S3 and S2 is normal, but the network traffic between S1 and S2 may just be congested, causing the voting request to be transmitted slowly.

Finally, at time T5, S1 gets one vote Y from itself and one vote N from S2. In addition to its own vote Y, S3 also received a vote Y from S2. At this time, S3 not only obtained more than half of the Leader's votes, but also reached the preset quorum value (quorum is 2), so it finally became the Leader. Then, S3 will start to perform the master selection operation, and after the new master library is selected, it will notify other slave libraries and clients of the new master library information.

If S3 does not get 2 Y votes, then this round of voting will not produce a Leader. The Sentinel cluster will wait for a period of time (that is, twice the Sentinel failover timeout) before re-election . This is because the successful voting of the sentinel cluster largely depends on the normal network propagation of election commands. If the network pressure is high or there is short-term congestion, it may cause no sentinel to get more than half of the votes in favor. Therefore, wait until the network congestion improves before voting, and the probability of success will increase.

It should be noted that if there are only 2 instances in the sentinel cluster, at this time, if a sentinel wants to become a leader, it must obtain 2 votes instead of 1 vote. Therefore, if a sentinel hangs up, then the cluster at this time cannot perform master-slave library switching. Therefore, usually we configure at least 3 Sentinel instances. This is very important, and you can't ignore it in practical applications.

Why Sentinels don't vote for themselves at the same time?

In order for S1, S2, and S3 to vote with themselves at the same time, it is necessary for these three sentinels to determine that the main library is objectively offline at the same time. However, the network connections and system pressures of different Sentinels are not exactly the same, and the time of receiving the offline negotiation message may also be different. Therefore, the probability of them making an objective offline judgment of the main database at the same time is relatively small, and generally there is a sequence relationship. The example in the article is that S1 and S3 are judged first, and S2 has not been judged.

Operations such as the online status check of the master-slave library by the sentinel are a kind of time event, which is completed by a timer. Generally speaking, these events are executed every 100ms. A small random time offset will be added to the timer execution cycle of each sentinel. The purpose is to make the time for each sentinel to perform the above operations slightly staggered, and also to prevent them from simultaneously determining that the main library is offline and electing at the same time. Leader.

Redis has 1 master and 4 slaves, 5 sentinels, and the quorum of the sentinels is 2. If 3 sentries fail, when the main database is down, can the sentries judge that the main database is "objectively offline"? Can it switch automatically?

1. The sentinel cluster can determine that the main database is "subjectively offline" . Since quorum=2, when a sentinel judges that the main database is "subjectively offline", it will get the same result after asking another sentry. The sentinel cluster can determine that the main library is "objectively offline".

2. However, Sentinel cannot complete the master-slave switch . After the sentinel marks the main database "objectively offline", when electing the "sentry leader", a sentinel must get more than a majority of votes (5/2+1=3 votes). But currently there are only 2 sentries alive. No matter how you vote, a sentry can only get 2 votes at most, and it will never reach the result of a majority of votes.

Are more sentinel instances better?

No, we have also seen that the sentinel needs to communicate with other nodes and exchange information when it judges "subjective offline" and elects the "sentry leader" . When deploying multiple Sentinels, they will be distributed on different machines. The more nodes there are, the greater the risk of machine failure. These problems will affect the communication and election of Sentinels. When there is a problem, it means that the election time will be longer. , the time to switch master-slave becomes longer.
The more sentinel instances, the lower the misjudgment rate will be. However, when the main library is determined to be offline and the leader is elected, the instance needs to get more votes, and the waiting time for all sentinels to vote may also increase accordingly. The time to switch from the library will also become longer, and the client will easily accumulate more request operations, which may cause client request overflow, resulting in request loss. If the business layer has response time requirements for Redis operations, a timeout alarm may occur because the new master library has not been selected and the new operation cannot be executed.
After the down-after-milliseconds is increased, it may lead to such a situation: the main library has actually failed, but it took a long time for the sentry to judge, which will affect the availability of Redis for business.

Is it beneficial to reduce false positives by increasing the down-after-milliseconds value?

It is beneficial. Properly increase the down-after-milliseconds value. When there are short-term fluctuations in the network between the sentinel and the main library, the probability of misjudgment can be reduced . However, increasing the down-after-milliseconds value also means that the master-slave switchover time will be longer, and the longer the impact on the business, we need to weigh it according to the actual scenario and set a reasonable threshold.

3. Summary: The key mechanism of the sentinel (the sentinel contacts the master-slave guest sentinel, and the leader chooses the master)

When we solve a system problem, we will introduce a new mechanism or design a layer of new functions. The main content of the sentinel: In order to realize the master-slave switch, we introduce the sentinel; in order to avoid the failure of the master-slave switch after a single sentinel fails, and In order to reduce the misjudgment rate, a sentinel cluster is introduced; the sentinel cluster needs some mechanisms to support its normal operation.

3.1 These key mechanisms of the sentinel cluster:

Sentinel cluster formation process based on pub/sub mechanism;
Slave list based on INFO command, which can help Sentinel and slave library to establish connection;
Based on Sentinel's own pub/sub functionality, this enables event notification between clients and Sentinel.

For master-slave switching, of course, not any sentinel can execute it if it wants to, otherwise it will be messed up. Therefore, this requires the sentinel cluster to elect a Leader after judging that the main library is "objectively offline" through voting arbitration, and it is responsible for the actual master-slave switchover, that is, it completes the selection of the new master library and notifies the slave Libraries and clients.

Finally, I would like to share another experience with you: ensure that the configurations of all sentinel instances are consistent, especially the judgment value of down-after-milliseconds for subjective offline . This value is inconsistently configured on different Sentinel instances. As a result, the Sentinel cluster has not reached a consensus on the faulty main library, and has not switched the main library in time. The final result is that the cluster service is unstable. Therefore, you must not ignore this seemingly simple experience.

3. Summary of all Sentinels

1. How does Sentinel connect with the main library

Sentinel is directly associated with the main library, manually set, you can set multiple sentinels

2. How does the sentinel send messages to the slave library?

The sentinel sends the info command to the master library, the master library returns the slave collection of the slave library, establishes a connection with the slave library, and sends the information of the new master library to the slave library

3. How does the sentinel contact the client?

The client subscribes to a channel of Sentinel, who is the master in the channel, reads the configuration file of the sentinel, obtains the ip address and port number, establishes a connection with the sentinel, subscribes to the information after connecting, obtains the information of the master library, and communicates with the master The library establishes a connection

Detailed Interpretation of Redis Sentinel Mechanism

1. Basic interpretation of sentinel mechanism