Redis High Availability Sentinel Mechanism Implementation Details

This article comes from my technotes [1] Redis article, you are welcome to visit often.

text

In the previous article "Redis High Availability Panorama" , we learned about the high availability of Redis. High availability has two meanings: one is less service interruption, and the other is less data loss. The master-slave library mode and sentinels ensure less service interruption, and AOF logs and RDB snapshots ensure less data loss.

And we have learned the three responsibilities of the sentinel, namely: monitoring, master selection (choose the master library) and notification. Today we will learn in detail.

First of all, before Sentinel starts, we need to configure Sentinel. The Redis source code contains a file named sentinel.conf [2], some of which are configured as follows:

sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 60000

The first line of configuration instructs Sentinel to monitor a master server named mymaster. The IP address of this master server is 127.0.0.1 and the port number is 6379. At least 2 Sentinels need to agree to judge this master server as invalid.

As you can see, we only set the IP and port of the main library. The connection information of other sentinels is not configured. Since these sentinel instances do not know each other's addresses, how do they form a cluster?

Sentinel instances can discover each other thanks to the pub/sub mechanism provided by Redis, that is, the publish/subscribe mechanism. In the master-slave cluster, there is a __sentinel__:hellochannel named on the master library, through which different sentinels discover each other and communicate with each other.

1. The composition of the Sentinel cluster

Every 2 seconds, each Sentinel node will __sentinel__:hellosend the Sentinel node's judgment on the master node and the information of the current Sentinel node to the channel of the Redis data node.

Communication between Sentinels

for example. In the figure above, Sentinel 1 publishes its own IP (172.16.19.3) and port (26579) to __sentinel__:hellothe channel, and Sentinel 2 and 3 subscribe to the channel. Then at this time, Sentinel 2 and 3 can directly obtain the IP address and port number of Sentinel 1 from this channel.

Then, Sentinel 2, 3 can establish a network connection with Sentinel 1. In this way, Sentinel 2 and 3 can also establish a network connection, so that a Sentinel cluster is formed. They can communicate with each other through network connections, such as judging and negotiating whether the main library is offline or not.

In addition to establishing connections with each other to form a cluster, Sentinels also need to establish connections with slave libraries. This is because, in Sentinel's monitoring task, it needs to perform heartbeat judgment on both the master and slave libraries, and after the master-slave library switch is completed, it also needs to notify the slave libraries to synchronize them with the new master library.

So, how does Sentinel know the IP address and port of the slave library?

2. Obtain slave node information

This is done by the sentinel sending the INFO command to the main library. As shown in the figure below, Sentinel 2 sends an INFO command to the main library. After receiving the command, the main library will return the slave library list to the sentinel. Then, Sentinel can establish a connection with each slave library according to the connection information in the slave library list. Sentinel 1 and 3 can also establish a connection with the slave library through the same method.

Get slave node information

Below is a snippet of the info command executed on a master node:

# Replication
role:master
connected_slaves:2
slave0:ip=127.0.0.1,port=6380,state=online,offset=4917,lag=1
slave1:ip=127.0.0.1,port=6381,state=online,offset=4917,lag=1

After that, Sentinel will continuously monitor the slave library on this connection. Every 10 seconds, the sentinel node will send the info command to the master node and the slave node to obtain the latest topology of the cluster. In this way, when a new slave node joins, it can be sensed immediately. After the node is unreachable or a new master library is selected, the node topology information can also be updated in real time through the info command.

Execute the info command

Armed with information about the cluster, Sentinel can finally start its work. The first responsibility: to determine whether the master-slave library is offline.

3. How to judge that the master-slave database is offline?

3.1 Execute the ping command regularly

When the sentinel process is running, it will send a ping command to the master node, slave node, and other Sentinel nodes every 1 second to check whether they are still running online. If the master and slave libraries do not respond to the sentinel's ping command within the specified time, the sentinel will mark it as "offline status".

Execute the ping command

If the detection is the main library, then the sentinel cannot simply enable the master-slave switch. Because it is very likely that there is such a situation: that is, the sentinel misjudged, but in fact the main library is not faulty.

Misjudgments generally occur when the pressure on the cluster network is high, the network is congested, or the main library itself is under high pressure. Once the sentinel misjudges and initiates the master-slave switch, subsequent master selection and notification operations will bring additional computing and communication overhead.

So how to reduce misjudgment?

3.2 Subjective offline and objective offline

The sentinel mechanism is usually deployed in a cluster mode composed of multiple instances, which is also called a sentinel cluster. Introducing multiple sentinel instances to judge together can prevent a single sentinel from misjudging that the main library is offline because of its own poor network condition. At the same time, the probability that the network of multiple Sentinels is unstable at the same time is small, and they make decisions together, and the misjudgment rate can also be reduced.

Subjective offline and objective offline

Remember the sentinel.conf configuration I gave at the beginning of the article?

sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 60000

down-after-millisecondsThe option is the critical threshold at which Sentinel considers the server to be offline.

If the server does not return a reply to the ping command sent by Sentinel within the number of milliseconds, or returns an error, then Sentinel marks the server as subjectively down (SDOWN for short).

If there are not enough Sentinels to agree that the main library has gone offline, when the main library returns a valid reply to the Sentinel's PING command, the subjective offline status of the main library will be removed. And if more than 2 Sentinels mark the main library as subjectively offline, the main library will be marked as objectively down (ODOWN for short). The sentinel is about to start the next decision-making process, which is to select a slave library from many slave libraries to be the new master library.

4. How to select a new main library?

4.1 Initial Screening

Imagine that if a slave library is running normally when the master is selected, we choose it as the new master library and start using it. However, soon its network failed, and at this time, we had to re-elect the master. This is obviously not what we expected. Therefore, when selecting a master, in addition to checking the current online status of the slave library, it is also necessary to judge its previous network connection status. If the slave library is always disconnected from the master library, and the number of disconnections exceeds a certain threshold, we have reason to believe that the network condition of the slave library is not very good, so we can filter out the slave library.

initial screening

How to judge? You use the configuration item down-after-milliseconds * 10. Among them, down-after-milliseconds is the maximum connection timeout time that we believe the master-slave library is disconnected. If within down-after-milliseconds milliseconds, the master and slave nodes are not connected through the network, we can consider the master and slave nodes to be disconnected. If the number of disconnections exceeds 10 times, it means that the network condition of the slave library is not good, and it is not suitable to be used as the new master library.

In this way, we filter out the slave libraries that are not suitable for the main library, and complete the screening work.

The next step is to score the remaining slave libraries. We can perform three rounds of scoring in sequence according to three rules, which are slave library priority, slave library copy progress, and slave library ID number.

4.2 Three rounds of scoring

The first round: The slave with the highest priority will get a higher score.

Users can set different priorities for different slave libraries through the slave-priority configuration item. For example, if you have two slave libraries with different memory sizes, you can manually set a high priority for the instance with large memory.

The second round: the slave library that is closest to the synchronization degree of the old master library has a high score.

The basis of this rule is that if the slave library that is closest to the old master library is selected as the master library, then the new master library will have the latest data.

How to judge the synchronization progress between the slave library and the old master library?

There is a process of command propagation when the master-slave library is synchronized. During this process, the master library will use master_repl_offset to record the position of the latest write operation in repl_backlog_buffer, and the slave library will use the value of slave_repl_offset to record the current replication progress.

As shown in the figure below, slave library 2 should be selected as the new master library.

The third round: the slave with the smaller ID number gets the higher score.

Each instance will have an ID, which is similar to the number of the slave library here. At present, when Redis selects the master library, there is a default rule: in the case of the same priority and replication progress, the slave library with the smallest ID number has the highest score and will be selected as the new master library.

At this point, the new main library is selected, and the next step is to upgrade the slave library to the main library. But here comes the question again, with so many sentinels, who should perform the master-slave switching operation?

4.3 Which sentinel performs master-slave switching?

Any Sentinel instance will send the SENTINEL is-master-down-by-addr command to other Sentinels as long as it judges that the master library is "subjectively offline" to ask whether the other party thinks the master library is offline. Then, other sentinel instances will respond with Y or N according to their connection with the main library, Y is equivalent to a vote in favor, and N is equivalent to a negative vote.

At this point, the sentinel can send commands to other sentries, indicating that it wants to perform the master-slave switch by itself, and let all other sentries vote. This voting process is called "Leader Election". The elected Leader is the sentinel that finally performs the master-slave switchover.

For example, now there are 3 sentinels, and the quorum configuration is 2. Let's see what the election process looks like.

At T1, S1 judges that the main database is "objectively offline". If it wants to become a leader, it first votes for itself, and then sends commands to S2 and S3 respectively, indicating that it wants to become a leader.

At T2, S3 judges that the main database is "objectively offline", and it also wants to become a leader, so it votes for itself first, and then sends commands to S1 and S2 respectively, indicating that it wants to become a leader.

At time T3, S1 receives the Leader voting request from S3. Because S1 has voted Y for itself, it can no longer vote for other sentries, so S1 replies N to express its disagreement. At the same time, S2 receives the Leader voting request sent by S3 at T2. Because S2 has not voted before, it will reply Y to the first sentinel who sent a voting request to it, and reply N to the sentinel who sent a voting request later. Therefore, at T3, S2 replies to S3 and agrees that S3 becomes the Leader.

At T4, S2 receives the voting command sent by S1 at T1. Because S2 has agreed to S3's voting request at T3, at this time, S2 replies N to S1, expressing its disapproval of S1 becoming the leader. This happens because the network traffic between S3 and S2 is normal, but the network traffic between S1 and S2 may just be congested, causing the voting request to be transmitted slowly.

At time T5, S1 gets one vote Y from itself and one vote N from S2. In addition to its own vote Y, S3 also received a vote Y from S2. At this time, S3 not only obtained more than half of the Leader's votes, but also reached the preset quorum value (quorum is 2), so it finally became the Leader.

Then, S3 will start to perform the master selection operation, and after the new master library is selected, it will notify other slave libraries and clients of the new master library information.

5. Notify the new master library to the slave library and the client

Through the above study, we know that the sentinel can send the INFO command to the master library to obtain the IP address and port of the slave library.

However, Sentinel cannot only be connected to the master and slave libraries. Because, after the master-slave library is switched, the client also needs to know the connection information of the new master library before sending a request operation to the new master library. Therefore, Sentinel also needs to tell the client the information of the new main library.

Then how to tell the client the information of the new main library?

5.1 Client event notification based on pub/sub mechanism

In essence, Sentinel is a Redis instance running in a specific mode, but it does not serve request operations, but only completes the tasks of monitoring, master selection and notification. So, each Sentinel instance also provides pub/sub mechanism, clients can subscribe messages from Sentinel.

The figure below shows some important channels and several key events involved. You can check out more channels at the link [3] at the end of the article.

After the client reads the configuration file of the sentinel from the main library, it can obtain the address and port of the sentinel, and establish a network connection with the sentinel.

When the sentinel selects the new master library, the client will see the following switch-master event. This event indicates that the main library has been switched, and the IP address and port information of the new main library is already available. At this time, the client can use the address and port of the new main library to communicate.

switch-master <master name> <oldip> <oldport> <newip> <newport>

summary

So far, we have finished learning the job responsibilities and details of the sentry. I have sorted out the knowledge digestion link of this article, as follows.

在sentinel.conf中配置哨兵
-> 没有配置其他哨兵ip，怎么组成集群的？
-> 哨兵是怎么知道从库的IP和端口的？
-> 职责1：如何判断主从库下线了？
-> 职责2：如何选定新主库？
-> 由哪个哨兵执行主从切换？
-> 职责3：如何把新主库告诉客户端？

I will continue to share more technical dry goods in the following content, so stay tuned. The public account "Student Yang technotes" welcomes technical exchanges.

Links mentioned in the text

[1] https://www.dbses.cn/technotes

[2] https://github.com/redis/redis/blob/unstable/sentinel.conf

[3] https://redis.io/docs/management/sentinel/#pubsub-messages

Attached Redis document

http://www.redis.cn/topics/sentinel.html

https://redis.io/docs/management/sentinel