[Redis study notes (12)] Sentinel detailed introduction

This article is published by the official account [Developing Pigeon]! Welcome to follow! ! !

Old Rules-Sister Town House:

One. Sentinel

(I. Overview

       Sentinel Sentinel is a highly available solution for Redis. The Sentinel system consisting of one or more Sentinel instances can monitor any number of master servers and slave servers. And when the monitored master server goes offline, one of its slave servers will be automatically upgraded to a new master server, and it will continue to process command requests.

       When the offline time of the master server exceeds the upper limit set by the user, the Sentinel system will perform a failover operation on it. First, select one of its slave servers to upgrade to the master server, and then send new replication instructions to the other slave servers , Let them become the slave servers of the new master server. When all slave servers start to replicate the new master server, the failover operation is complete. In addition, Sentinel will also monitor the offline master server and set it as a slave server when it comes back online.

(2) Start and initialize Sentinel

       The steps to start a Sentinel are as follows:

1. Initialize the server

       Sentinel is essentially just a Redis server running in a special mode, so the first step is to initialize an ordinary Redis server, but it does not use a database, so it will not load RDB files and AOF files.

2. Use Sentinel dedicated code

       Replace part of the code used by ordinary Redis servers with Sentinel-specific code. For example, Sentinel does not use a database, so there are no commands such as set, and the command table of the server also needs to be replaced.

3. Initialize the Sentinel state

       Initially, Ahu has a Sentinel state structure that saves all Sentinel-related states in the server.

4. Initialize the master attribute in the Sentinel state

       The master dictionary records all relevant information of the master server monitored by Sentinel. The key of the dictionary is the name of the master server, and the value is the sentinelRedisInstance structure corresponding to the monitored master server, that is, the instance structure, which represents a monitored Redis server instance. The initialization of the dictionary is performed according to the loaded Sentinel configuration file.

5. Create a network connection to the main server

       The last step is to create a network connection to the monitored main server. Sentinel will become the client of the main server. It can send commands to the main server or obtain relevant information from the command reply. For each main server monitored by Sentinel, Sentinel creates two asynchronous network connections to the main server, one is a command connection, used to send and receive commands, and the other is a subscription connection, dedicated to subscribing to the main server __sentinel__: hello channel.

       Why are there two connections? Because of the current Redis publish and subscribe functions, the sent information will not be stored in the Redis server. In order not to lose any information of the channel, a dedicated subscription connection is used to receive messages from the channel.

(3) Obtain the main server information

       Sentinel will send an INFO command to the main server through a command connection at a frequency of once every 10 seconds by default, and obtain the main server information by replying. By analyzing the message reply, you can get the information of the master server itself, such as the running ID, server role; you can also get the information of the slave server, such as IP, port, Sentinel does not need the user to provide the address information of the slave server, and automatically discovers the slave server.

       According to the obtained master server information, the instance structure of the master server is updated. The slave server information will also be used to update the slaves dictionary in the master server instance structure. The key of the dictionary is the name of the slave server automatically set by Sentinel, in the format ip:port , Value This is the instance structure corresponding to the slave server.

(4) Obtain information from the server

       When Sentinel discovers that the master server has a new slave server, in addition to creating a corresponding instance structure for the new slave server, it will also create a command connection and a subscription connection to the slave server. After creating a command connection, Sentinel sends an INFO command to the slave server every 10 seconds to get a reply. According to the reply Sentinel can know the running ID, role, IP and port, priority, copy offset of the slave server, and then update the corresponding instance structure.

(5) Send information to the master server and slave server

       By default, Sentinel will send a command to all monitored master and slave servers through the command connection at a frequency of once every two seconds. This command sends a message to the server's __sentinel__:hello channel. The parameters of the message are recorded by Sentinel and the master server. IP, port, running ID, current configuration epoch.

(6) Receive channel information from the master server and the slave server

       After Sentinel establishes a subscription connection with the master-slave server, it will subscribe to this connection until the connection is disconnected. Sentinel not only sends messages to the channel through the command connection, but also receives messages from the channel through the subscription connection. For Sentinel monitoring the same server, the information sent by one Sentinel will be received by other Sentinel, and this information will be used to update the structure of other Sentinel.

1. Sentinels' automatic discovery

       The sentinels dictionary in the instance structure created by Sentinel for the main server saves the information of other Sentinels that also monitor the main server. The key is the name of the Sentinel, the format is ip:port, and the value is the instance structure corresponding to the Sentinels. When the target Sentinel receives the information sent by the source Sentinel, it will extract the parameters related to Sentinel and the parameters related to the master server, and then follow the master server instance structure in the new master dictionary and the sentinels dictionary in the instance structure, which is Say that a Sentinels can learn the existence of other Sentinels by receiving channel information, and send channel information to let other Sentinels until their existence, so Sentinels can also be automatically discovered.

2. Create command connections to other Sentinels

       When Sentinels discover a new Sentinel through channel information, it will not only create a new instance structure, but also create a command connection to the new Sentinel, and the new Sentinel will also create a command connection to this Sentinel. An interconnected network is formed to exchange information, but there will be no subscription connection between them, because the subscription connection is used to discover unknown Sentinel, and they are all known to each other.

(7) Detect subjective offline status

       By default, Sentinel sends a PING command to all instances that have created a command connection with it at a frequency of once per second, and judges whether the instance is online through the returned PONG command reply. The down-after-milliseconds option in the Sentinel configuration file specifies the time limit for Sentinel to judge the instance's subjective offline. If an instance continuously returns invalid responses to Sentinel within this time, Sentinel will modify the flags attribute of the instance structure and add SRI_S_DOWN The mark indicates that the instance has been subjectively offline.

(8) Detect objective offline status

       However, the subjective offline time set by multiple Sentinels monitoring the same main server may be different, and other Sentinels may think that the server is not subjectively offline. In order to confirm whether this server is really offline, if a Sentinel judges a server to be offline subjectively, it will send a query to other Sentinel that also monitors this server to see if they also think that the server has entered the offline state ( Subjective or objective), if the Sentinel receives a sufficient number of offline judgments from other Sentinels, Sentinel will judge the server as objectively offline and perform a failover operation.

       The judgment condition for objective offline is the value of the quorum parameter set in the Sentinel configuration. As long as the quorum Sentinel including the current Sentinel believes that the server is offline, the current Sentinel will judge the server as objectively offline.

(9) Election leader Sentinel

       When a primary server is judged to be objectively offline, the Sentinel monitoring this server will negotiate and elect a leader Sentinel to perform failover operations on the primary server of the offline server. All online Sentinel are eligible to be elected as the leader Sentinel. After each leader Sentinel election, regardless of whether it is successful or not, the election era of all Sentinel will be +1, which is actually a counter. In a configuration epoch, all Sentinels have an opportunity to set a certain Sentinel as the local leader Sentinel. Once the local leader is set, it cannot be changed in this configuration epoch.

       Every Sentinel that finds that the main server enters the objective offline will ask other Sentinels to set themselves as the local leader Sentinel. The rule for setting the local leader Sentinel is first come, first served. The target Sentinel will reply to its own part after receiving a setting command. The running ID and configuration epoch of the leader Sentinel. After the source Sentinel receives the command reply sent by the target Sentinel, it checks whether the configuration epoch in the reply is the same as that of itself, and whether the running ID is consistent with itself. If they are consistent, it means that the target Sentinel sets the source Sentinel as the local leader Sentinel. .

       If a certain Sentinel is set as the local leader Sentinel by more than half of the Sentinel, then this Sentinel becomes the leader Sentinel. If within a given time limit, none of them becomes the leader, then it will be re-elected after a certain period of time.

(10) Failover

       The leader Sentinel will perform a failover operation on the main server that has gone offline.

1. Choose a new master server

       Pick a slave server with good status and complete data, and then send the SLAVEOF no one command to this server to turn it into the master server. The selection rule is to save all the servers to a list, delete the offline, disconnected, delete the ones that have not communicated recently, delete the older ones with the saved data, and finally sort and select the ones with higher priority.

2. Modify the replication target of the slave server

       Send the SLAVEOF command to the slave server to achieve.

3. Change the offline master server to the slave server

       Modify the instance structure of the offline master server. When it goes online again, Sentinel will send it a SLAVEOF command to make it a slave server.