There are roughly three types of Redis cluster solutions: 1) redis cluster cluster solution; 2) master/slave master-slave solution; 3) sentinel mode for master-slave replacement and fault recovery.
1. Sentinel Sentinel Mode Introduction
Sentinel (Sentinel) is a tool for monitoring the Master status in a redis cluster. Sentinel Sentinel mode has been integrated in versions after redis 2.4.
Sentinel functions:
1) Master status detection
2) If the Master is abnormal, the Master-Slave switch will be performed, one of the Slaves will be used as the Master, and the previous Master will be used as the Slave.
3) After the Master-Slave switch, the contents of master_redis.conf, slave_redis.conf and sentinel.conf will change, that is, there will be one more line of slaveof configuration in master_redis.conf, and the monitoring target of sentinel.conf will be changed accordingly.
How Sentinel works (timed tasks performed by each Sentinel instance):
1) Each Sentinel sends a PING command to the Master, Slave and other Sentinel instances it knows about once per second.
2) If an instance (instance) is more than the value specified by the own-after-milliseconds option from the last valid reply to the PING command, the instance will be marked as subjective offline by Sentinel.
3) If a Master is marked as subjective offline, all Sentinels monitoring the Master must confirm that the Master has indeed entered the subjective offline state at a frequency of once per second.
4) When a sufficient number of Sentinels (greater than or equal to the value specified in the configuration file) confirm that the Master has indeed entered the subjective offline state within the specified time range, the Master will be marked as an objective offline.
5) In general, each Sentinel will send an INFO command to all Masters and Slaves known to it every 10 seconds.
6) When the Master is marked as objectively offline by Sentinel, the frequency of sending INFO commands to all slaves of the Master who is offline by Sentinel will be changed from once every 10 seconds to once per second.
7) If there is not enough Sentinel to agree that the Master has been offline, the objective offline status of the Master will be removed. If the Master returns a valid reply to Sentinel's PING command, the Master's subjective offline status will be removed.
Subjective Downline and Objective Downline
1) Subjectively Down (SDOWN for short) refers to the offline judgment made by a single Sentinel instance on the server.
2) Objectively Down (ODOWN for short) refers to the server obtained after multiple Sentinel instances make SDOWN judgments on the same server and communicate with each other through the SENTINEL is-master-down-by-addr command Offline judgment, and then open failover.
Objective offline conditions apply only to masters: For any other type of Redis instance, Sentinel does not need to negotiate before determining them as offline, so slaves or other Sentinels will never reach objective offline conditions. As long as a Sentinel finds that a master server has entered an objectively offline state, this Sentinel may be elected by other Sentinels and perform automatic failover operations on the failed master server.
Sentinel API
has two ways to communicate with Sentinel: command, publish and subscribe.
Sentinel command
PING : returns PONG.
SENTINEL masters : Lists all monitored masters, and the current status of those masters;
SENTINEL slaves <master name> : Lists all slaves for a given master, and the current status of those slaves;
SENTINEL get-master -addr-by-name <master name> : Returns the IP address and port number of the master server with the given name. If this primary server is performing a failover operation, or a failover operation for this primary server has completed, then this command returns the IP address and port number of the new primary server;
SENTINEL reset <pattern> : resets all names and the given The master server that matches the pattern pattern. The pattern parameter is a Glob-style pattern. The reset operation clears all the current status of the master server, including the ongoing failover, and removes all slave servers and Sentinels of the master server that have been discovered and associated so far;
SENTINEL failover <master name> : When the master server fails, Forces an automatic failover to begin without asking other Sentinels for input.
Clients can obtain the current master server IP address and port number through SENTINEL get-master-addr-by-name <master name> , and SENTINEL slaves <master name> to obtain all slaves information.
Publish and subscribe messages
Clients can think of Sentinel as a Redis server that only provides subscription functionality: you cannot use the PUBLISH command to send messages to this server, but you can use the SUBSCRIBE command or the PSUBSCRIBE command to subscribe to a given channel by to get the corresponding event reminder.
A channel can receive events with the same name as the channel. For example, a channel named +sdown can receive events for all instances to enter the subjective downline (SDOWN) state. All event information can be received by executing the PSUBSCRIBE * command.
+switch-master <master name> <oldip> <oldport> <newip> <newport> : Configuration changed, the IP and address of the master server have changed. This is information that the vast majority of external users care about.
It can be seen that we can use the Sentinel command and the publish-subscribe mechanism to achieve a good integration with the client:
use the get-master-addr-by-name and slaves commands to obtain the current Master and Slaves addresses and information ; When a failover occurs, that is, when the Master switches, the latest Master information can be obtained through the subscribed +switch-master event.
The notification-script
in sentinel.conf can be configured with multiple sentinel notification-script <master name> <shell script-path> in sentinel.conf, such as sentinel notification-script mymaster ./check.sh
This is used when the cluster failsover Triggers execution of the specified script. If the execution result of the script is 1, it will be retried later (the maximum number of retries is 10); if it is 2, the execution will end. And the maximum execution time of the script is 60 seconds, and the execution will be terminated after the timeout.
At present, there is a problem that the script is executed multiple times. The explanation obtained from online search is: the script is divided into two levels, SENTINEL_LEADER and SENTINEL_OBSERVER, the former is only executed by the leading Sentinel (a Sentinel), while the latter is monitored by the same master All Sentinel executions (multiple Sentinels) .
Redis Sentinel's master-slave switching solution
Redis version 2.8 officially provides a master-slave switching solution called Sentinel. Sentinel is used to manage multiple Redis server instances and can implement a functionally HA cluster. It is mainly responsible for three tasks:
1) Monitoring: Sentinel will constantly check whether your master and slave servers are functioning properly.
2) Notification: When there is a problem with a monitored Redis server, Sentinel can send notifications to administrators or other applications through API.
3) Automatic failover: When a master server fails to work properly, Sentinel will start an automatic failover operation, which will upgrade one of the slave servers of the failed master server to a new master server, and let the failed master server. The other slave servers of the server are changed to replicate the new master server; when the client tries to connect to the failed master server, the cluster will also return the address of the new master server to the client, so that the cluster can use the new master server instead of the failed server.
Redis Sentinel is a distributed system that can run multiple Sentinel processes (progress) in an architecture. These processes use gossip protocols to receive information about whether the master server is offline or not, and use voting protocols (agreement protocols) to decide whether to perform an automatic failover and which slave to choose as the new master.
The architecture diagram of a simple master-slave structure plus sentinel cluster is as follows:
The above picture shows one master and one slave node, plus two clusters where Sentinel is deployed. Sentinel clusters will communicate with each other, communicate the status of redis nodes, make corresponding judgments and process them. The subjective offline status here and The objective offline state is a more important state, and they determine whether to perform failover.
You can subscribe to the specified channel information and notify the administrator when the server fails.
Clients can regard Sentinel as a Redis that only provides subscription functions Server, you cannot use PUBLISH command to send information to this server, but you can use SUBSCRIBE command or PSUBSCRIBE command to get corresponding event alerts by subscribing to a given channel.
A channel can receive events with the same name as the channel. For example, a channel named +sdown can receive events for all instances to enter the subjective downline (SDOWN) state.
Second, redis-sentinel sentinel mode cluster environment deployment record