Sentinel is also a Redis server, but it has different functions from the Redis server we usually mentioned. Sentinel is responsible for monitoring ordinary Redis servers and improving the robustness and reliability of a server cluster. Sentinel and ordinary Redis server use the same server framework, which includes: network framework, underlying data structure, subscription and publishing mechanism, etc.
The timing program is an important role of the sentinel server. The main tasks include: monitoring ordinary Redis servers (including master and slave), performing fault repair, and executing script commands.
1. The role of Sentinel:
A. Master status monitoring
B. If the master is abnormal, the master-slave conversion will be performed, one of the slaves will be used as the master, and the previous master will be used as the slave
C. After the Master-Slave switch, the contents of master_redis.conf, slave_redis.conf and sentinel.conf will change, that is, there will be one more line of slaveof configuration in master_redis.conf, and the monitoring target of sentinel.conf will be changed accordingly
2. How Sentinel works:
1) Each Sentinel sends a PING command to the Master, Slave and other Sentinel instances it knows about once per second
2) If an instance (instance) is more than the value specified by the down-after-milliseconds option from the last valid reply to the PING command, the instance will be marked as subjective offline by Sentinel.
3) If a Master is marked as subjectively offline, all Sentinels monitoring the Master should confirm that the Master has indeed entered the subjectively offline state at a frequency of once per second.
4) When a sufficient number of Sentinels (greater than or equal to the value specified in the configuration file) confirm that the Master has indeed entered the subjective offline state within the specified time range, the Master will be marked as an objective offline
5) Under normal circumstances, each Sentinel will send an INFO command to all known Masters and Slaves every 10 seconds.
6) When the Master is marked as objective offline by Sentinel, the frequency of sending INFO commands to all slaves of the offline Master by Sentinel will be changed from once every 10 seconds to once per second
7) If there are not enough Sentinels to agree that the Master has been offline, the objective offline status of the Master will be removed.
If the Master returns a valid reply to Sentinel's PING command, the Master's subjective offline status will be removed.
Redis Sentinel Documentation
Redis Sentinel provides high availability for Redis. In practical terms this means that using Sentinel you can create a Redis deployment that resists without human intervention to certain kind of failures.
Redis Sentinel also provides other collateral tasks such as monitoring, notifications and acts as a configuration provider for clients.
This is the full list of Sentinel capabilities at a macroscopical level (i.e. the big picture):
Monitoring. Sentinel constantly checks if your master and slave instances are working as expected.
Notification. Sentinel can notify the system administrator, another computer programs, via an API, that something is wrong with one of the monitored Redis instances.
Automatic failover. If a master is not working as expected, Sentinel can start a failover process where a slave is promoted to master, the other additional slaves are reconfigured to use the new master, and the applications using the Redis server informed about the new address to use when connecting.
Configuration provider. Sentinel acts as a source of authority for clients service discovery: clients connect to Sentinels in order to ask for the address of the current Redis master responsible for a given service. If a failover occurs, Sentinels will report the new address.
The Sentinel system for Redis is used to manage multiple Redis servers (instances). The system performs the following three tasks:
Monitoring : Sentinel constantly checks that your master and slave servers are functioning properly.
Notification (Notification) : Sentinel can send notifications to administrators or other applications through API when there is a problem with one of the monitored Redis servers.
Automatic failover : When a master server fails to work properly, Sentinel will start an automatic failover operation, which will upgrade one of the slave servers of the failed master server to the new master server, and let the failed master server The other slave servers are changed to replicate the new master server; when the client tries to connect to the failed master server, the cluster will also return the address of the new master server to the client, so that the cluster can use the new master server instead of the failed server.
Configuration Provider: Sentinel as an authoritative source for Redis client discovery: The address where clients connect to Sentinel to request the current reliable master. In the event of a failure, Sentinel will report the new address.
Distributed nature of Sentinel
Redis Sentinel is a distributed system:
Sentinel itself is designed to run in a configuration where there are multiple Sentinel processes cooperating together. The advantage of having multiple Sentinel processes cooperating are the following:
- Failure detection is performed when multiple Sentinels agree about the fact a given master is no longer available. This lowers the probability of false positives.
- Sentinel works even if not all the Sentinel processes are working, making the system robust against failures. There is no fun in having a fail over system which is itself a single point of failure, after all.
The sum of Sentinels, Redis instances (masters and slaves) and clients connecting to Sentinel and Redis, are also a larger distributed system with specific properties. In this document concepts will be introduced gradually starting from basic information needed in order to understand the basic properties of Sentinel, to more complex information (that are optional) in order to understand how exactly Sentinel works.
Sentinel commands
The following is a list of accepted commands, not covering commands used in order to modify the Sentinel configuration, which are covered later.
- PING This command simply returns PONG.
- SENTINEL masters Show a list of monitored masters and their state.
- SENTINEL master
<master name>Show the state and info of the specified master. - SENTINEL slaves
<master name>Show a list of slaves for this master, and their state. - SENTINEL sentinels
<master name>Show a list of sentinel instances for this master, and their state. - SENTINEL get-master-addr-by-name
<master name>Return the ip and port number of the master with that name. If a failover is in progress or terminated successfully for this master it returns the address and port of the promoted slave. - SENTINEL reset
<pattern>This command will reset all the masters with matching name. The pattern argument is a glob-style pattern. The reset process clears any previous state in a master (including a failover in progress), and removes every slave and sentinel already discovered and associated with the master. - SENTINEL failover
<master name>Force a failover as if the master was not reachable, and without asking for agreement to other Sentinels (however a new version of the configuration will be published so that the other Sentinels will update their configurations). - SENTINEL ckquorum
<master name>Check if the current Sentinel configuration is able to reach the quorum needed to failover a master, and the majority needed to authorize the failover. This command should be used in monitoring systems to check if a Sentinel deployment is ok. - SENTINEL flushconfig Force Sentinel to rewrite its configuration on disk, including the current Sentinel state. Normally Sentinel rewrites the configuration every time something changes in its state (in the context of the subset of the state which is persisted on disk across restart). However sometimes it is possible that the configuration file is lost because of operation errors, disk failures, package upgrade scripts or configuration managers. In those cases a way to to force Sentinel to rewrite the configuration file is handy. This command works even if the previous configuration file is completely missing.
Sentinel command
Below is a series of receive commands, not all of them, to modify Sentinel configuration.
PING - This command simply returns PONE.
SENTINEL masters - Displays a list of monitored masters and their status.
SENTINEL master [master name] - Displays the status and information of the specified master.
SENTINEL slaves [master name] - Displays the master's slave list and their status.
SENTINEL sentinels [master name] - 展示master的sentinel实例的清单和它们的状态。
SENTINEL get-master-addr-by-name [master name] - 返回master的IP和端口。如果故障转移在处理中或成功终止，返回晋升的slave的IP和端口。
SENTINEL reset [pattern] - 这个命令将重置所有匹配名字的masters。参数是blog风格的。重置的过程清空master的所有状态，并移除已经发现和关联master的所有slave和sentinel。
SENTINEL failover [master name] - 如果master不可到达，强制执行一个故障转移，而不征求其他Sentinel的同意。
SENTINEL ckquorum [master name] - 检查当前的Sentinel配置是否能够到达故障转移需要的法定人数，并且需要授权故障转移的多数。这个命令应该用于监控系统检查部署是否正确。
SENTINEL flushconfig - 强制Sentinel在磁盘上重写它的配置，包括当前的Sentinel状态。通常Sentinel每次重写配置改变它的状态。然而有时由于操作错误、硬盘故障、包升级脚本或配置管理器可能导致配置文件丢失。在这种情况下收到强制Sentinel重写配置文件。这个命令即使上面的配置文件完全不见了。
运行sentinel有两种方式：
第一种
redis-sentinel /path/to/sentinel.conf
第二种
redis-server /path/to/sentinel.conf --sentinel
以上两种方式，都必须指定一个sentinel的配置文件sentinel.conf，如果不指定，将无法启动sentinel。sentinel默认监听26379端口，所以运行前必须确定该端口没有被别的进程占用。