Author: Chinese Shi Shan
Redis Sentinel high availability cluster
Sentinel Introduction
sentinel, Chinese name is Sentinel. Sentinel is a very important institution in a redis cluster components, mainly in the following functions:
- Cluster Monitoring: redis master and slave is responsible for monitoring the process are working properly.
- Message notification: If a redis instance fails, then the sentry responsible for sending the message as an alarm notification to the administrator.
- Failover: If the master node hang, will be automatically transferred to the slave node.
- Configuration Center: If a failover occurs, notify the client the client a new master address.
Sentinel achieve high availability for redis cluster itself is distributed, a sentry to run as a cluster, with each other to work together.
- Failover, determine whether a master node goes down, most of the guards are required consent to it, it comes to the problems of distributed election.
- Even if some sentinel node hang, Sentinel cluster can still work properly, because if a failover system as an important part of the high availability mechanism itself is a single point, it would be very pit father.
Sentinel's core knowledge
- Sentinel requires at least three instances, to ensure their robustness.
- Sentinel + redis deployment architecture master-slave is no guarantee zero data loss, can only ensure high availability redis clusters.
- For Sentinel + redis master-slave architecture of this complex deployments, as far as possible in the test and production environments, we have carried out adequate tests and drills.
Sentinel cluster must deploy two or more nodes, the cluster only if the sentinel sentinel deployed two instances, quorum = 1.
+----+ +----+
| M1 |---------| R1 |
| S1 | | S2 |
+----+ +----+
Configuration quorum=1, if the master down, s1 and s2 as long as there is a Sentinel that the master is down, it can be switched, while s1 and s2 sentinel will elect to perform a failover. But at the same time this is required majority, that is, most of the guards are running.
2 个哨兵，majority=2
3 个哨兵，majority=2
4 个哨兵，majority=2
5 个哨兵，majority=3
...
If this time is just down the M1 process, Sentinel s1 normal operation, the failover is OK. However, if the entire machine downtime M1 and S1 operation, then only a sentinel, then there is no majority to allow a failover, although there is a another machine R1, but does not perform failover.
Classic 3-Sentinel node cluster like this:
+----+
| M1 |
| S1 |
+----+
|
+----+ | +----+
| R2 |----+----| R3 |
| S2 | | S3 |
+----+ +----+
Configuration quorum=2, if M1 where the machine is down, then the remaining three Sentinel 2, S2 and S3 may agree that the master is down, then elected to perform a failover, while three sentinels majority is 2, the remaining two guards running, it can allow failover.
Sentinel redis standby switching data loss problem
In both cases and lead to data loss
Standby switching process, may result in loss of data:
- Asynchronous replication of data loss caused by
Because master-> slave replication is asynchronous, so there may be some data is not copied to the slave, master it goes down, then this part of the data is lost.
- Data loss caused by brain split
Split brain, that is to say, a master machine where suddenly out of the normal network, the machine can not connect with other slave, but in fact master is also running. At this time, Sentinel might think that master is down, and then turn the election, the other slave switch became master. This time, the cluster will have two master, the so-called split-brain.
At this time, although a slave is switched became a master, but the client may not had time to switch to a new master, but also continued to write data to the old master. So when the old master once again restored, it will be as a slave to hang up the new master, your data will be cleared, re-copy the data from the new master. The new master data later client did not write, so this part of the data will be lost.
Data Loss Problem
As follows:
min-slaves-to-write 1
min-slaves-max-lag 10
表示，要求至少有 1 个 slave，数据复制和同步的延迟不能超过 10 秒。
如果说一旦所有的 slave，数据复制和同步的延迟都超过了 10 秒钟，那么这个时候，master 就不会再接收任何请求了。
- 减少异步复制数据的丢失
有了 min-slaves-max-lag 这个配置，就可以确保说，一旦 slave 复制数据和 ack 延时太长，就认为可能 master 宕机后损失的数据太多了，那么就拒绝写请求，这样可以把 master 宕机时由于部分数据未同步到 slave 导致的数据丢失降低的可控范围内。
- 减少脑裂的数据丢失
如果一个 master 出现了脑裂，跟其他 slave 丢了连接，那么上面两个配置可以确保说，如果不能继续给指定数量的 slave 发送数据，而且 slave 超过 10 秒没有给自己 ack 消息，那么就直接拒绝客户端的写请求。因此在脑裂场景下，最多就丢失 10 秒的数据。
sdown 和 odown 转换机制
- sdown 是主观宕机，就一个哨兵如果自己觉得一个 master 宕机了，那么就是主观宕机
- odown 是客观宕机，如果 quorum 数量的哨兵都觉得一个 master 宕机了，那么就是客观宕机
sdown 达成的条件很简单，如果一个哨兵 ping 一个 master，超过了 is-master-down-after-milliseconds 指定的毫秒数之后，就主观认为 master 宕机了；如果一个哨兵在指定时间内，收到了 quorum 数量的其它哨兵也认为那个 master 是 sdown 的，那么就认为是 odown 了。
哨兵集群的自动发现机制
哨兵互相之间的发现，是通过 redis 的 pub/sub 系统实现的，每个哨兵都会往 __sentinel__:hello 这个 channel 里发送一个消息，这时候所有其他哨兵都可以消费到这个消息，并感知到其他的哨兵的存在。
每隔两秒钟，每个哨兵都会往自己监控的某个 master+slaves 对应的 __sentinel__:hello channel 里发送一个消息，内容是自己的 host、ip 和 runid 还有对这个 master 的监控配置。
每个哨兵也会去监听自己监控的每个 master+slaves 对应的 __sentinel__:hello channel，然后去感知到同样在监听这个 master+slaves 的其他哨兵的存在。
每个哨兵还会跟其他哨兵交换对 master 的监控配置，互相进行监控配置的同步。
slave 配置的自动纠正
哨兵会负责自动纠正 slave 的一些配置，比如 slave 如果要成为潜在的 master 候选人，哨兵会确保 slave 复制现有 master 的数据；如果 slave 连接到了一个错误的 master 上，比如故障转移之后，那么哨兵会确保它们连接到正确的 master 上。
slave->master 选举算法
如果一个 master 被认为 odown 了，而且 majority 数量的哨兵都允许主备切换，那么某个哨兵就会执行主备切换操作，此时首先要选举一个 slave 来，会考虑 slave 的一些信息：
- 跟 master 断开连接的时长
- slave 优先级
- 复制 offset
- run id
如果一个 slave 跟 master 断开连接的时间已经超过了 down-after-milliseconds 的 10 倍，外加 master 宕机的时长，那么 slave 就被认为不适合选举为 master。
(down-after-milliseconds * 10) + milliseconds_since_master_is_in_SDOWN_state
接下来会对 slave 进行排序：
- 按照 slave 优先级进行排序，slave priority 越低，优先级就越高。
- 如果 slave priority 相同，那么看 replica offset，哪个 slave 复制了越多的数据，offset 越靠后，优先级就越高。
- If the above two conditions are identical, then select a run id relatively small one slave.
quorum and majority
Every time a sentry to do the switchover, first quorum required number of Sentinel think odown, then elected to switch to do a sentinel, the sentinel sentry majority also need to be authorized in order to execute the formal handover.
If the quorum <majority, such as 5 sentinel, Majority is 3, quorum is set to 2, then the three sentinel can perform handover authorization.
However, if the quorum> = majority, you must have authorization quorum number of sentries, such as five Sentinel, quorum is 5, then you must have agreed to authorize five guards to perform switching.
configuration epoch
Sentinel a redis master + slaves will monitor, a corresponding configuration of the monitor.
The sentinel performing handover will get from switching to a new master (salve-> master) a configuration epoch, which is a number version, version number every time you switch must be unique.
If the first elected Sentinel switch fails, then the other sentinel, waits for failover-timeout period, then proceed to take over the switch at this time will retrieve a new configuration epoch, as the new version number.
configuration propagation
Sentinel after completing the handover, will be updated in the latest generation of its own local master configuration, and then sync to other Sentinel is said before by the pub/sub message mechanism.
Before the version number here is very important, because all kinds of messages are posted and listen to through a channel, so after a sentry to complete a new switch, the new master configuration is followed by a new version number. Other sentinels are based on the size of the version number to update their master configuration.