Redis - sentinel mode
1. Redis sentinel mode
The method of master-slave switching technology is: when the server is down, it is necessary to manually switch a slave machine to the master machine, which requires manual intervention, which is not only time-consuming and laborious, but also causes the service to be unavailable for a period of time. In order to solve the shortcomings of master-slave replication, there is a sentinel mechanism.
Sentinel's core functionality: On the basis of master-slave replication, Sentinel introduces automatic failover of the master node.
1. The role of sentinel mode
● Monitoring: Sentry will constantly check whether the master node and slave nodes are functioning properly.
●Automatic failover: When the master node fails to work normally, Sentinel will start an automatic failover operation, which will upgrade one of the slave nodes of the failed master node to a new master node, and let other slave nodes replicate the new master node .
● Notification (reminder): Sentinel can send the result of failover to the client.
The sentinel structure consists of two parts, sentry nodes and data nodes:
●Sentry node: The sentinel system consists of one or more sentinel nodes, which are special redis nodes that do not store data.
● Data nodes: both master nodes and slave nodes are data nodes.
2. Failover mechanism
-
The sentinel node regularly monitors to find out whether the master node is faulty.
Each sentinel node will send a ping command to the master node, slave node and other sentinel nodes every 1 second for a heartbeat detection. If the master node does not reply within a certain time frame or replies with an error message, then the sentinel will consider the master node to be offline subjectively (unilaterally). When more than half of the sentinel nodes think that the master node is offline subjectively, it is objectively offline. -
When the master node fails, the sentinel node will implement the election mechanism through the Raft algorithm (election algorithm) to jointly elect a sentinel node as the leader to be responsible for handling the failover and notification of the master node. So the number of clusters running Sentinels must not be less than 3 nodes.
-
The failover is performed by the leader sentinel node, and the process is as follows:
● Upgrade a certain slave node to a new master node, and let other slave nodes point to the new master node;
● If the original master node recovers, it also becomes a slave node and points to the new master node Node;
●Notify the client that the primary node has been replaced.
It is important to note that objective offline is a concept unique to the master node; if a slave node or sentinel node fails and is subjectively offlined by the sentinel, there will be no subsequent objective offline and failover operations.
3. Election of the master node
1. Filter out unhealthy (offline) slave nodes that do not respond to sentinel ping responses.
2. Select the slave node with the highest priority configuration in the configuration file. (replica-priority, the default value is 100)
3. Select the slave node with the largest replication offset, that is, the most complete replication.
The start of sentry depends on the master-slave mode, so the master-slave mode must be installed before doing the sentinel mode
2. Build Redis sentinel mode
Master node: 192.168.30.50
Slave1 node: 192.168.30.40
Slave2 node: 192.168.30.30
systemctl stop firewalld
setenforce 0
1. Modify the configuration file of Redis sentinel mode (all node operations)
cp /opt/redis-7.0.9/sentinel.conf /usr/local/redis/conf/
chown redis.redis /usr/local/redis/conf/sentinel.conf
vim /usr/local/redis/conf/sentinel.conf
protected-mode no #6行，关闭保护模式
port 26379 #10行，Redis哨兵默认的监听端口
daemonize yes #15行，指定sentinel为后台启动
pidfile /usr/local/redis/log/redis-sentinel.pid #20行，指定 PID 文件
logfile "/usr/local/redis/log/sentinel.log" #25行，指定日志存放路径
dir /usr/local/redis/data #54行，指定数据库存放路径
sentinel monitor mymaster 192.168.30.50 6379 2 #73行，修改 指定该哨兵节点监控192.168.30.50:6379这个主节点，该主节点的名称是mymaster，最后的2的含义与主节点的故障判定有关：至少需要2个哨兵节点同意，才能判定主节点故障并进行故障转移
#sentinel auth-pass mymaster abc123 #76行，可选，指定Master节点的密码，仅在Master节点设置了requirepass
sentinel down-after-milliseconds mymaster 3000 #114行，判定服务器down掉的时间周期，默认30000毫秒（30秒）
sentinel failover-timeout mymaster 180000 #214行，同一个sentinel对同一个master两次failover之间的间隔时间（180秒）
2. Start sentry mode
先启master，再启slave
cd /usr/local/redis/conf/
redis-sentinel sentinel.conf &
3. View sentinel information
redis-cli -p 26379 info Sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_tilt_since_seconds:-1
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.30.50:6379,slaves=2,sentinels=3
4. Fault simulation
查看redis-server进程号：
ps -ef | grep redis
root 65210 1 0 14:11 ? 00:00:04 redis-sentinel *:26379 [sentinel]
redis 65225 1 0 14:21 ? 00:00:02 /usr/local/redis/bin/redis-server 0.0.0.0:6379
root 65241 4316 0 14:44 pts/2 00:00:00 grep --color=auto redis
杀死 Master 节点上redis-server的进程号
kill -9 65210 Master节点上redis-server的进程号
5. Verify the result
tail -f /usr/local/redis/log/sentinel.log
7578:X 03 Jul 2023 14:51:40.505 * +slave-reconf-done slave 192.168.30.40:6379 192.168.30.40 6379 @ mymaster 192.168.30.50 6379
7578:X 03 Jul 2023 14:51:40.561 # +failover-end master mymaster 192.168.30.50 6379
7578:X 03 Jul 2023 14:51:40.561 # +switch-master mymaster 192.168.30.50 6379 192.168.30.30 6379
7578:X 03 Jul 2023 14:51:40.561 * +slave slave 192.168.30.40:6379 192.168.30.40 6379 @ mymaster 192.168.30.30 6379
7578:X 03 Jul 2023 14:51:40.561 * +slave slave 192.168.30.50:6379 192.168.30.50 6379 @ mymaster 192.168.30.30 6379
7578:X 03 Jul 2023 14:51:40.563 * Sentinel new configuration saved on disk
7578:X 03 Jul 2023 14:51:43.570 # +sdown slave 192.168.30.50:6379 192.168.30.50 6379 @ mymaster 192.168.30.30 6379
7578:X 03 Jul 2023 14:58:42.527 # -sdown slave 192.168.30.50:6379 192.168.30.50 6379 @ mymaster 192.168.30.30 6379
7578:X 03 Jul 2023 14:58:52.461 * +reboot slave 192.168.30.50:6379 192.168.30.50 6379 @ mymaster 192.168.30.30 6379
7578:X 03 Jul 2023 15:02:21.767 # +sdown slave 192.168.30.50:6379 192.168.30.50 6379 @ mymaster 192.168.30.30 6379
redis-cli -p 26379 INFO Sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_tilt_since_seconds:-1
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.30.30:6379,slaves=2,sentinels=3
