Redis sentinel手动切换主备

Redis sentinel手动切换主备

实验环境

操作系统： Ubuntu 18.04.1 LTS
redis版本： redis 5.0.2

机器规划：

role	ip	port
Master	127.0.0.1	6379
Slave1	127.0.0.1	6380
Slave2	127.0.0.1	6381
Sentinel1	127.0.0.1	26379
Sentinel2	127.0.0.1	26380
Sentinel3	127.0.0.1	26381

触发主备切换

在任意sentinel节点执行failover命令：

root@sbc-VirtualBox:/etc/redis# redis-cli -p 26380 sentinel failover mymaster
OK

查看sentinel节点日志

root@sbc-VirtualBox:/etc/redis# vim /opt/soft/redis/data/26380.log

2031:X 26 Dec 2018 15:24:48.825 # Executing user requested FAILOVER of 'mymaster'
2031:X 26 Dec 2018 15:24:48.825 # +new-epoch 1
2031:X 26 Dec 2018 15:24:48.825 # +try-failover master mymaster 127.0.0.1 6379
2031:X 26 Dec 2018 15:24:48.887 # +vote-for-leader 75cd9743a892825573793c4417e7e8465d65b616 1
2031:X 26 Dec 2018 15:24:48.887 # +elected-leader master mymaster 127.0.0.1 6379
2031:X 26 Dec 2018 15:24:48.887 # +failover-state-select-slave master mymaster 127.0.0.1 6379
2031:X 26 Dec 2018 15:24:48.970 # +selected-slave slave 127.0.0.1:6380 127.0.0.1 6380 @ mymaster 127.0.0.1 6379
2031:X 26 Dec 2018 15:24:48.970 * +failover-state-send-slaveof-noone slave 127.0.0.1:6380 127.0.0.1 6380 @ mymaster 127.0.0.1 6379
2031:X 26 Dec 2018 15:24:49.033 * +failover-state-wait-promotion slave 127.0.0.1:6380 127.0.0.1 6380 @ mymaster 127.0.0.1 6379
2031:X 26 Dec 2018 15:24:49.944 # +promoted-slave slave 127.0.0.1:6380 127.0.0.1 6380 @ mymaster 127.0.0.1 6379
2031:X 26 Dec 2018 15:24:49.944 # +failover-state-reconf-slaves master mymaster 127.0.0.1 6379
2031:X 26 Dec 2018 15:24:50.010 * +slave-reconf-sent slave 127.0.0.1:6381 127.0.0.1 6381 @ mymaster 127.0.0.1 6379
2031:X 26 Dec 2018 15:24:50.953 * +slave-reconf-inprog slave 127.0.0.1:6381 127.0.0.1 6381 @ mymaster 127.0.0.1 6379
2031:X 26 Dec 2018 15:24:50.953 * +slave-reconf-done slave 127.0.0.1:6381 127.0.0.1 6381 @ mymaster 127.0.0.1 6379
2031:X 26 Dec 2018 15:24:51.028 # +failover-end master mymaster 127.0.0.1 6379
2031:X 26 Dec 2018 15:24:51.028 # +switch-master mymaster 127.0.0.1 6379 127.0.0.1 6380
2031:X 26 Dec 2018 15:24:51.029 * +slave slave 127.0.0.1:6381 127.0.0.1 6381 @ mymaster 127.0.0.1 6380
2031:X 26 Dec 2018 15:24:51.029 * +slave slave 127.0.0.1:6379 127.0.0.1 6379 @ mymaster 127.0.0.1 6380

提示已触发主备切换，原先的master节点6379变为slave，原先的slave节点6380变为master

查看6379的日志：

root@sbc-VirtualBox:/etc/redis# vim /opt/soft/redis/data/6379.log

1920:M 26 Dec 2018 15:24:49.034 # Connection with replica 127.0.0.1:6380 lost.
1920:M 26 Dec 2018 15:24:50.014 # Connection with replica 127.0.0.1:6381 lost.
1920:S 26 Dec 2018 15:25:00.069 * Before turning into a replica, using my master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
1920:S 26 Dec 2018 15:25:00.069 * REPLICAOF 127.0.0.1:6380 enabled (user request from 'id=1115 addr=127.0.0.1:44936 fd=8 name=sentinel-b6983fbf-cmd age=11 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=148 qbuf-free=32620 obl=36 oll=0 omem=0 events=r cmd=exec')
1920:S 26 Dec 2018 15:25:00.070 # CONFIG REWRITE executed with success.
1920:S 26 Dec 2018 15:25:00.532 * Connecting to MASTER 127.0.0.1:6380
1920:S 26 Dec 2018 15:25:00.532 * MASTER <-> REPLICA sync started
1920:S 26 Dec 2018 15:25:00.532 * Non blocking connect for SYNC fired the event.
1920:S 26 Dec 2018 15:25:00.532 * Master replied to PING, replication can continue...
1920:S 26 Dec 2018 15:25:00.533 * Trying a partial resynchronization (request 3f0a80689afa2c4fe0cc33b82ce97af74b847c88:3987638).
1920:S 26 Dec 2018 15:25:00.533 * Full resync from master: d62e33c39fd886cc9b78408cde7ed815c17ecc2d:3988096
1920:S 26 Dec 2018 15:25:00.533 * Discarding previously cached master state.
1920:S 26 Dec 2018 15:25:00.635 * MASTER <-> REPLICA sync: receiving 189 bytes from master
1920:S 26 Dec 2018 15:25:00.635 * MASTER <-> REPLICA sync: Flushing old data
1920:S 26 Dec 2018 15:25:00.635 * MASTER <-> REPLICA sync: Loading DB in memory
1920:S 26 Dec 2018 15:25:00.635 * MASTER <-> REPLICA sync: Finished with success

6379已经从master变为slave

查看6380节点日志：

root@sbc-VirtualBox:/etc/redis# vim /opt/soft/redis/data/6380.log

1928:M 26 Dec 2018 15:24:49.033 # Setting secondary replication ID to 3f0a80689afa2c4fe0cc33b82ce97af74b847c88, valid up to offset: 3985643. New replication ID is d62e33c39fd886cc9b78408cde7ed815c17ecc2d
1928:M 26 Dec 2018 15:24:49.033 # Connection with master lost.
1928:M 26 Dec 2018 15:24:49.033 * Caching the disconnected master state.
1928:M 26 Dec 2018 15:24:49.033 * Discarding previously cached master state.
1928:M 26 Dec 2018 15:24:49.033 * MASTER MODE enabled (user request from 'id=7 addr=127.0.0.1:51534 fd=11 name=sentinel-75cd9743-cmd age=20204 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=140 qbuf-free=32628 obl=36 oll=0 omem=0 events=r cmd=exec')
1928:M 26 Dec 2018 15:24:49.034 # CONFIG REWRITE executed with success.
1928:M 26 Dec 2018 15:24:50.400 * Replica 127.0.0.1:6381 asks for synchronization
1928:M 26 Dec 2018 15:24:50.400 * Partial resynchronization not accepted: Requested offset for second ID was 3986042, but I can reply up to 3985643
1928:M 26 Dec 2018 15:24:50.400 * Starting BGSAVE for SYNC with target: disk
1928:M 26 Dec 2018 15:24:50.400 * Background saving started by pid 4136
4136:C 26 Dec 2018 15:24:50.414 * DB saved on disk
4136:C 26 Dec 2018 15:24:50.415 * RDB: 0 MB of memory used by copy-on-write
1928:M 26 Dec 2018 15:24:50.502 * Background saving terminated with success
1928:M 26 Dec 2018 15:24:50.502 * Synchronization with replica 127.0.0.1:6381 succeeded
1928:M 26 Dec 2018 15:25:00.533 * Replica 127.0.0.1:6379 asks for synchronization
1928:M 26 Dec 2018 15:25:00.533 * Partial resynchronization not accepted: Requested offset for second ID was 3987638, but I can reply up to 3985643
1928:M 26 Dec 2018 15:25:00.533 * Starting BGSAVE for SYNC with target: disk
1928:M 26 Dec 2018 15:25:00.533 * Background saving started by pid 4137
4137:C 26 Dec 2018 15:25:00.547 * DB saved on disk
4137:C 26 Dec 2018 15:25:00.548 * RDB: 0 MB of memory used by copy-on-write
1928:M 26 Dec 2018 15:25:00.634 * Background saving terminated with success
1928:M 26 Dec 2018 15:25:00.635 * Synchronization with replica 127.0.0.1:6379 succeeded

6380从slave升为了master，其后6379节点作为slave连入集群

查看6381节点日志：

root@sbc-VirtualBox:/etc/redis# vim /opt/soft/redis/data/6381.log

1934:S 26 Dec 2018 15:24:50.013 # Connection with master lost.
1934:S 26 Dec 2018 15:24:50.013 * Caching the disconnected master state.
1934:S 26 Dec 2018 15:24:50.013 * REPLICAOF 127.0.0.1:6380 enabled (user request from 'id=7 addr=127.0.0.1:54626 fd=11 name=sentinel-75cd9743-cmd age=20205 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=281 qbuf-free=32487 obl=36 oll=0 omem=0 events=r cmd=exec')
1934:S 26 Dec 2018 15:24:50.014 # CONFIG REWRITE executed with success.
1934:S 26 Dec 2018 15:24:50.399 * Connecting to MASTER 127.0.0.1:6380
1934:S 26 Dec 2018 15:24:50.400 * MASTER <-> REPLICA sync started
1934:S 26 Dec 2018 15:24:50.400 * Non blocking connect for SYNC fired the event.
1934:S 26 Dec 2018 15:24:50.400 * Master replied to PING, replication can continue...
1934:S 26 Dec 2018 15:24:50.400 * Trying a partial resynchronization (request 3f0a80689afa2c4fe0cc33b82ce97af74b847c88:3986042).
1934:S 26 Dec 2018 15:24:50.400 * Full resync from master: d62e33c39fd886cc9b78408cde7ed815c17ecc2d:3986064
1934:S 26 Dec 2018 15:24:50.400 * Discarding previously cached master state.
1934:S 26 Dec 2018 15:24:50.502 * MASTER <-> REPLICA sync: receiving 189 bytes from master
1934:S 26 Dec 2018 15:24:50.502 * MASTER <-> REPLICA sync: Flushing old data
1934:S 26 Dec 2018 15:24:50.502 * MASTER <-> REPLICA sync: Loading DB in memory
1934:S 26 Dec 2018 15:24:50.502 * MASTER <-> REPLICA sync: Finished with success

6381节点脸上6380端口的master

验证主备切换成果

在6380端执行：

root@sbc-VirtualBox:/etc/redis# redis-cli -p 6380 info replication
# Replication
role:master
connected_slaves:2
slave0:ip=127.0.0.1,port=6381,state=online,offset=4120328,lag=0
slave1:ip=127.0.0.1,port=6379,state=online,offset=4120328,lag=1
master_replid:d62e33c39fd886cc9b78408cde7ed815c17ecc2d
master_replid2:3f0a80689afa2c4fe0cc33b82ce97af74b847c88
master_repl_offset:4120328
second_repl_offset:3985643
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:3071753
repl_backlog_histlen:1048576

可看到主备切换已成功实现。

流程

• 1.手动触发failover模拟主节点出现故障，此时两个从节点与主节点失去连接，主从复制失败。
• 2.集群自动选出一个从节点（6380），对其执行slaveof no one命令使其成为新的主节点。
• 3.原来的从节点（6380）成为新的主节点后，更新sentinel的主节点信息。
• 4.sentinel命令另一个从节点（6381）去复制新的主节点（6380）。
• 5.让原来的主节点（6379）去复制新的主节点。