We know that the master-slave replication mode, if the primary node is not due to a failure to provide normal services, the need for human promotion from the main node node, and at the same time to notify the application side to update the master node address, in the actual scene, this failure treatment It is clearly unreasonable. From the beginning Redis 2.8 provides Redis Sentinel (Sentinel) architecture to achieve automatic fault transfer, Sentine Sentinel is a high-availability solutions redis official, you can use it to monitor the operation of multiple redis service instances.
monitor
1) Sentinel will continue to check the master and slave is normal
2) if the Sentinel hung up, you can not monitor all require multiple Sentry, Sentinel network composed of
3) Monitoring with a master's Sentinel will automatically connect to form a distributed network of Sentine , exchanged with each other and communicate with each other on the monitored information server
4) when a Sentinel that has been monitored server offline, it will confirm the Sentinel other network, the server determines whether it has been offline
5) If off the assembly line of server-based servers, nameSentinel network off the assembly line will be the primary server for automatic failover, the primary server offline by an upgrade from the server to the new master server, and let transferred from the server to the new primary server next, the system in order to return to the normal state, the handover process, the presence of a short break in the main access.
6) off the assembly line of the old primary server back on line, it will be from the Sentinel, linked to the new primary server
sentinel Sentinel Configuration
Configuration 3 three sentinel composition Sentinel network specific configuration information is as follows:
# 10063812 sentinel surveillance master, select the new master takes a few sentries vote sentinel auth-pass mymaster 123456 # redis if a password is configured, then there must configure authentication, or can not automatically switch sentinel down-after-milliseconds mymaster 30000 # prison heartbeat sentinel and the master (in milliseconds), default 30 seconds sentinel parallel-syncs mymaster 1 # refers to the failover process, can be sentinel arranged in parallel from the number of nodes sentinel failover-timeout mymaster 180000 # failover timeout (ms) default 180 seconds. [root @ k8smaster config] # [root @ k8smaster config] #
grep -v "#" sentinel26381.conf |grep -v "^$"
daemonize yes
logfile "/data/redis/redis/log/sentinel26381.log"
bind 192.168.23.100
port 26381
dir "/data/redis/redis/data/26381"
sentinel monitor mymaster 192.168.23.100 6381 2
sentinel auth-pass mymaster 123456
sentinel down-after-milliseconds mymaster 30000
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 180000
[root@k8smaster config]#
[root@k8smaster config]# grep -v "#" sentinel26382.conf |grep -v "^$"
daemonize yes
logfile "/data/redis/redis/log/sentinel26382.log"
bind 192.168.23.100
port 26382
dir "/data/redis/redis/data/26382"
sentinel monitor mymaster 192.168.23.100 6381 2
sentinel auth-pass mymaster 123456
sentinel down-after-milliseconds mymaster 30000
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 180000
[root@k8smaster config]#
Sentinel log
Sentinel starts, monitors the master and slave redis redis
5500: X-12 is On Feb 05: 32: 57.837 IS 4b1bc5f719e3698fecdfd70074075fb91534d5d3 a runid to the Sentinel #
5500: X-12 is On Feb 05: 32: 57.837 # + Master Monitor 6381 Quorum mymaster 192.168.23.100 2
5500 : X-12 is On Feb 05: 32: 57.837 + * Slave Slave 192.168.23.100:6382 192.168.23.100 192.168.23.100 mymaster 6381 6382 @
5500: X-12 is On Feb 05: 32: 57.843 + * Slave Slave 192.168.23.100:6380 192.168.23.100 @ mymaster 192.168.23.100 6381 6380
5500: X-12 is On Feb 05: 32: 58.650 + * 26380 Sentinel Sentinel 192.168.23.100:26380 192.168.23.100 192.168.23.100 mymaster 6381 @
5500: On Feb 12 is X-05: 33 is: Sentinel Sentinel + 02.214 * 192.168.23.100:26382 192.168.23.100 26382 @ mymaster 192.168.23.100 6381
[root@k8smaster bin]# ./redis-cli -h 192.168.23.100 -p 6381 -a 123456 #当前的主从角色情况
192.168.23.100:6381> info replication
# Replication
role:master
connected_slaves:2
slave0:ip=192.168.23.100,port=6382,state=online,offset=164399,lag=1
slave1:ip=192.168.23.100,port=6380,state=online,offset=164256,lag=1
master_repl_offset:164542
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2
repl_backlog_histlen:164541
192.168.23.100:6381>
查看哨兵状态
[root@k8smaster bin]# ./redis-cli -h 192.168.23.100 -p 26381
192.168.23.100:26381> info
# Server
redis_version:3.0.7
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:ab5e1e9c6570ed79
redis_mode:sentinel
os:Linux 3.10.0-123.el7.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.8.5
process_id:5500
run_id:4b1bc5f719e3698fecdfd70074075fb91534d5d3
tcp_port:26381
uptime_in_seconds:212
uptime_in_days:0
hz:17
lru_clock:4445485
config_file:/data/redis/redis/config/sentinel26381.conf
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
master0:name=mymaster,status=ok,address=192.168.23.100:6381,slaves=2,sentinels=3
192.168.23.100:26381>
192.168.23.100:26381> sentinel master mymaster #查看主redis
1) "name"
2) "mymaster"
3) "ip"
4) "192.168.23.100"
5) "port"
6) "6381"
7) "runid"
8) "0f47b0f0665bca13cea1299ccf4be0ffca162d18"
9) "flags"
10) "master"
11) "pending-commands"
12) "0"
13) "last-ping-sent"
14) "0"
15) "last-ok-ping-reply"
16) "482"
17) "last-ping-reply"
18) "482"
19) "down-after-milliseconds"
20) "30000"
21) "info-refresh"
22) "8570"
23) "role-reported"
24) "master"
25) "role-reported-time"
26) "299561"
27) "config-epoch"
28) "0"
29) "num-slaves" #从redis数量
30) "2"
31) "num-other-sentinels"
32) "2"
33) "quorum"
34) "2"
35) "failover-timeout"
36) "180000"
37) "parallel-syncs"
38) "1"
192.168.23.100:26381>
192.168.23.100:26381> . 4) "192.168.23.100" . 3) "IP" 2) "192.168.23.100:6380". 1). 1) "name"sentinel slaves mymaster #redis node information
5) "port"
6) "6380"
7) "runid"
8) "91f858d501a66e63b5fe10932ebd412fd075e784"
9) "flags"
10) "slave"
11) "pending-commands"
12) "0"
13) "last-ping-sent"
14) "0"
15) "last-ok-ping-reply"
16) "813"
17) "last-ping-reply"
18) "813"
19) "down-after-milliseconds"
20) "30000"
21) "info-refresh"
22) "2388"
23) "role-reported"
24) "slave"
25) "role-reported-time"
26) "323882"
27) "master-link-down-time"
28) "0"
29) "master-link-status"
30) "ok"
31) "master-host"
32) "192.168.23.100"
33) "master-port"
34) "6381"
35) "slave-priority"
36) "100"
37) "slave-repl-offset"
38) "192037"
2) 1) "name"
2) "192.168.23.100:6382"
3) "ip"
4) "192.168.23.100"
5) "port"
6) "6382"
7) "runid"
8) "696f02bd78782e2518bf67fb7771fe5c6d8248df"
9) "flags"
10) "slave"
11) "pending-commands"
12) "0"
13) "last-ping-sent"
14) "0"
15) "last-ok-ping-reply"
16) "813"
17) "last-ping-reply"
18) "813"
19) "down-after-milliseconds"
20) "30000"
21) "info-refresh"
22) "2388"
23) "role-reported"
24) "slave"
25) "role-reported-time"
26) "323888"
27) "master-link-down-time"
28) "0"
29) "master-link-status"
30) "ok"
31) "master-host"
32) "192.168.23.100"
33) "master-port"
34) "6381"
35) "slave-priority"
36) "100"
37) "slave-repl-offset"
38) "192037"
192.168.23.100: 26381> 2) "192.168.23.100:26380". 1). 1) "name"Sentinel Sentinels other Sentry # mymaster
192.168.23.100:26381>
3) "ip"
4) "192.168.23.100"
5) "port"
6) "26380"
7) "runid"
8) "28397c385ecd7fbcd1b570ac879511943fecc086"
9) "flags"
10) "sentinel"
11) "pending-commands"
12) "0"
13) "last-ping-sent"
14) "0"
15) "last-ok-ping-reply"
16) "132"
17) "last-ping-reply"
18) "132"
19) "down-after-milliseconds"
20) "30000"
21) "last-hello-message"
22) "867"
23) "voted-leader"
24) "?"
25) "voted-leader-epoch"
26) "0"
2) 1) "name"
2) "192.168.23.100:26382"
3) "ip"
4) "192.168.23.100"
5) "port"
6) "26382"
7) "runid"
8) "6cdfe14263030ce126184c4f6bdec8696dcc9e57"
9) "flags"
10) "sentinel"
11) "pending-commands"
12) "0"
13) "last-ping-sent"
14) "0"
15) "last-ok-ping-reply"
16) "968"
17) "last-ping-reply"
18) "968"
19) "down-after-milliseconds"
20) "30000"
21) "last-hello-message"
22) "163"
23) "voted-leader"
24) "?"
25) "voted-leader-epoch"
26) "0"
192.168.23.100:26381>
Command to switch from the master redis
192.168.23.100:26381> Sentinel mymaster # failover command master redis6381 offline failover
the OK
192.168.23.100:26381> Sentinel GET-Master-by-name-addr mymaster #redis main switch from 6381 to 6382
. 1) "192.168.23.100"
2) "6382"
192.168.23.100:26381>
[root@k8smaster bin]# ./redis-cli -h 192.168.23.100 -p 6382 -a 123456
192.168.23.100:6382> info replication
# Replication
role:master
connected_slaves:2
slave0:ip=192.168.23.100,port=6380,state=online,offset=239310,lag=1
slave1:ip=192.168.23.100,port=6381,state=online,offset=239310,lag=1
master_repl_offset:239453
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:211913
repl_backlog_histlen:27541
192.168.23.100:6382>
哨兵切换日志
5500:X 12 Feb 05:39:53.790 # Executing user requested FAILOVER of 'mymaster' #redis主failover
5500:X 12 Feb 05:39:53.790 # +new-epoch 1
5500:X 12 Feb 05:39:53.790 # +try-failover master mymaster 192.168.23.100 6381
5500:X 12 Feb 05:39:53.838 # +vote-for-leader 4b1bc5f719e3698fecdfd70074075fb91534d5d3 1
5500:X 12 Feb 05:39:53.838 # +elected-leader master mymaster 192.168.23.100 6381
5500:X 12 Feb 05:39:53.838 # +failover-state-select-slave master mymaster 192.168.23.100 6381
5500:X 12 Feb 05:39:53.891 # +selected-slave slave 192.168.23.100:6382 192.168.23.100 6382 @ mymaster 192.168.23.100 6381
5500:X 12 Feb 05:39:53.891 * +failover-state-send-slaveof-noone slave 192.168.23.100:6382 192.168.23.100 6382 @ mymaster 192.168.23.100 6381
5500:X 12 Feb 05:39:53.974 * +failover-state-wait-promotion slave 192.168.23.100:6382 192.168.23.100 6382 @ mymaster 192.168.23.100 6381
5500:X 12 Feb 05:39:54.851 # +promoted-slave slave 192.168.23.100:6382 192.168.23.100 6382 @ mymaster 192.168.23.100 6381
5500:X 12 Feb 05:39:54.851 # +failover-state-reconf-slaves master mymaster 192.168.23.100 6381
5500:X 12 Feb 05:39:54.928 * +slave-reconf-sent slave 192.168.23.100:6380 192.168.23.100 6380 @ mymaster 192.168.23.100 6381
5500:X 12 Feb 05:39:55.910 * +slave-reconf-inprog slave 192.168.23.100:6380 192.168.23.100 6380 @ mymaster 192.168.23.100 6381
5500:X 12 Feb 05:39:55.911 * +slave-reconf-done slave 192.168.23.100:6380 192.168.23.100 6380 @ mymaster 192.168.23.100 6381
5500:X 12 Feb 05:39:55.992 # +failover-end master mymaster 192.168.23.100 6381
5500:X 12 Feb 05:39:55.993 # +switch-master mymaster 192.168.23.100 6381 192.168.23.100 6382 #redis主切换到6382
5500:X 12 Feb 05:39:55.993 * +slave slave 192.168.23.100:6380 192.168.23.100 6380 @ mymaster 192.168.23.100 6382
5500:X 12 Feb 05:39:55.993 * +slave slave 192.168.23.100:6381 192.168.23.100 6381 @ mymaster 192.168.23.100 6382
Simulation of downtime main redis
192.168.23.100:6382> the shutdown # 6382 closed redis main service
not connected>
5500:X 12 Feb 05:45:29.251 # +sdown master mymaster 192.168.23.100 6382 #redis主6382下线
5500:X 12 Feb 05:45:29.685 # +new-epoch 2
5500:X 12 Feb 05:45:29.690 # +vote-for-leader 6cdfe14263030ce126184c4f6bdec8696dcc9e57 2
5500:X 12 Feb 05:45:30.370 # +odown master mymaster 192.168.23.100 6382 #quorum 3/2
5500:X 12 Feb 05:45:30.370 # Next failover delay: I will not start a failover before Wed Feb 12 05:51:29 2020
5500:X 12 Feb 05:45:30.520 # +config-update-from sentinel 192.168.23.100:26382 192.168.23.100 26382 @ mymaster 192.168.23.100 6382
5500:X 12 Feb 05:45:30.520 # +switch-master mymaster 192.168.23.100 6382 192.168.23.100 6381 #redis的主切换到6381
5500:X 12 Feb 05:45:30.520 * +slave slave 192.168.23.100:6380 192.168.23.100 6380 @ mymaster 192.168.23.100 6381
5500:X 12 Feb 05:45:30.521 * +slave slave 192.168.23.100:6382 192.168.23.100 6382 @ mymaster 192.168.23.100 6381
192.168.23.100:26381> Sentinel GET-Master-by-name-addr mymaster #redis main switching to 6381
. 1) "192.168.23.100"
2) "6381"
192.168.23.100:26381>
192.168.23.100:6381> info replication
# Replication
role:master
connected_slaves:1
slave0:ip=192.168.23.100,port=6380,state=online,offset=16152,lag=0
master_repl_offset:16152
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2
repl_backlog_histlen:16151
192.168.23.100:6381>
[root @ k8smaster bin] # ./redis-server ../config/redis6382.conf # redis 6382 recovery services
[root @ k8smaster bin] #
192.168.23.100:6381> info replication #redis6382自动加入redis集群的从节点
# Replication
role:master
connected_slaves:2
slave0:ip=192.168.23.100,port=6380,state=online,offset=29821,lag=1
slave1:ip=192.168.23.100,port=6382,state=online,offset=0,lag=1
master_repl_offset:29987
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2
repl_backlog_histlen:29986
192.168.23.100:6381>
