Preface
We all know that if the service is deployed on only one node, it is prone to a single point of failure, resulting in unavailability of the service. Therefore, it is necessary for services to be highly available, and Redis services are naturally the same. This article mainly explains Redis high-availability clusters from both theoretical and practical aspects. Teach you how to build a highly available redis master-slave replication cluster.
This article adopts a method of interspersed explanation between theory and practice. If you only care about the construction of the cluster, you can skip the theoretical part of the article.
Share a Java interview information. Due to the length of the article, it is impossible to display all the interview questions in text format. This article has selected some interview questions for everyone. If you need this full version of the interview notes To get the interview information, just: click here to get it!!! Password: CSDN
Pre-reading
Redis persistence: Redis: Persistence
experimental environment
VMware Workstation 15
CentOS Linux release 7.7.1908
Redis-5.0.8
Precautions
The ip of the three nodes are 192.168.1.101, 192.168.1.102
, and 192.168.1.103 respectively. Ensure that the three nodes can access the Internet, and the three nodes can communicate with each other.
Ensure that the basic commands and compilers such as yum, wget, tar, and gcc of Linux are available
It is recommended to turn off the firewall first. Centos 7 operations are as follows: firewall-cmd --state ## View firewall status not running means that it has been turned off systemctl stop firewalld.service ## Turn off the firewall systemctl disable firewalld.service ## Prohibit booting the firewall
Redis stand-alone installation
Download wget download.redis.io/relea
unzip tar -zxvf redis-5.0.8.tar.gz
compile cd redis-5.0.8 make
install make install ## Or specify the installation directory make install PREFIX=specify the path. The default path is /usr/local/bin ./utils/install_server.sh ## Install as a service, if PREFIX is configured in the previous step, you need to configure the installation path to the environment variable /etc/profile
install_server.sh is a script provided by redis, After running, you will be asked to specify several configurations: port number, configuration file path, log file path, and data file path.
If they are all set to default values, redis distinguishes different instances on the same host according to the port number, because install_server.sh can be run multiple times, and each run is equivalent to installing an instance.
If the installation process is all installed by default, there will be the following configurations:
Port number: 6379
Configuration file path: /etc/redis/6379.conf
Log file path: /var/log/redis_6379.log
Data file path: /var/lib/redis/6379/
redis-server.sh path: /usr /local/bin/
redis-cli.sh path: /usr/local/bin/ The
following log will appear if the installation is successful
Copied /tmp/6379.conf => /etc/init.d/redis_6379 Installing service…
Successfully added to chkconfig! Successfully added to runlevels 345!
Starting Redis server… Installation successful!
You can see that the redis service has been started automatically.
Master-slave replication
Redis master-slave replication is a cluster implementation that comes with redis 3.0 and does not require other middleware. It is a master-slave implementation based on asynchronous replication. Therefore, Redis master-slave replication does not guarantee strong data consistency. The cluster may lose write operations under certain conditions.
Cluster structure
Now let's build a one-master and two-slave cluster. The cluster topology is as follows. 
The master node can be written and read and is generally used to process write requests. The slave node is read-only by default, so it is used to process read requests. The data of the two slave nodes are copied from the master node. So this kind of cluster is also called read-write separation.
Configuration
The default path of the redis configuration file is /etc/redis/6379.conf, opened with vi/vim, and the three nodes are configured as follows
The ip address that needs to be bound bind 127.0.0.1 192.168.1.101 192.168.1.102 192.168.1.103
Turn off the background operation, easy to observe daemonize no
Annotate the log path so that the log is output directly on the console for easy observation
logfile /var/log/redis_6379.log
Turn off AOF persistence mode
appendonly no
start
After the configuration is complete, start three nodes separately
cd /usr/local/bin redis-server /etc/redis/6379.conf
Set master-slave relationship
After the two slave nodes connect to the redis-server with the redis-cli client, execute the following commands to set themselves as the slave of the master node
> replicaof 192.168.1.101 6379
Replicaof can also be written directly in the configuration file (the text is executed by command for experimental effect)
################################# REPLICATION #################################
Master-Replica replication. Use replicaof to make a Redis instance a copy of
another Redis server. A few things to understand ASAP about Redis replication.
±-----------------+ ±--------------+
| Master | —> | Replica |
| (receive writes) | | (exact copy) |
±-----------------+ ±--------------+
1) Redis replication is asynchronous, but you can configure a master to
stop accepting writes if it appears to be not connected with at least
a given number of replicas.
2) Redis replicas are able to perform a partial resynchronization with the
master if the replication link is lost for a relatively small amount of
time. You may want to configure the replication backlog size (see the next
sections of this file) with a sensible value depending on your needs.
3) Replication is automatic and does not need user intervention. After a
network partition replicas automatically try to reconnect to masters
and resynchronize with them.
replicaof
The version of replicaof before redis5.0 is called slaveof, and the command is described as follows
127.0.0.1:6379> help slaveof
SLAVEOF host port summary: Make the server a replica of another
instance, or promote it as master. Deprecated starting with Redis 5.
Use REPLICAOF instead. since: 1.0.0 group: server127.0.0.1:6379> help replicaof
REPLICAOF host port summary: Make the server a replica of another
instance, or promote it as master. since: 5.0.0 group: server
After the command is successfully executed, the following log will appear on 192.168.1.101 (master)
1817:M 16 Apr 2020 22:33:36.802 * Replica 192.168.1.102:6379 asks for
synchronization 1817:M 16 Apr 2020 22:33:36.802 * Partial
resynchronization not accepted: Replication ID mismatch (Replica asked
for ‘e801c600a0a2381a65e1aec22daba7db82cb02f8’, my replication IDs are
‘be75572b8e6624da4971aa16448600c9822fd42a’ and
‘0000000000000000000000000000000000000000’) 1817:M 16 Apr 2020
22:33:36.803 * Starting BGSAVE for SYNC with target: disk 1817:M 16
Apr 2020 22:33:36.837 * Background saving started by pid 1822 1822:C
16 Apr 2020 22:33:36.944 * DB saved on disk 1822:C 16 Apr 2020
22:33:36.944 * RDB: 6 MB of memory used by copy-on-write 1817:M 16 Apr
2020 22:33:37.038 * Background saving terminated with success 1817:M
16 Apr 2020 22:33:37.038 * Synchronization with replica
192.168.1.102:6379 succeeded
Let's look at what 192.168.1.101 (master) did line by line.
The first line means that there is a salve node 192.168.1.102:6379 requesting synchronization. The
second line means that full synchronization will be performed, because it is the first time to request synchronization. The
third line means that BGSAVE will be executed and the data will be persisted to disk. The
fourth line It means that the pid is the 1822 child process to start the persistence. The
fifth line means that the persistence is completed. The
sixth line means that the copy-on-write mechanism uses 6M of memory. The
last two lines indicate that the synchronization process has been completed. The master node persists the data to disk in the form of RDB, and then sends it to the slave through the network. If the parameter repl-diskless-sync is set to no, it means that the data is sent directly to the slave without going through the disk.
Look at the log of 192.168.1.101 (master), then look at the log of the salve, and take any slave log
2013:S 16 Apr 2020 22:33:36.233 * Before turning into a replica, using
my master parameters to synthesize a cached master: I may be able to
synchronize with the new master with just a partial transfer. 2013:S
16 Apr 2020 22:33:36.233 * REPLICAOF 192.168.1.101:6379 enabled (user
request from ‘id=3 addr=127.0.0.1:33550 fd=8 name= age=4 idle=0
flags=N db=0 sub=0 psub=0 multi=-1 qbuf=49 qbuf-free=32719 obl=0 oll=0
omem=0 events=r cmd=replicaof’) 2013:S 16 Apr 2020 22:33:36.808 *
Connecting to MASTER 192.168.1.101:6379 2013:S 16 Apr 2020
22:33:36.808 * MASTER <-> REPLICA sync started 2013:S 16 Apr 2020
22:33:36.809 * Non blocking connect for SYNC fired the event. 2013:S
16 Apr 2020 22:33:36.810 * Master replied to PING, replication can
continue… 2013:S 16 Apr 2020 22:33:36.811 * Trying a partial
resynchronization (request
e801c600a0a2381a65e1aec22daba7db82cb02f8:1). 2013:S 16 Apr 2020
22:33:36.946 * Full resync from master:
a9861cdcfdb3358ea0a3bb5a4df2895938c1c2d0:0 2013:S 16 Apr 2020
22:33:36.946 * Discarding previously cached master state. 2013:S 16
Apr 2020 22:33:37.048 * MASTER <-> REPLICA sync: receiving 175 bytes
from master 2013:S 16 Apr 2020 22:33:37.048 * MASTER <-> REPLICA sync:
Flushing old data 2013:S 16 Apr 2020 22:33:37.048 * MASTER <-> REPLICA
sync: Loading DB in memory 2013:S 16 Apr 2020 22:33:37.048 * MASTER
<-> REPLICA sync: Finished with success
There are many salve node logs, tell us what we did
Request synchronization to 192.168.1.101:6379 ( master )
Send command SYNC and
receive master's reply.
Full synchronization, received 175 bytes.
Flushing old data.
Loads data sent by master to memory (Loading DB in memory)
combined with master And the slave log, you can see the general process of replication.
The complete master-slave replication process is as follows: When the 
master receives the first synchronization requested by a slave, it will perform full synchronization. During the synchronization, the executed data modification command will be written into the cache, and the synchronization will be completed before sending Execute to the slave node. After the first full synchronization is completed, the master will continue to send write commands to the slave nodes to ensure data consistency between the master and slave nodes.
Here is a question to think about, can the data in the slave node be queried by the client during the time when the slave node is fully synchronized?
The replicaof-server-stale-data parameter is set to yes to indicate that it can be checked, and set to no to indicate that synchronization must be completed to check.
operating
Write data to the master node first
192.168.1.101:6379> set key1 hello OK
Then get it from the slave node (note the ip address in the prompt), there is no doubt that it can be obtained
192.168.1.102:6379> get key1 “hello”
What happens if you write data to the slave node?
By default, slave nodes are forbidden to write, so an error will be reported.
192.168.1.102:6379> set key2 world (error) READONLY You can’t write against a read only replica.
The replica-read-only parameter can set the slave to allow writing
You can configure a replica instance to accept writes or not. Writing against
a replica instance may be useful to store some ephemeral data (because data
written on a replica will be easily deleted after resync with the master) but
may also cause problems if clients are writing to it because of a
misconfiguration.
Since Redis 2.6 by default replicas are read-only.
Note: read only replicas are not designed to be exposed to untrusted clients
on the internet. It’s just a protection layer against misuse of the instance.
Still a read only replica exports by default all the administrative commands
such as CONFIG, DEBUG, and so forth. To a limited extent you can improve
security of read only replicas using ‘rename-command’ to shadow all the
administrative / dangerous commands. replica-read-only yes
So far, the simplest master-slave replication cluster has been built.
malfunction
You are already a mature programmer, and you should learn fault-oriented programming.
There are three nodes in this cluster with two roles. The salve may hang, and the master may hang. Let's first look at what happens when the salve node hangs.
slave failure
First, let one slave go down. Because two slave nodes are configured, one fails, and the entire service is not unavailable. Just deal with the failure as soon as possible and restore the slave. The experimental steps are as follows.
Now restart the failed slave node
/usr/local/bin/redis-server /etc/redis/6379.conf --replicaof
192.168.1.101 6379
Observe the master, it will print the following log information
2168:M 17 Apr 2020 13:38:16.282 * Replica 192.168.1.102:6379 asks for
synchronization 2168:M 17 Apr 2020 13:38:16.282 * Partial
resynchronization request from 192.168.1.102:6379 accepted. Sending
143 bytes of backlog starting from offset 1473.
You can see that only 2 lines of logs are printed. It means that the synchronization request from the 192.168.1.102:6379 (slave) node has been received, and the synchronization has been accepted, starting from the offset (offset) 1473, and a total of 143 bytes have been transmitted. This means that the reconnection of the slave does not trigger full synchronization, but incremental synchronization. The synchronized data is only the part of the data written in the master during the failure.
The above operation is the case when AOF is not turned on. If AOF is turned on, the situation is different. Next, let's operate the situation of opening AOF. The operation steps are the same as above, except that the AOF is turned on when the slave node restarts.
/usr/local/bin/redis-server /etc/redis/6379.conf --replicaof 192.168.1.101 6379 --appendonly yes
Observe the master node, you can see the following log
2168:M 17 Apr 2020 13:45:21.977 * Replica 192.168.1.102:6379 asks for
synchronization 2168:M 17 Apr 2020 13:45:21.977 * Full resync
requested by replica 192.168.1.102:6379 2168:M 17 Apr 2020
13:45:21.977 * Starting BGSAVE for SYNC with target: disk 2168:M 17
Apr 2020 13:45:21.978 * Background saving started by pid 2306 2306:C
17 Apr 2020 13:45:22.009 * DB saved on disk 2306:C 17 Apr 2020
13:45:22.010 * RDB: 8 MB of memory used by copy-on-write 2168:M 17 Apr
2020 13:45:22.111 * Background saving terminated with success 2168:M
17 Apr 2020 13:45:22.111 * Synchronization with replica
192.168.1.102:6379 succeeded
According to the log, if AOF is turned on when the slave node is restarted, full synchronization will be triggered. Even if AOF is turned on for all nodes at the beginning of the entire experiment, full synchronization will be triggered here.
The following is the slave log, which can also prove that the full synchronization is triggered.
2598:S 17 Apr 2020 13:45:21.967 * Ready to accept connections 2598:S
17 Apr 2020 13:45:21.968 * Connecting to MASTER 192.168.1.101:6379
2598:S 17 Apr 2020 13:45:21.968 * MASTER <-> REPLICA sync started
2598:S 17 Apr 2020 13:45:21.969 * Non blocking connect for SYNC fired
the event. 2598:S 17 Apr 2020 13:45:21.971 * Master replied to PING,
replication can continue… 2598:S 17 Apr 2020 13:45:21.973 * Partial
resynchronization not possible (no cached master) 2598:S 17 Apr 2020
13:45:21.977 * Full resync from master:
8b57ea32e3bada6e91d3f371123cb693df2eec8b:2235 2598:S 17 Apr 2020
13:45:22.107 * MASTER <-> REPLICA sync: receiving 271 bytes from
master 2598:S 17 Apr 2020 13:45:22.108 * MASTER <-> REPLICA sync:
Flushing old data 2598:S 17 Apr 2020 13:45:22.122 * MASTER <-> REPLICA
sync: Loading DB in memory 2598:S 17 Apr 2020 13:45:22.122 * MASTER
<-> REPLICA sync: Finished with success 2598:S 17 Apr 2020
13:45:22.125 * Background append only file rewriting started by pid
2602 2598:S 17 Apr 2020 13:45:22.178 * AOF rewrite child asks to stop
sending diffs. 2602:C 17 Apr 2020 13:45:22.179 * Parent agreed to stop
sending diffs. Finalizing AOF… 2602:C 17 Apr 2020 13:45:22.179 *
Concatenating 0.00 MB of AOF diff received from parent. 2602:C 17 Apr
2020 13:45:22.179 * SYNC append only file rewrite performed 2602:C 17
Apr 2020 13:45:22.180 * AOF rewrite: 4 MB of memory used by
copy-on-write 2598:S 17 Apr 2020 13:45:22.274 * Background AOF rewrite
terminated with success 2598:S 17 Apr 2020 13:45:22.274 * Residual
parent diff successfully flushed to the rewritten AOF (0.00 MB) 2598:S
17 Apr 2020 13:45:22.275 * Background AOF rewrite finished
successfully master故障
Since there is only one master node in this cluster, in case of a downtime, the entire service cannot write data, which is equivalent to the service unavailability. The savior appeared at this time. Oh, no, it was the Sentinel who appeared.
Redis Sentinel (Sentinel) is the official Redis high-availability solution for managing multiple Redis servers (instance). Sentinel has three main functions:
Monitoring: Sentinel will constantly check whether your master server and slave server are operating normally.
Notification: When a monitored Redis server has a problem, Sentinel can send notifications to administrators or other applications through the API.
Automatic failover (Automatic failover): When a master server fails to work normally, Sentinel will start an automatic failover operation. It will upgrade one of the failed master servers (slave) to a new master server ( master), and let the other slave servers of the failed master server to replicate the new master server; when the client tries to connect to the failed master server, the cluster will also return the address of the new master server to the client so that the cluster can use the new master server The server replaces the failed server.
If there is only one Sentinel instance to monitor the cluster, then Sentinel must also have a single point of failure, so multiple Sentinel instances are needed. The cluster structure after joining the sentinel is as follows: 
26379 is the default port of sentinel, and the three sentinels are placed on three nodes.
sentinel
There will be a sentinel.conf file in the decompression directory of the redis installation package. This is the sentinel configuration file. For convenience, copy it to the same directory as the redis configuration file
Copy sentinel configuration file
cp sentinel.conf /etc/redis/
Configure sentinel's configuration file
vim /etc/redis/sentinel.conf
There is only one area that needs to be changed, which is to specify which master the sentry should monitor, because the master can know which slave nodes are connected to it, so monitoring the master is enough. Note that the three sentinel nodes are all configured with the ip and port of the master
sentinel monitor
Tells Sentinel to monitor this master, and to consider it in O_DOWN
(Objectively Down) state only if at least sentinels agree.Note that whatever is the ODOWN quorum, a Sentinel will require to
be elected by the majority of the known Sentinels in order to start a
failover, so no failover can be performed in minority.Replicas are auto-discovered, so you don’t need to specify replicas in
any way. Sentinel itself will rewrite this configuration file adding
the replicas using additional configuration options. Also note that
the configuration file is rewritten when a replica is promoted to
masterNote: master name should not include special characters or spaces. The
valid charset is A-z 0-9 and the three characters “.-_”. sentinel
monitor mymaster 192.168.1.101 6379 2
The configuration instructs Sentinel to monitor a master server named mymaster. The IP address of this master server is 192.168.1.101 and the port number is 6379. The latter 2 means that the master server is judged to be invalid and requires at least 2 Sentinel's consent (as long as the number of Sentinels is agreed to not meet the standard, automatic fault migration will not be executed).
However, it should be noted that no matter how many Sentinel consents are set to determine a server failure, a Sentinel needs to obtain the support of the majority Sentinel in the system to initiate an automatic failure migration.
It is to better distinguish the majority from the minority. Generally, an odd number of sentinel instances are used to monitor the cluster.
After the configuration file is modified, three sentries are started. There are two ways to start the sentries: directly run redis-sentinel, run redis-server --sentinel
redis-server /etc/redis/sentinel.conf --sentinel
The first sentinel startup log is as follows
2873:X 17 Apr 2020 20:56:54.495 # WARNING: The TCP backlog setting of
511 cannot be enforced because /proc/sys/net/core/somaxconn is set to
the lower value of 128. 2873:X 17 Apr 2020 20:56:54.498 # Sentinel ID
is 643817dcf5ba6d53a737782a75706a62df869e33 2873:X 17 Apr 2020
20:56:54.498 # +monitor master mymaster 192.168.1.101 6379 quorum 2
2873:X 17 Apr 2020 20:56:54.500 * +slave slave 192.168.1.102:6379
192.168.1.102 6379 @ mymaster 192.168.1.101 6379 2873:X 17 Apr 2020 20:56:54.503 * +slave slave 192.168.1.103:6379 192.168.1.103 6379 @
mymaster 192.168.1.101 6379
You can see that the sentry printed out its ID, and also monitored 192.168.1.101 6379 (master) and two slave nodes
3031:X 17 Apr 2020 20:59:59.153 # WARNING: The TCP backlog setting of
511 cannot be enforced because /proc/sys/net/core/somaxconn is set to
the lower value of 128. 3031:X 17 Apr 2020 20:59:59.158 # Sentinel ID
is e784d728f7a813de688ea800a88bda6aca0512ff 3031:X 17 Apr 2020
20:59:59.158 # +monitor master mymaster 192.168.1.101 6379 quorum 2
3031:X 17 Apr 2020 20:59:59.164 * +slave slave 192.168.1.102:6379
192.168.1.102 6379 @ mymaster 192.168.1.101 6379 3031:X 17 Apr 2020 20:59:59.166 * +slave slave 192.168.1.103:6379 192.168.1.103 6379 @
mymaster 192.168.1.101 6379 3031:X 17 Apr 2020 21:00:00.115 *
+sentinel sentinel 643817dcf5ba6d53a737782a75706a62df869e33 192.168.1.101 26379 @ mymaster 192.168.1.101 6379
When the second sentry was activated, the same log was printed. In addition, an extra line about sentinel was printed. It can be seen that the sentinel ID printed out is the first sentinel. That is to say, when the sentry is monitoring the master, in addition to sending the slave node, it can also find other sentry monitoring the master node. Looking back at the log of the first sentry, one more line will be printed, which is the ID of the second sentry.
The three sentries are ready, and then let the master down.
After the master is down for 30 seconds, Sentinel thinks the server is down, which is specified by the parameter sentinel down-after-milliseconds
sentinel down-after-milliseconds
Number of milliseconds the master (or any attached replica or
sentinel) should
#be unreachable (as in, not acceptable reply to PING, continuously, for the specified period) in order to consider it in S_DOWN state
(Subjectively Down).
Default is 30 seconds.
More than half of Sentinel will vote after the master is down, and choose a master from the remaining two slaves. The three sentinel logs are as follows
2873:X 17 Apr 2020 21:02:57.687 # +sdown master mymaster 192.168.1.101
6379 2873:X 17 Apr 2020 21:02:57.765 # +new-epoch 1 2873:X 17 Apr 2020
21:02:57.766 # +vote-for-leader
a32bc56146695d9ebcbceaff2b0b8a5339c61a5b 1 2873:X 17 Apr 2020
21:02:58.326 # +config-update-from sentinel
a32bc56146695d9ebcbceaff2b0b8a5339c61a5b 192.168.1.103 26379 @
mymaster 192.168.1.101 6379 2873:X 17 Apr 2020 21:02:58.326 #
+switch-master mymaster 192.168.1.101 6379 192.168.1.103 6379 2873:X 17 Apr 2020 21:02:58.327 * +slave slave 192.168.1.102:6379
192.168.1.102 6379 @ mymaster 192.168.1.103 6379 2873:X 17 Apr 2020 21:02:58.327 * +slave slave 192.168.1.101:6379 192.168.1.101 6379 @
mymaster 192.168.1.103 6379 2873:X 17 Apr 2020 21:03:28.343 # +sdown
slave 192.168.1.101:6379 192.168.1.101 6379 @ mymaster 192.168.1.103
6379 3031:X 17 Apr 2020 21:02:57.686 # +sdown master mymaster
192.168.1.101 6379 3031:X 17 Apr 2020 21:02:57.743 # +new-epoch 1 3031:X 17 Apr 2020 21:02:57.745 # +vote-for-leader
a32bc56146695d9ebcbceaff2b0b8a5339c61a5b 1 3031:X 17 Apr 2020
21:02:57.776 # +odown master mymaster 192.168.1.101 6379 #quorum 3/2
3031:X 17 Apr 2020 21:02:57.776 # Next failover delay: I will not
start a failover before Fri Apr 17 21:08:57 2020 3031:X 17 Apr 2020
21:02:58.308 # +config-update-from sentinel
a32bc56146695d9ebcbceaff2b0b8a5339c61a5b 192.168.1.103 26379 @
mymaster 192.168.1.101 6379 3031:X 17 Apr 2020 21:02:58.308 #
+switch-master mymaster 192.168.1.101 6379 192.168.1.103 6379 3031:X 17 Apr 2020 21:02:58.309 * +slave slave 192.168.1.102:6379
192.168.1.102 6379 @ mymaster 192.168.1.103 6379 3031:X 17 Apr 2020 21:02:58.309 * +slave slave 192.168.1.101:6379 192.168.1.101 6379 @
mymaster 192.168.1.103 6379 3031:X 17 Apr 2020 21:03:28.352 # +sdown
slave 192.168.1.101:6379 192.168.1.101 6379 @ mymaster 192.168.1.103
6379 2833:X 17 Apr 2020 21:02:57.690 # +sdown master mymaster
192.168.1.101 6379 2833:X 17 Apr 2020 21:02:57.749 # +odown master mymaster 192.168.1.101 6379 #quorum 2/2 2833:X 17 Apr 2020
21:02:57.749 # +new-epoch 1 2833:X 17 Apr 2020 21:02:57.749 #
+try-failover master mymaster 192.168.1.101 6379 2833:X 17 Apr 2020 21:02:57.750 # +vote-for-leader
a32bc56146695d9ebcbceaff2b0b8a5339c61a5b 1 2833:X 17 Apr 2020
21:02:57.759 # 643817dcf5ba6d53a737782a75706a62df869e33 voted for
a32bc56146695d9ebcbceaff2b0b8a5339c61a5b 1 2833:X 17 Apr 2020
21:02:57.759 # e784d728f7a813de688ea800a88bda6aca0512ff voted for
a32bc56146695d9ebcbceaff2b0b8a5339c61a5b 1 2833:X 17 Apr 2020
21:02:57.841 # +elected-leader master mymaster 192.168.1.101 6379
2833:X 17 Apr 2020 21:02:57.841 # +failover-state-select-slave master
mymaster 192.168.1.101 6379 2833:X 17 Apr 2020 21:02:57.924 #
+selected-slave slave 192.168.1.103:6379 192.168.1.103 6379 @ mymaster 192.168.1.101 6379 2833:X 17 Apr 2020 21:02:57.925 * +failover-state-send-slaveof-noone slave 192.168.1.103:6379 192.168.1.103 6379 @ mymaster 192.168.1.101 6379 2833:X 17 Apr 2020 21:02:58.001 * +failover-state-wait-promotion slave 192.168.1.103:6379
192.168.1.103 6379 @ mymaster 192.168.1.101 6379 2833:X 17 Apr 2020 21:02:58.266 # +promoted-slave slave 192.168.1.103:6379 192.168.1.103
6379 @ mymaster 192.168.1.101 6379 2833:X 17 Apr 2020 21:02:58.266 #
+failover-state-reconf-slaves master mymaster 192.168.1.101 6379 2833:X 17 Apr 2020 21:02:58.317 * +slave-reconf-sent slave
192.168.1.102:6379 192.168.1.102 6379 @ mymaster 192.168.1.101 6379 2833:X 17 Apr 2020 21:02:58.817 # -odown master mymaster 192.168.1.101
6379 2833:X 17 Apr 2020 21:02:59.292 * +slave-reconf-inprog slave
192.168.1.102:6379 192.168.1.102 6379 @ mymaster 192.168.1.101 6379 2833:X 17 Apr 2020 21:02:59.292 * +slave-reconf-done slave
192.168.1.102:6379 192.168.1.102 6379 @ mymaster 192.168.1.101 6379 2833:X 17 Apr 2020 21:02:59.347 # +failover-end master mymaster
192.168.1.101 6379 2833:X 17 Apr 2020 21:02:59.347 # +switch-master mymaster 192.168.1.101 6379 192.168.1.103 6379 2833:X 17 Apr 2020
21:02:59.347 * +slave slave 192.168.1.102:6379 192.168.1.102 6379 @
mymaster 192.168.1.103 6379 2833:X 17 Apr 2020 21:02:59.347 * +slave
slave 192.168.1.101:6379 192.168.1.101 6379 @ mymaster 192.168.1.103
6379 2833:X 17 Apr 2020 21:03:29.355 # +sdown slave 192.168.1.101:6379
192.168.1.101 6379 @ mymaster 192.168.1.103 6379
You can see the general process from the log
The three sentinels all sent the master down, set its status to odown to
open a round of voting, and selected the new master as 192.168.1.103:6379
sentinel to update the configuration file
192.168.1.103:6379 becomes the new master, and the failover is complete.
As can be seen from the last few lines of logs, the current master is 192.168.1.103 6379, and the slave is 192.168.1.102:6379 and 192.168.1.101:6379, and 192.168.1.101 :6379 is the slave in the down state.
There are two different concepts about down in Redis Sentinel:
Subjectively Down (Subjectively Down, SDOWN for short) refers to the offline judgment made by a single Sentinel instance to the server.
Objectively Down (ODOWN) refers to the server offline after multiple Sentinel instances make SDOWN judgments on the same server and communicate with each other through the SENTINEL is-master-down-by-addr command judgment. (A Sentinel can send the Sentinel is-master-down-by-addr command to the other Sentinel to ask whether the other party thinks a given server is offline.)
Check the sentinel configuration file and find that the node monitored by the sentinel has become the new master
[root@localhost redis-5.0.8]> cat /etc/redis/sentinel.conf |grep
“sentinel monitor mymaster” sentinel monitor mymaster 192.168.1.103
6379 2
The previous master failed, but now the failure is repaired and ready to restart. Restart the original 192.168.1.101 (master), will it become a slave willingly, or will it regain its master status?
If you want to know what happened, please listen to the next breakdown.
Look at the sentry log to know that the sentry will print the following log
3031:X 17 Apr 2020 21:05:32.297 * +convert-to-slave slave
192.168.1.101:6379 192.168.1.101 6379 @ mymaster 192.168.1.103 6379
It is to turn 192.168.1.101:6379 into an available slave, so even if the original master restarts, it will not grab the master status.
At this point, the high-availability redis cluster based on the sentinel has been completed.
supplement
Here is a summary of the theories related to master-slave replication. When the slave node first follows the master, it will send a sync request to synchronize. Request synchronization is completed by the psync [runId] [offset] command after Redis 2.8. The psync command supports both full replication and incremental replication. After Redis4.0, psync has been optimized again.
runId: is the unique identifier generated by each redis node when it starts. After each redis restart, the runId will also change.
Offset: is the offset of the replication. The master and slave will record the replication offset of themselves and each other, if they are inconsistent , Indicating that you need to continue syncing
In addition, the master node also maintains a buffer queue (replication backlog buffer, the default size of the replication backlog buffer is 1M, and the parameter repl-backing-size is set). When the slave is replicating the master, if a network exception occurs, the command is lost. The slave will request the master to resend the lost command data. If the master's replication backlog buffer memory sends this part of the data directly to the slave, so that the consistency of the master-slave node replication can be maintained.
However, the redis 2.8 version of psync still has two problems that cannot be solved: full replication is triggered when redis restarts, and after failover, the slave follows the new master to trigger full synchronization.
These two problems have been resolved in the redis4.0 version of psync. Mainly through two replication ids (master_replid and master_replid2) to achieve
These information can be queried through the info replication command.
This is the information of the master node.
192.168.1.103:6379> info replication
Replication
role:master connected_slaves:2
slave0:ip=192.168.1.101,port=6379,state=online,offset=98,lag=0
slave1:ip=192.168.1.102,port=6379,state=online,offset=98,lag=0
master_replid:8b1d6db7a9e63c0360ffed0ec6d3a51199f08f2b
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:98 second_repl_offset:-1 repl_backlog_active:1
repl_backlog_size:1048576 repl_backlog_first_byte_offset:1
repl_backlog_histlen:98 这是slave节点的信息
192.168.1.101:6379> info replication
Replication
role:slave master_host:192.168.1.103 master_port:6379
master_link_status:up master_last_io_seconds_ago:3
master_sync_in_progress:0 slave_repl_offset:5334 slave_priority:100
slave_read_only:1 connected_slaves:0
master_replid:8b1d6db7a9e63c0360ffed0ec6d3a51199f08f2b
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:5334 second_repl_offset:-1 repl_backlog_active:1
repl_backlog_size:1048576 repl_backlog_first_byte_offset:1
repl_backlog_histlen:5334
to sum up
This article explains the practice and some principles of redis master-slave replication in an interspersed way, which may make it look a little messy. The reason for the interspersed approach is to allow readers to link theory and practice to form a complete knowledge system, not just fragmentary knowledge points.
Friends who only care about the experiment can skip the theoretical part of the article first, and the effect of the experiment will not be affected.