What is it?
Whistleblowers inspect and monitor whether the master host in the background is faulty. If it fails, according to the number of votes\textcolor{red}{votes} votes, a certain slave library will be automatically converted to a new master library to continue external services
Function: Commonly known as unattended operation and maintenance
What can you do?
Master-slave monitoring: monitor whether the master-slave redis library is running normally
Message notification: Sentinel can send the result of failover to the client
Failover: If the master is abnormal, a master-slave switch will be performed, and one of the slaves will be used as the new master
Configuration center: the client obtains the address of the master node of the current Redis service by connecting to the sentinel
The case demonstrates
the Redis Sentinel architecture, and the premise explains
3 sentinels: automatic monitoring and maintenance of the cluster, no data storage, just whistlers
1 master 2 slaves: for data reading and storage
Operation steps
Create or copy the sentinel.conf file in the myredis directory Press, the name must never be wrong
cp -a /opt/redis-7.0.0/sentinel.conf /myredis/
First look at the contents of the default sentinel.conf file in the /opt directory
cat /opt/redis-7.0.0/sentinel.conf
Description of important parameter items
bind: service listening address, used for client connection, default local address
daemonize: Whether to run as a background daemon
protected-model: security protection model
port: port
logfile: log file path
pidfile: pid file path
dir: working directory
sentinel monitor
Set the master server to monitor
Quorum indicates that at least a few sentinels agree to objectively go offline and agree to the statutory number of votes for fault migration
What does the quorum at the end of the line mean? Quorum: Confirm the minimum number of sentinels that are objectively offline.
The network is unreliable. Sometimes a sentinel will mistakenly think that a master redis is dead due to network congestion. In a sentinel cluster environment, multiple sentinels need to communicate with each other to confirm a certain number of sentinels. Whether a master is really dead, the quorum parameter is a basis for objective offline, which means that at least quorum sentinels think that the master is faulty, and then the master will be offline and failover. Because sometimes, a sentinel node may not be able to connect to the master due to its own network reasons, and the master is not faulty at this time, so this requires multiple sentinels to agree that the master has a problem before proceeding to the next step operation, which ensures fairness and high availability.
sentinel auth-pass
The master has set a password, the password to connect to the master service
Other
sentinel general configuration and master-slave configuration
sentinel26379.conf
bind 0.0.0.0
daemonize yes
protected-mode no
port 26379
logfile "/myredis/sentinel26379.log"
pidfile /var/run/redis-sentinel26379.pid
dir /myredis
sentinel monitor mymaster 192.168.111.169 6379 2
sentinel auth-pass mymaster 111111
sentinel26380.conf
bind 0.0.0.0
daemonize yes
protected-mode no
port 26380
logfile "/myredis/sentinel26380.log"
pidfile /var/run/redis-sentinel26380.pid
dir "/myredis"
sentinel monitor mymaster 192.168.111.169 6379 2
sentinel auth-pass mymaster 111111
sentinel26381.conf
bind 0.0.0.0
daemonize yes
protected-mode no
port 26381
logfile "/myredis/sentinel26381.log"
pidfile /var/run/redis-sentinel26381.pid
dir "/myredis"
sentinel monitor mymaster 192.168.111.169 6379 2
sentinel auth-pass mymaster 111111
Master host configuration file description
Theoretically, the sentinel configuration file should be deployed on different servers to form a cluster, but this demonstration puts it on a single machine.
First, start three redis instances, one master and two slaves, to test normal master-slave replication.
Architecture description
Host 6379 configuration modification
6379 may become a slave in the future, and you need to set a password to access the new host. Please set the access password of the masterauth item to 111111, otherwise an error may be reported in the future master_link_status:down
redis6380.conf, redis6381.conf configuration file modification
The specific IP address and password should be modified according to your local real situation.
3 different virtual machine instances, start three real machine instances and connect
redis-server redis6379.conf
redis-server redis6380.conf
redis-server redis6381.conf
redis-cli -a 123456 -p 6379
redis-cli -a 123456 -p 6380
redis-cli -a 123456 -p 6381
Start 3 more sentinel monitors and test the master-slave replication again
redis-server sentinel26379.conf --sentinel
redis-server sentinel26380.conf --sentinel
redis-server sentinel26381.conf --sentinel
's original master is down
We manually shut down the 6379 server by ourselves, and the simulated master hangs up
problem thinking
Is the data of the two slave machines OK ? Will the master machine that was down before the
new master be selected from the remaining two machines and restarted? Who will be the new boss?
Will there be a dual master conflict?
reveal the answer
The data of the two slave machines is OK
Will vote for a new master host
Who is the master, only for this case
In this case, 6381 was elected as the new master, and the master succeeded
After restarting 6379, it will downgrade from the original master to slave
6380 is still a slave, just changed to a new boss 6381 (from following 6379 to following 6381)
Compare configuration files
The redis6379.conf file of the old master
The redis6381.conf file of the new master
Conclusion
The content of the file will be dynamically changed by sentinel during operation
. After the master-slave switch, the contents of master_redis.conf, slave_redis.conf, and sentinel.conf will change\change, that is, there will be one more line of slaveof configuration in master_redis.conf, The host upgraded to master will remove the original slaveof configuration, and the monitoring target of sentinel.conf will be replaced accordingly. Sentinel
operation process and election principle
When a master-slave configuration fails, sentinel can elect a new master for automatic To take over the work of the original master, other redis servers in the master-slave configuration automatically point to the new master to synchronize data. Generally, it is recommended to use an odd number of sentinels to prevent a certain sentinel from being unable to connect to the master and cause a wrong switch
Operation process, failover
Three sentries monitor one master and two slaves,
SDown subjectively goes offline during normal operation (Subjectively Down)
SDOWN (subjectively unavailable) is the state of the master subjectively detected by a single sentinel. From the perspective of sentinel, if a legal reply is not received within a certain period of time after sending a PING heartbeat, SDOWN is reached. condition.
The down-after-milliseconds in the sentinel configuration file sets the length of time for judging subjective offline
illustrate
ODown Objectively Down
ODOWN requires a certain number of sentinels, and multiple sentinels can reach a consensus to consider that a master is objectively down
illustrate
Elect the leader sentinel (select the soldier king from the sentinel)
When the master node is judged to be objectively offline, each sentinel node will negotiate and first elect a leader sentinel node (soldier king) \textcolor{red}{\large leader sentinel node (soldier king)}leader sentinel node ( Soldier king) and the leader, that is, the elected soldier king, performs failover (failover).
Interpretation and Analysis of Sentinel Log Files
Sentinel leader, how is the king of soldiers elected? -> The Raft algorithm
monitors all the sentries of the master node that may be elected as the leader. The algorithm used in the election is the Raft algorithm; Send an application to become a leader. If B has not agreed to other sentinels, it will agree to A to become the leader.
The king of soldiers starts to promote the failover process and elects a new master
new lord enthroned
A Slave is selected as the new Master
The rules for selecting a new master. Under the premise that the remaining Slave nodes are healthy, they will be elected according to the rules shown in the figure below.
In the redis.conf file, the slave node with the highest priority slave-priority or replica-priority (the smaller the number, the higher the priority)
replicates The slave node with the largest offset position (that is, when the master is not down, the copied data is more than other slaves) the slave
node with the smallest Run ID, in lexicographical order, ASCII code
new lord enthroned
Once the emperor and the courtiers, change the pier and worship again}
Execute the slaveof no one command to make the selected slave node become the new master node, and use the slaveof command to make other nodes become its slave nodes. The sentinel leader
will execute slaveof on the elected new master On one operation, promote it to the master node
sentinel leader to send commands to other slaves, so that the remaining slaves become slaves of the new master node
old lord worship
When the old master comes back, he admits that he will be demoted to a slave.
After the old master comes back online, he will set it as the slave node of the newly elected master.
The sentinel leader will degrade the original master to a slave and resume normal work.
Suggestions for using
Sentinel The number of Sentinel nodes should be multiple, and Sentinel itself should be clustered to ensure high availability The
number of Sentinel nodes should be an odd number The
configuration of each Sentinel node should be consistent
If the Sentinel node is deployed in a container such as Docker, especially pay attention to the correct port Mapping
sentinel cluster + master-slave replication does not guarantee zero data loss, so you need to use a cluster