To put it simply, a sentinel is a whistleblower who inspects and monitors whether the master host in the background is faulty. If it fails, it will automatically convert a certain slave database to a new master database according to the number of votes, and continue to serve the outside world.

effect

Sentinel has the following four main functions:

Master-slave monitoring

Monitor whether the master-slave redis library is running normally

notification

Sentry can send failover results to clients

failover

If the master is abnormal, a master-slave switch will be performed, and one of the slaves will be used as the new master

configuration center

The client obtains the master node address of the current Redis service by connecting to the sentinel

To sum up in one sentence, the main functions of Sentinels are as follows:

1. Monitor redis running status, including master and slave

2. When the master is down, it can automatically switch the slave to the new master

How to use Sentinel (case demonstration + practical steps)

Redis sentinel architecture explained in advance

Our configuration to verify the role of redis sentry is as follows:

3 Sentinels

Automatically monitor and maintain clusters, do not store data, just monitor

1 master 2 slaves

For data reading and storage

Create or copy sentinel.conf in the /myredis directory

First look at the contents of the default sentinel.conf file in the redis decompression directory

Key parameter configuration

Sentinel file parameters (leader)
bind service listening address, used for client connection (local address by default)
daemonize whether to run as background daemon
protected-mode security protection mode
port port
logfile log file path
pidfile pid log path
dir working directory

sentinel monitor <master> <redis-port> <quorum>

Setting the master quorm to be monitored means that at least a few sentries agree to objectively go offline and agree to the statutory number of votes for fault migration

We know that the network is unreliable. Sometimes a sentinel will mistakenly think that a master redis has died due to network congestion. In a sentinel cluster environment, multiple sentinels need to communicate with each other to confirm whether a master is really dead . The parameter quorum is a basis for objective offline, which means that at least quorum sentinels believe that the master is faulty, and then the master will be offline and failover. Because sometimes, a sentinel node may not be able to connect to the master due to its own network reasons, and the master is not faulty at this time, so it requires multiple sentinels to agree that the master has a problem before proceeding to the next step operation, which ensures fairness and high availability.

sentinel auth-pass <master-name> <password> // Connect to master through password

//Connect to the host through a password. It should be noted here that even if it is a configuration file in the host, we still need to configure this, because after our host hangs up and restarts, it is no longer the host by default, and it is recognized as a slave. , at this time he also needs to connect to the new host

Common file configuration for this case

We configure the following sentinel configuration files

sentinel26381.conf

bind 0.0.0.0
daemonize yes
protected-mode no
port 26379
logfile "/myredis/sentinel26379.log"
pidfile /var/run/redis-sentinel26379.pid
dir /myredis
sentinel monitor mymaster 192.168.111.169 6379 2
sentinel auth-pass mymaster 111111

sentinel26380.conf

bind 0.0.0.0
daemonize yes
protected-mode no
port 26380
logfile "/myredis/sentinel26380.log"
pidfile /var/run/redis-sentinel26380.pid
dir "/myredis"
sentinel monitor mymaster 192.168.111.169 6379 2
sentinel auth-pass mymaster 111111

sentinel26379.conf

bind 0.0.0.0
daemonize yes
protected-mode no
port 26381
logfile "/myredis/sentinel26381.log"
pidfile /var/run/redis-sentinel26381.pid
dir "/myredis"
sentinel monitor mymaster 192.168.111.169 6379 2
sentinel auth-pass mymaster 111111

About other configurations in the sentinel file

sentinel down-after-milliseconds <master-name> <milliseconds>：

指定多少毫秒之后，主节点没有应答哨兵，此时哨兵主观上认为主节点下线

 

sentinel parallel-syncs <master-name> <nums>：

表示允许并行同步的slave个数，当Master挂了后，哨兵会选出新的Master，此时，剩余的slave会向新的master发起同步数据

 

sentinel failover-timeout <master-name> <milliseconds>：

故障转移的超时时间，进行故障转移时，如果超过设置的毫秒，表示故障转移失败

 

sentinel notification-script <master-name> <script-path> ：

配置当某一事件发生时所需要执行的脚本

 

sentinel client-reconfig-script <master-name> <script-path>：

客户端重新配置主节点参数脚本

Start a redis instance with one master and two slaves, configure sentinels, and demonstrate normal master-slave replication

1	Create a new redis6379.conf configuration file on the 169 machine. To match this case, please set the access password of the masterauth item to 111111. Otherwise, an error may be reported later master_link_status:down
2	Create a new redis6380.conf configuration file on the 172 machine, set the replicaof <masterip> <masterport>
3	Create a new redis6381.conf configuration file on the 173 machine, set the replicaof <masterip> <masterport>

On the basis of one master and two slaves configured in the previous chapter, the host redis6379.conf needs to be changed

redis6379.conf

6379 may become a slave in the future, you need to set a password to access the new host, please set the access password of the masterauth item to 111111

redis 6380.conf

The specific IP address and password should be modified according to your local real situation.

redis 6381.conf

Start one master and two slaves

6379.conf
    redis-server /myredis/redis6379.conf 
    redis-cli -a 111111 -p6379
 
6380.conf
    redis-server /myredis/redis6380.conf 
    redis-cli -a 111111 -p 6380
 
6381.conf
    redis-server /myredis/redis6381.conf 
    redis-cli -a 111111-p 6381

start sentinel

redis-sentinel sentinel26379.conf --sentinel
redis-sentinel sentinel26380.conf --sentinel
redis-sentinel sentinel26381.conf --sentinel

View sentinel information

Check the master-slave replication information and find that everything is normal

The host is down, check the log

Shut down the redis server manually, and the simulated host hangs up

We check the changes of the slave after the master hangs up

Problems that may arise:

A certain explanation of Broken pipe:

Know broken pipe	Pipe means pipeline, and inside the pipeline is a data stream, usually data read from a file or network socket. When the pipe is suddenly closed from the other end, the data will be interrupted suddenly, which is broken. For the socket, Probably the network was unplugged or the process on the other end crashed
Solve the problem	In fact, when the exception occurs, it does not have much impact on the server. This error may be caused by a client that terminated the process suddenly
SummaryBroken Pipe	This exception is that the client read timed out and closed the connection. At this time, the server sends the client A broken pipe exception occurs when a disconnected connection writes data!

To sum up, the hangup of the master has a transient impact on the slave. Don’t panic when encountering this kind of problem, just re-enter the same command again.

Continue to view copy information

6380

6381

We found that 6381 became the host

Compare configuration files

old master vim redis6379.conf

redis6379 offline

vim redis6380.conf

new master vim redis6381.conf

we come to the conclusion:

The sentinel26379.log, sentinel26380.log, and sentinel26381.log in this article will be dynamically changed during operation

In the master-slave switch, the content in the redis###.conf file will be changed automatically

In addition, multiple masters can be monitored in Sentinel, one per line

Sentinel's operating process and selection principle

When a master in a master-slave configuration fails, sentinel can elect a new master to take over the work of the original master, and other redis servers in the master-slave configuration automatically point to the new master to synchronize data. It is generally recommended to use an odd number of sentinels to prevent a certain sentinel from being unable to connect to the master and causing an accidental switch.

Subjective offline and objective offline

SDOWN subjective offline

1. SDOWN is the state of the master subjectively detected by a single sentinel. From the perspective of sentinel, if no legal reply is received within a certain period of time after the PING heartbeat is sent, the SDOWN condition 2 is met
. The down-after-milliseconds in the sentinel configuration file sets the length of time for judging subjective offline

The so-called subjectively down (Subjectively Down, referred to as SDOWN) refers to the offline judgment made by a single Sentinel instance to the server, that is, a single sentinel thinks that a service is offline (it may be that the subscription cannot be received, the network between them is not connected, etc. and other reasons). Subjective offline means that if the server does not respond to the PING command or returns an error message within the given number of milliseconds in [sentinel down-after-milliseconds], then the Sentinel will subjectively (unilaterally) consider the master unavailable Yes, o(╥﹏╥)o

sentinel down-after-milliseconds <masterName> <timeout>

Indicates the interval time for the master to be considered invalid by the current sentinel instance. This configuration is actually a basis for subjective offline

How long the master has not returned valid information to Sentine, it is considered that the master is subjectively offline. That is to say, if redis-servevr has not been contacted for a long time, it is considered that the redis-server has entered the SDOWN state.

ODOWN objective offline

ODOWN requires a certain number of sentinels, and multiple sentinels can reach a consensus to consider that a master is objectively down

The meaning of the four parameters:

masterName is a distinguishing identifier for a master+slave combination (a set of sentinels can monitor multiple groups of master+slave combinations)

The parameter quorum is a basis for objective offline , the quorum/quorum of votes

It means that at least quorum sentinels think that the master is faulty before going offline and failover the master. Because sometimes, a sentinel node may not be able to connect to the master due to its own network, but the master is not faulty at this time, so this requires multiple sentinels to agree that the master has a problem before proceeding to the next step. This ensures fairness and high availability.

Election leader sentinel (choose the king of soldiers from the sentinel)

When the master node is judged to be objectively offline, each sentinel node will negotiate, and the county will elect a leader sentinel node and the leader node will perform failover (fault migration)

The Raft algorithm selects the leader node

All Sentinels monitoring the master node may be elected as the leader. The algorithm used for the election is the Raft algorithm; the basic idea of the Raft algorithm is first-come-first-served: that is, in a round of elections, Sentinel A sends a message to B to become the leader. , if B has not agreed to other sentinels, it will agree to A as the leader

log changes

The king of soldiers turns on the failover and elects a new master

new lord enthroned

A slave candidate becomes the new master to select a new master rule . In the redis.conf file
under the condition that the remaining slave nodes are healthy, the slave node with the highest priority slave-priority or replica-priority (the smaller the number, the higher the priority)

Copy the slave node with the largest offset position (the one with the largest number of records in the log) and
the slave node with the smallest Run ID, that is, in lexicographical order, ASCII code

All the officials bowed their heads, once the emperor and the courtiers, re-recognized the boss

Execute the slaveof no one command to make the selected slave node the new master node, and use the slaveof command to make other nodes its slave nodes. The Sentinel leader
will execute the slaveof no one operation on the elected new master and promote it to the master node
Sentinel The leader sends commands to other slaves, making the remaining slaves the slaves of the new master node

The old master worships, and the old master has to be persuaded when he comes back

Set the old master that has been offline before as the slave node of the newly elected new master. When the old master comes online again, it will become the slave node of the new master. Sentinel leader will degrade the original master to slave and resume normal work.

Suggestions for using Sentinel

The number of sentinel nodes should be multiple, and the sentry itself should be clustered to ensure high availability

The number of sentinel nodes should be an odd number

The configuration of each sentinel node should be consistent

If the sentinel node is deployed in a container such as Docker, pay attention to the correct port mapping

Sentinel cluster + master-slave replication does not guarantee zero data loss

[redis basics] Sentinel

concept