Article directory
1. Sentinel mode
1. Introduction to Sentry
1.1 Sentinel concept
First, let's look at a business scenario: what should we do if the redis master is down?
At this time, we may need to re-elect a new master from a bunch of slaves. What is the operation process like? What problems will arise here?
To realize these functions, we need the redis sentinel, so what is the sentinel?
sentinel
Sentinel is a distributed system used to monitor each server in the master-slave structure . When a failure occurs, a new master is selected through a voting mechanism and all slaves are connected to the new master.
1.2 Sentinel role
The role of the sentinel:
-
Monitoring: monitoring master and slave
Constantly check whether the master and slave are running normally
Master survival detection, master and slave operation detection
-
Notification (reminder): When there is a problem with the monitored server, send a notification to others (sentry room, client)
-
Automatic failover: disconnect the master from the slave, select a slave as the master, connect other slaves to the new master, and inform the client of the new server address
Note: Sentinel is also a redis server, but it does not provide data-related services. Usually, the number of sentinels is configured as an odd number
2. Enable Sentry
Configure Sentry
-
Configure a one-to-two master-slave structure (use the previous method to start)
-
Configure three sentinels (same configuration, different ports), see sentinel.conf
1: Set the main server information monitored by the sentinel, sentinel_number indicates the number of sentinels participating in the voting
sentinel monitor master_name master_host master_port sentinel_number
2: Set to determine the length of server downtime, this setting controls whether to perform master-slave switching
sentinel down-after-milliseconds master_name million_seconds
3: When setting the maximum timeout for failover
sentinel failover-timeout master_name million_seconds
4: After setting the master-slave switch, the number of slaves performing data synchronization at the same time, the larger the value, the higher the network resources required, and the smaller the value, the longer the synchronization time
sentinel parallel-syncs master_name sync_slave_number
- start sentinel
redis-sentinel filename
3. Working principle of Sentinel
Sentinel goes through three stages in the process of master-slave switching
- monitor
- notify
- failover
3.1 Monitoring
Used to synchronize the status information of each node
-
Get the status of each sentinel (whether online or not)
-
Get the status of the master
master属性
prunid
prole:master
各个slave的详细信息
- Get the status of all slaves (according to the slave information in the master)
slave属性
prunid
prole:slave
pmaster_host、master_port
poffset
Its internal working principle is as follows:
3.2 Notification
In the notification phase, sentinel needs to continuously obtain master/slave information, and then share it among sentinels. The specific process is as follows:
3.3 Failover
When the master is down, how does sentinel know and judge that the master is really down? Let's look at the specific operation process
When sentinel determines that the master is offline, it needs to decide to replace the master. Which sentinel will do this? At this time, there will be an election between sentinels, as shown in the following figure:
Everyone has one vote in the election, and everyone wants to be the person who handles the accident, so what should we do? Everyone started to fight, so everyone will send out an instruction to tell everyone on the intranet that I want to be an elector. For example, if sentinel1 and sentinel4 have issued this election instruction, then sentinel2 can receive both sentinel1 and sentinel1. For sentinel4, after receiving their application, sentinel2 will vote for one of them, who will he vote for? I will vote for whoever comes first, assuming sentinel1 comes first, so this vote is given to sentinel1. Then for the past, now sentinel1 has got one vote. According to such a form, there will be an election result in the end. The one who gets more votes in the corresponding election will naturally become the one who handles the accident. It should be noted that there may be failures in this process, that is, if there is no selection after one round of elections, then the second round and the third round will continue until the election is completed.
The next step is to select a new master from the slave by the sentinel who won the election. What is the process like?
First of all, it has a principle of selecting alternative masters in the server list
-
offline OUT
-
slow response OUT
-
OUT that has been disconnected from the original master for a long time
-
priority principle
priority
offset
runid
After selecting a new master, send commands (sentinel) to other slaves:
-
Send slaveof no one to the new master
-
Send slaveof new masterIP port to other slaves
Summary : Failover Phases
- Problem discovery, subjective logout and objective logout
- campaign manager
- Select new master
- The new master takes office, other slaves switch to the master, and the original master connects as a slave after recovery