1. Redis Sentry

Redis provides a Sentinel mechanism to achieve automatic failure recovery of the master-slave cluster.

1.1. Sentinel principle

Cluster structure and role

The structure of Sentinel is shown in the figure:

The role of the sentinel is as follows:

Monitoring : Sentinel constantly checks that your master and slave are working as expected

Automatic failure recovery : If the master fails, Sentinel will promote a slave to master. When the faulty instance recovers, the new master is also the main one

Notification : Sentinel acts as the service discovery source for the Redis client, and when the cluster fails over, it will push the latest information to the Redis client

Cluster Monitoring Principle

Sentinel monitors the service status based on the heartbeat mechanism, and sends a ping command to each instance of the cluster every 1 second:

Subjective offline: If a sentinel node finds that an instance does not respond within the specified time, it considers the instance to be offline subjectively .

Objective offline: If more than the specified number (quorum) of sentinels think that the instance is subjectively offline, the instance will be objectively offline . The quorum value is preferably more than half of the number of Sentinel instances.

Principles of Cluster Failure Recovery

Once a master failure is found, sentinel needs to select one of the salves as the new master. The selection basis is as follows:

First, it will judge the length of disconnection between the slave node and the master node. If it exceeds the specified value (down-after-milliseconds * 10), the slave node will be excluded

Then judge the slave-priority value of the slave node, the smaller the priority, the higher the priority, if it is 0, it will never participate in the election

If the slave-prority is the same, judge the offset value of the slave node. The larger the value, the newer the data and the higher the priority

The last is to judge the size of the running id of the slave node, the smaller the priority, the higher the priority.

When a new master is elected, how to implement the switch?

The process is as follows:

Sentinel sends the slaveof no one command to the candidate slave1 node to make the node the master

Sentinel sends the slaveof 192.168.150.101 7002 command to all other slaves to make these slaves become slave nodes of the new master and start synchronizing data from the new master.

Finally, sentinel marks the failed node as a slave, and when the failed node recovers, it will automatically become the slave node of the new master

summary

What are the three functions of Sentinel?

monitor
failover
notify

How does Sentinel judge whether a redis instance is healthy?

Send a ping command every 1 second, if there is no communication for a certain period of time, it is considered as a subjective offline
If most of the sentinels think that the instance is offline subjectively, it is determined that the service is offline

What are the failover steps?

First select a slave as the new master, execute slaveof no one
Then let all nodes execute slaveof new master
Modify the faulty node configuration, add slaveof new master

1.2. Build a sentinel cluster

cluster structure

Here we build a Sentinel cluster formed by three nodes to supervise the previous Redis master-slave cluster. As shown in the picture:

The information of the three sentinel instances is as follows:

node IP PORT

s1 192.168.xxx.xxx 27001

s2 192.168.xxx.xxx 27002

s3 192.168.xxx.xxx 27003

node	IP	PORT
s1	192.168.xxx.xxx	27001
s2	192.168.xxx.xxx	27002
s3	192.168.xxx.xxx	27003

Prepare instance and configuration

To start three instances on the same virtual machine, three different configuration files and directories must be prepared. The directory where the configuration files are located is also the working directory.

We create three folders named s1, s2, s3:
# 进入/tmp目录
cd /tmp
# 创建目录
mkdir s1 s2 s3
Then we create a sentinel.conf file in the s1 directory and add the following content:
port 27001
sentinel announce-ip 192.168.150.101
sentinel monitor mymaster 192.168.150.101 7001 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 60000
dir "/tmp/s1"
Interpretation:

port 27001: is the port of the current sentinel instance

sentinel monitor mymaster 192.168.150.101 7001 2: Specify master node information

mymaster: master node name, user-defined, arbitrary write

192.168.150.101 7001: the ip and port of the master node

2: quorum value when electing master

Then copy the s1/sentinel.conf file to the s2 and s3 directories (execute the following commands in the /tmp directory):
# 方式一：逐个拷贝
cp s1/sentinel.conf s2
cp s1/sentinel.conf s3
# 方式二：管道组合命令，一键拷贝
echo s2 s3 | xargs -t -n 1 cp s1/sentinel.conf
Modify the configuration files in the two folders s2 and s3, and change the ports to 27002 and 27003 respectively:
sed -i -e 's/27001/27002/g' -e 's/s1/s2/g' s2/sentinel.conf
sed -i -e 's/27001/27003/g' -e 's/s1/s3/g' s3/sentinel.conf

start up

In order to view the logs conveniently, we open three ssh windows, start three redis instances respectively, and start the command:
# 第1个
redis-sentinel s1/sentinel.conf
# 第2个
redis-sentinel s2/sentinel.conf
# 第3个
redis-sentinel s3/sentinel.conf

test

Try to shut down the master node 7001, check the sentinel log:

Check the log of 7003

Check the log of 7002:

1.3、RedisTemplate

In the Redis master-slave cluster under the supervision of Sentinel cluster, its nodes will change due to automatic failover, and the Redis client must perceive this change and update the connection information in time. The underlying layer of Spring's RedisTemplate uses lettuce to realize node perception and automatic switching.

Next, we implement the RedisTemplate integration sentinel mechanism through a test.

Introduce dependencies into the project

Introduce dependencies in the project's pom file:
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>

Configure Redis address

Then specify the sentinel related information of redis in the configuration file application.yml:
spring:
  redis:
    sentinel:
      master: mymaster
      nodes:
        - 192.168.150.101:27001
        - 192.168.150.101:27002
        - 192.168.150.101:27003

Configure read-write separation

In the startup class of the project, add a new bean:
@Bean
public LettuceClientConfigurationBuilderCustomizer clientConfigurationBuilderCustomizer(){
    return clientConfigurationBuilder -> clientConfigurationBuilder.readFrom(ReadFrom.REPLICA_PREFERRED);
}
This bean is configured with read and write strategies, including four types:

MASTER: read from the master node

MASTER_PREFERRED: Read from the master node first, read the replica when the master is unavailable

REPLICA: read from the slave (replica) node

REPLICA _PREFERRED: Read from the slave (replica) node first, all slaves are unavailable to read the master

2. Redis Fragmentation Cluster

2.1. Build a shard cluster

Master-slave and sentry can solve the problem of high availability and high concurrent reading. But there are still two unresolved issues:

Mass data storage problem
The problem of high concurrent writing

Using fragmented clusters can solve the above problems, as shown in the figure:

Sharded cluster features:

There are multiple masters in the cluster, and each master saves different data
Each master can have multiple slave nodes
The master monitors each other's health status through ping
Client requests can access any node in the cluster and will eventually be forwarded to the correct node

cluster structure

Sharded clusters require a large number of nodes. Here we build a minimal sharded cluster, including 3 master nodes, and each master contains a slave node. The structure is as follows:

Here we will open 6 redis instances in the same virtual machine to simulate a fragmented cluster. The information is as follows:

IP PORT Role

192.168.xxx.xxx 7001 master

192.168.xxx.xxx 7002 master

192.168.xxx.xxx 7003 master

192.168.xxx.xxx 8001 slave

192.168.xxx.xxx 8002 slave

192.168.xxx.xxx 8003 slave

IP	PORT	Role
192.168.xxx.xxx	7001	master
192.168.xxx.xxx	7002	master
192.168.xxx.xxx	7003	master
192.168.xxx.xxx	8001	slave
192.168.xxx.xxx	8002	slave
192.168.xxx.xxx	8003	slave

Prepare instance and configuration

Create 7001, 7002, 7003, 8001, 8002, 8003 directories:

# 进入/tmp目录
cd /tmp
# 删除旧的，避免配置干扰
rm -rf 7001 7002 7003
# 创建目录
mkdir 7001 7002 7003 8001 8002 8003

Prepare a new redis.conf file under /tmp with the following content:

port 6379
# 开启集群功能
cluster-enabled yes
# 集群的配置文件名称，不需要我们创建，由redis自己维护
cluster-config-file /tmp/6379/nodes.conf
# 节点心跳失败的超时时间
cluster-node-timeout 5000
# 持久化文件存放目录
dir /tmp/6379
# 绑定地址
bind 0.0.0.0
# 让redis后台运行
daemonize yes
# 注册的实例ip
replica-announce-ip 192.168.150.101
# 保护模式
protected-mode no
# 数据库数量
databases 1
# 日志
logfile /tmp/6379/run.log

Copy this file into each directory:

# 进入/tmp目录
cd /tmp
# 执行拷贝
echo 7001 7002 7003 8001 8002 8003 | xargs -t -n 1 cp redis.conf

Modify redis.conf in each directory, and modify 6379 to be consistent with the directory:

# 进入/tmp目录
cd /tmp
# 修改配置文件
printf '%s\n' 7001 7002 7003 8001 8002 8003 | xargs -I{} -t sed -i 's/6379/{}/g' {}/redis.conf

start up

Since the background startup mode has been configured, the service can be started directly:
# 进入/tmp目录
cd /tmp
# 一键启动所有服务
printf '%s\n' 7001 7002 7003 8001 8002 8003 | xargs -I{} -t redis-server {}/redis.conf
View status via ps:
ps -ef | grep redis
If you want to close all processes, you can execute the command:
ps -ef | grep redis | awk '{print $2}' | xargs kill
or (recommended this way):
printf '%s\n' 7001 7002 7003 8001 8002 8003 | xargs -I{} -t redis-cli -p {} shutdown

create cluster

Although the service is started, each service is currently independent without any association.

We need to execute commands to create a cluster. It is troublesome to create a cluster before Redis5.0. After 5.0, cluster management commands are integrated into redis-cli.

1. Before Redis5.0

Cluster commands before Redis 5.0 were implemented using src/redis-trib.rb under the redis installation package. Because redis-trib.rb is written in ruby language, it is necessary to install the ruby environment.
# 安装依赖
yum -y install zlib ruby rubygems
gem install redis
Then use the command to manage the cluster:
# 进入redis的src目录
cd /tmp/redis-6.2.4/src
# 创建集群
./redis-trib.rb create --replicas 1 192.168.150.101:7001 192.168.150.101:7002 192.168.150.101:7003 192.168.150.101:8001 192.168.150.101:8002 192.168.150.101:8003
2. After Redis5.0

We are using Redis6.2.4 version, cluster management and integrated into redis-cli, the format is as follows:
redis-cli --cluster create --cluster-replicas 1 192.168.150.101:7001 192.168.150.101:7002 192.168.150.101:7003 192.168.150.101:8001 192.168.150.101:8002 192.168.150.101:8003
Command description:

redis-cli --clusterOr ./redis-trib.rb: represents a cluster operation command

create: represents the creation of a cluster

--replicas 1Or --cluster-replicas 1: specify that the number of copies of each master in the cluster is 1, and 节点总数 ÷ (replicas + 1)the number of masters is obtained at this time. Therefore, the first n in the node list is the master, and the other nodes are all slave nodes, which are randomly assigned to different masters.

What it looks like after running:

Enter yes here, and the cluster will start to be created:

You can view the cluster status with the command:
redis-cli -p 7001 cluster nodes

test

Try to connect to node 7001 and store a piece of data:
# 连接
redis-cli -p 7001
# 存储数据
set num 123
# 读取数据
get num
# 再次存储
set a 1
During cluster operation, you need to redis-cliadd -cparameters:
redis-cli -c -p 7001

2.2. Hash slot

Slot principle

Redis will map each master node to a total of 16384 slots (hash slots) ranging from 0 to 16383, which can be seen when viewing the cluster information:

Data keys are not bound to nodes, but to slots. Redis will calculate the slot value based on the effective part of the key, in two cases:

The key contains "{}", and "{}" contains at least 1 character, and the part in "{}" is a valid part
The key does not contain "{}", the entire key is a valid part

For example: if the key is num, then it will be calculated according to num, if it is {itcast}num, then it will be calculated according to itcast. The calculation method is to use the CRC16 algorithm to obtain a hash value, and then take the remainder of 16384, and the result obtained is the slot value.

As shown in the figure, when set a 1 is executed on node 7001, a hash operation is performed on a, and the remainder of 16384 is obtained, and the result is 15495, so it needs to be stored in node 103.

After reaching 7003, get numwhen executing, perform a hash operation on num, take the remainder of 16384, and the result is 2765, so you need to switch to 7001 node

summary

How does Redis determine which instance a key should be in?

Allocate 16384 slots to different instances
Calculate the hash value according to the effective part of the key, and take the remainder of 16384
The remainder is used as the slot, just find the instance where the slot is located

How to save the same type of data in the same Redis instance?

This type of data uses the same effective part, for example, keys are all prefixed with {typeId}

2.3. Cluster scaling

redis-cli --cluster provides many commands to operate the cluster, which can be viewed in the following ways:

For example, the command to add a node:

demand analysis

Requirement: Add a new master node to the cluster and store num = 10 in it

Start a new redis instance with port 7004

Add 7004 to the previous cluster and act as a master node

Assign a slot to the 7004 node so that the key num can be stored in the 7004 instance

Two new functions are needed here:

Add a node to the cluster

Assign some slots to new slots

Create a new redis instance

Create a folder:
mkdir 7004
Copy the configuration file:
cp redis.conf /7004
Modify the configuration file:
sed /s/6379/7004/g 7004/redis.conf
start up
redis-server 7004/redis.conf

Add new node to redis

The syntax for adding a node is as follows:

Excuting an order:
redis-cli --cluster add-node  192.168.150.101:7004 192.168.150.101:7001
Check the cluster status with the command:
redis-cli -p 7001 cluster nodes
As shown in the figure, 7004 has joined the cluster and is a master node by default:

However, it can be seen that the number of slots of the 7004 node is 0, so no data can be stored on the 7004

transfer slot

We want to store num to node 7004, so we need to see how many slots num has:

As shown above, the slot of num is 2765.

We can transfer the slots of 0~3000 from 7001 to 7004, the command format is as follows:

The specific commands are as follows:

establish connection:
redis-cli --cluster reshard 192.168.xxx.xxx:7001
Get the following feedback:

Ask how many slots to move, we plan to be 3000:

Here comes the new problem:

Which node to receive these slots? ?

Obviously it is 7004, so what is the id of node 7004?

Copy this id, and then copy it to the console just now:

Ask here, where did your slot move from?

all: represents all, that is, each of the three nodes transfers a part

Specific id: the id of the target node

done: no more

Here we want to get from 7001, so fill in the id of 7001:

Once filled, click done, and the slot transfer is ready:

Are you sure you want to transfer? Enter yes:

Then, view the results with the command:
redis-cli -p 7001 cluster node
can be seen:

2.4. Failover

The initial state of the cluster is as follows:

Among them, 7001, 7002, and 7003 are masters, and we plan to shut down 7002.

2.4.1, automatic failover

What happens when a master in the cluster goes down?

Stop a redis instance directly, such as 7002:

redis-cli -p 7002 shutdown

1) First, the instance loses connection with other instances

2) Then there is a suspected downtime:

3) Finally, it is determined to go offline and automatically promote a slave to the new master:

4) When the 7002 starts up again, it will become a slave node:

2.4.2. Manual failover

Using the cluster failover command, you can manually shut down a master in the cluster and switch to the slave node that executes the cluster failover command to realize data migration without perception. The process is as follows:

This failover command can specify three modes:

Default: The default process, as shown in Figure 1~6 steps
force: omits the consistency check of offset
takeover: Execute step 5 directly, ignoring data consistency, master status and other master opinions

Case requirements : Perform manual failover on the slave node 7002 to regain the master status

Proceed as follows:

1) Use redis-cli to connect to node 7002

2) Execute the cluster failover command

As shown in the picture:

Effect:

2.5, RedisTemplate access fragmentation cluster

The bottom layer of RedisTemplate also implements the support of fragmented clusters based on lettuce, and the steps used are basically the same as the sentinel mode:

1) Introduce the starter dependency of redis

2) Configure the shard cluster address

3) Configure read-write separation

Compared with sentinel mode, only the configuration of fragmented clusters is slightly different, as follows:

spring:
  redis:
    cluster:
      nodes:
        - 192.168.150.101:7001
        - 192.168.150.101:7002
        - 192.168.150.101:7003
        - 192.168.150.101:8001
        - 192.168.150.101:8002
        - 192.168.150.101:8003

Redis --- sentinel, shard cluster