1. Existing problems
The capacity of a single redis is limited, how to expand? Continue to add memory and hardware?
The concurrent write volume of a single redis is too large and there is a performance bottleneck. How to solve it?
Clusters are provided in redis 3.0 to solve these problems.
2. What is a cluster
The redis cluster is the horizontal expansion of redis, that is, start N redis nodes, distribute and store the entire data in these N nodes, and each node stores 1/N of the total data.
As shown in the figure below: a redis cluster composed of 3 masters and 3 slaves, each master undertakes one-third of the data requested and written by the client, when the master hangs up, the slave will automatically replace the master to achieve high availability.
3. How to configure the cluster?
3.1. Requirements: Configure 3 master 3 slave clusters
Next, let's configure a cluster with 3 masters and 3 slaves. A slave is hung under each master. After the master hangs up, the slave will be promoted to master.
For convenience, we simulate on one machine, my machine ip is: 192.168.200.129, 6 different nodes are distinguished by port, the configuration information is as follows
3.2. Create a case work directory: cluster
Execute the following command to create the /opt/cluster directory. All operations this time are performed in the cluster directory.
# 方便演示，停止所有的redis
ps -ef | grep redis | awk -F" " '{print $2;}' | xargs kill -9
mkdir /opt/cluster
cd /opt/cluster/
3.3. Copy redis.conf to the cluster directory
redis.conf is the default redis configuration file
cp /opt/redis-6.2.1/redis.conf /opt/cluster/
3.4. Create the configuration file of master1: redis-6379.conf
Create a redis-6379.conf file in the /opt/cluster directory, the content is as follows, note that 192.168.200.129 is the ip of the test machine, you need to replace it with your own
include /opt/cluster/redis.conf
daemonize yes
bind 192.168.200.129
dir /opt/cluster/
port 6379
dbfilename dump_6379.rdb
pidfile /var/run/redis_6379.pid
logfile "./6379.log"
# 开启集群设置
cluster-enabled yes
# 设置节点配置文件
cluster-config-file node-6379.conf
# 设置节点失联时间，超过该时间(毫秒)，集群自动进行主从切换
cluster-node-timeout 15000
3.5. Create the configuration file of master2: redis-6380.conf
Create a redis-6380.conf file in the /opt/cluster directory, the content is as follows, similar to the above master, just replace 6379 with 6380
include /opt/cluster/redis.conf
daemonize yes
bind 192.168.200.129
dir /opt/cluster/
port 6380
dbfilename dump_6380.rdb
pidfile /var/run/redis_6380.pid
logfile "./6380.log"
# 开启集群设置
cluster-enabled yes
# 设置节点配置文件
cluster-config-file node-6380.conf
# 设置节点失联时间，超过该时间(毫秒)，集群自动进行主从切换
cluster-node-timeout 150003.6. Create a master3 configuration file: redis-6381.conf
Create a redis-6381.conf file in the /opt/cluster directory with the following content
include /opt/cluster/redis.conf
daemonize yes
bind 192.168.200.129
dir /opt/cluster/
port 6381
dbfilename dump_6381.rdb
pidfile /var/run/redis_6381.pid
logfile "./6381.log"
# 开启集群设置
cluster-enabled yes
# 设置节点配置文件
cluster-config-file node-6381.conf
# 设置节点失联时间，超过该时间(毫秒)，集群自动进行主从切换
cluster-node-timeout 15000
3.7. Create a configuration file for slave1: redis-6389.conf
Create a redis-6389.conf file in the /opt/cluster directory with the following content
include /opt/cluster/redis.conf
daemonize yes
bind 192.168.200.129
dir /opt/cluster/
port 6389
dbfilename dump_6389.rdb
pidfile /var/run/redis_6389.pid
logfile "./6389.log"
# 开启集群设置
cluster-enabled yes
# 设置节点配置文件
cluster-config-file node-6389.conf
# 设置节点失联时间，超过该时间(毫秒)，集群自动进行主从切换
cluster-node-timeout 15000
3.8. Create a configuration file for slave2: redis-6390.conf
Create a redis-6390.conf file in the /opt/cluster directory with the following content
include /opt/cluster/redis.conf
daemonize yes
bind 192.168.200.129
dir /opt/cluster/
port 6390
dbfilename dump_6390.rdb
pidfile /var/run/redis_6390.pid
logfile "./6390.log"
# 开启集群设置
cluster-enabled yes
# 设置节点配置文件
cluster-config-file node-6390.conf
# 设置节点失联时间，超过该时间(毫秒)，集群自动进行主从切换
cluster-node-timeout 15000
3.9. Create a configuration file for slave3: redis-6391.conf
Create a redis-6391.conf file in the /opt/cluster directory with the following content
include /opt/cluster/redis.conf
daemonize yes
bind 192.168.200.129
dir /opt/cluster/
port 6391
dbfilename dump_6391.rdb
pidfile /var/run/redis_6391.pid
logfile "./6391.log"
# 开启集群设置
cluster-enabled yes
# 设置节点配置文件
cluster-config-file node-6391.conf
# 设置节点失联时间，超过该时间(毫秒)，集群自动进行主从切换
cluster-node-timeout 15000
3.10, start master, slave1, slave2
# 方便演示，停止所有的redis
ps -ef | grep redis | awk -F" " '{print $2;}' | xargs kill -9
# 下面启动6个redis
redis-server /opt/cluster/redis-6379.conf
redis-server /opt/cluster/redis-6380.conf
redis-server /opt/cluster/redis-6381.conf
redis-server /opt/cluster/redis-6389.conf
redis-server /opt/cluster/redis-6390.conf
redis-server /opt/cluster/redis-6391.conf
3.11. Check the startup status of 6 redis
ps -ef | grep redis
3.12. Make sure that the node-xxxx.conf file has been generated normally
Later we will merge 6 instances into one cluster. Before combining, we need to ensure that after the 6 redis instances are started, the nodes-xxxx.conf files are generated normally, as follows, the /opt/cluster directory is indeed generated successfully
3.13. Synthesize 6 nodes into a cluster
Execute the following command to combine 6 redis
/opt/redis-6.2.1/src/redis-cli --cluster create --cluster-replicas 1192.168.200.129:6379 192.168.200.129:6380 192.168.200.129:6381 192.168.200.129:6389 192.168.200.129:6390 192.168.200.129:6391
- The combined command will be followed by the ip:port list of all nodes, and the multiples are separated by spaces. Note that the ip should not write 127.0.0.1, but the real ip
- --cluster-replicas 1: Indicates that the cluster is configured in the simplest way, that is, each master is equipped with 1 slave, and 6 nodes form 3 masters and 3 slaves
The execution process is as follows, during which we will determine whether the allocation method is the same, enter: yes, and then wait for a few seconds, the cluster integration is successful
[root@hspEdu01 src]# redis-cli --cluster create --cluster-replicas 1 192.168.200.129:6379 192.168.200.129:6380 192.168.200.129:6381 192.168.200.129:6389 192.168.200.129:6390 192.168.200.129:6391
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.200.129:6390 to 192.168.200.129:6379
Adding replica 192.168.200.129:6391 to 192.168.200.129:6380
Adding replica 192.168.200.129:6389 to 192.168.200.129:6381
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: ccf3abb791e026380ad3ad2a166aa788df738437 192.168.200.129:6379
slots:[0-5460] (5461 slots) master
M: 3c372392d5a91dad64a6febadfe9524ea2cbd8c0 192.168.200.129:6380
slots:[5461-10922] (5462 slots) master
M: 2c905be9c975be367bd66c962167beca1ef66af3 192.168.200.129:6381
slots:[10923-16383] (5461 slots) master
S: 4a0f860081b969162767aac26801994de54d80a5 192.168.200.129:6389
replicates ccf3abb791e026380ad3ad2a166aa788df738437
S: 62c9f37a362459c212e8af6dd744b6562f5fe6a7 192.168.200.129:6390
replicates 3c372392d5a91dad64a6febadfe9524ea2cbd8c0
S: a2f89efc09681520f9d9502707b18e1f46a40b90 192.168.200.129:6391
replicates 2c905be9c975be367bd66c962167beca1ef66af3
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 192.168.200.129:6379)
M: ccf3abb791e026380ad3ad2a166aa788df738437 192.168.200.129:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
M: 2c905be9c975be367bd66c962167beca1ef66af3 192.168.200.129:6381
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
S: 62c9f37a362459c212e8af6dd744b6562f5fe6a7 192.168.200.129:6390
slots: (0 slots) slave
replicates 3c372392d5a91dad64a6febadfe9524ea2cbd8c0
M: 3c372392d5a91dad64a6febadfe9524ea2cbd8c0 192.168.200.129:6380
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 4a0f860081b969162767aac26801994de54d80a5 192.168.200.129:6389
slots: (0 slots) slave
replicates ccf3abb791e026380ad3ad2a166aa788df738437
S: a2f89efc09681520f9d9502707b18e1f46a40b90 192.168.200.129:6391
slots: (0 slots) slave
replicates 2c905be9c975be367bd66c962167beca1ef66af3
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
3.14. Connect to cluster nodes and view cluster information: cluster nodes
You need to use the redis-cli -c command to connect to any of the 6 nodes in the cluster. Note that it is a little different from the previous connection parameters. There is an additional -c parameter after the redis-cli command, which means that the cluster is used to connect. Later, use clusternodes to view cluster node information, as follows
192.168.200.129:6379> cluster nodes
2c905be9c975be367bd66c962167beca1ef66af3 192.168.200.129:6381@16381 master - 0
1650194157604 3 connected 10923-16383
62c9f37a362459c212e8af6dd744b6562f5fe6a7 192.168.200.129:6390@16390 slave
3c372392d5a91dad64a6febadfe9524ea2cbd8c0 0 1650194158611 2 connected
3c372392d5a91dad64a6febadfe9524ea2cbd8c0 192.168.200.129:6380@16380 master - 0
1650194158000 2 connected 5461-10922
4a0f860081b969162767aac26801994de54d80a5 192.168.200.129:6389@16389 slave
ccf3abb791e026380ad3ad2a166aa788df738437 0 1650194156000 1 connected
ccf3abb791e026380ad3ad2a166aa788df738437 192.168.200.129:6379@16379
myself,master - 0 1650194157000 1 connected 0-5460
a2f89efc09681520f9d9502707b18e1f46a40b90 192.168.200.129:6391@16391 slave
2c905be9c975be367bd66c962167beca1ef66af3 0 1650194159617 3 connected
192.168.200.129:6379>
As shown in the figure below, explain the results of cluster nodes. First, look at the notes in red. Each node in the cluster will generate an ID, and this ID information will be written into the node-xxxx.conf file. Why generate an ID? ?
Because the ip and port of the node may change, but the ID of the node will not change, and other nodes can recognize each node through the ID of other nodes.
3.15. Verify the read and write operations of cluster data
As follows, we connect the node 6379, and then perform a set operation, the effect is as follows, the write is successful
[root@hspEdu01 cluster]# redis-cli -c -h 192.168.200.129 -p 6379
192.168.200.129:6379> set name ready
-> Redirected to slot [5798] located at 192.168.200.129:6380
OK
192.168.200.129:6380>You may have noticed that we are obviously operating on 6379, but the request is forwarded to the node 6380 for processing. Here is the knowledge of the slot we will talk about later, let's look backward first.
4. How does the redis cluster allocate these 6 nodes?
A cluster has at least 3 master nodes, because the election of a new master requires the consent of more than half of the cluster master nodes to be elected successfully. If there are only two master nodes and one of them hangs up, the conditions for electing a new master cannot be met.
The option --cluster-replicas 1 indicates that we want to create a slave node for each master node in the cluster.
The allocation principle tries to ensure that each master library runs on a different ip, and each master library and slave library are not on the same ip, so as to achieve high availability.
5. What are slots (slots)
As shown in the figure below, let's take a look at some information output during the process of cluster merging
The Redis cluster internally divides 16384 slots (slots). When merging, each slot will be mapped to a master. For example, the relationship between the above three masters and slots is as follows:
| redis master node | Slot range |
| master1 (port: 6379) | [0-5460], the position of the slot starts from 0, and 0 means the first slot |
| master2 (port: 6380) | [5460-10922] |
| master3 (port: 6381) | [10923-16383] |
| slave1,slave2,slave3 | There is no slot for the slave node, and the slave is used as a substitute for the master |
And each key in the database belongs to one of the 16384 slots. When reading and writing data through the key, redis needs to calculate the slots corresponding to the key according to the key, and then find the corresponding slots according to the mapping relationship between the slots and the master. The redis node, the data corresponding to the key is on this node.
Use the formula CRC16(key)%16384 in the cluster to calculate which slot the key belongs to
6. Enter values in the cluster
Every time you enter and query the key value in redis-cli, redis will calculate the slot corresponding to the key. If it is not the slot of the current redis node, redis will report an error and inform you the redis instance address and port to go to. The effect is as follows, we connect The instance 6379 is used to operate k1. The node finds that the slot of k1 is above 6381 and returns an error message. What should I do?
[root@hspEdu01 cluster]# redis-cli -h 192.168.200.129 -p 6379
192.168.200.129:6379> set k1 v1
(error) MOVED 12706 192.168.200.129:6381
Using the redis-cli client to provide the -c parameter can solve this problem, which means that it is executed in a cluster mode. When the current node cannot handle the command when executing the command, it will automatically redirect the request to the target node. The effect is as follows, it is redirected to 6381
[root@hspEdu01 cluster]# redis-cli -c -h 192.168.200.129 -p 6379
192.168.200.129:6379> set k1 v1
-> Redirected to slot [12706] located at 192.168.200.129:6381
OK
192.168.200.129:6381>
Similarly, the execution of get will be redirected, the effect is as follows
[root@hspEdu01 cluster]# redis-cli -c -h 192.168.200.129 -p 6379
192.168.200.129:6379> get k1
-> Redirected to slot [12706] located at 192.168.200.129:6381
"v1"
192.168.200.129:6381>
If it is not under a slot, multi-key operations such as mget and mset cannot be used. The effect is as follows
192.168.200.129:6381> mset k1 v1 k2 v2
(error) CROSSSLOT Keys in request don't hash to the same slot
192.168.200.129:6381> mget k1 k2
(error) CROSSSLOT Keys in request don't hash to the same slot
You can use {} to define the concept of a group, so that the same key value in {} in the key can be placed in a slot, and the effect is as follows
192.168.200.129:6381> mset k1{g1} v1 k2{g1} v2 k3{g1} v3
OK
192.168.200.129:6381> mget k1{g1} k2{g1} k3{g1}
1) "v1"
2) "v2"
3) "v3"7. Some commands related to slot
- cluster keyslot <key>: Calculate the slot corresponding to the key
- cluster coutkeysinslot <slot>: Get the number of keys in the slot slot
- cluster getkeysinslot <slot> <count> returns the keys in count slots
192.168.200.129:6381> cluster keyslot k1{g1}
(integer) 13519
192.168.200.129:6381> cluster countkeysinslot 13519
(integer) 3
192.168.200.129:6381> cluster getkeysinslot 13519 3
1) "k1{g1}"
2) "k2{g1}"
3) "k3{g1}"8. Fault recovery
If the master node goes offline, can the slave node be promoted to the master node? Note: Wait for 15 seconds
Let's try, as follows, connect to master1, and then stop master1
[root@hspEdu01 cluster]# redis-cli -c -h 192.168.200.129 -p 6379
192.168.200.129:6379> shutdown
not connected>
Execute the following command to connect to master1 and see the information of the cluster nodes
redis-cli -c -h 192.168.200.129 -p 6380
cluster nodes
The output is as follows, you can see that slave1 (6389) has indeed become the master, and its original master: master1 (6379) is offline
Next, let's start 6379 again, and then see what the cluster looks like. The command is as follows
[root@hspEdu01 cluster]# redis-server /opt/cluster/redis-6379.conf
[root@hspEdu01 cluster]# redis-cli -c -h 192.168.200.129 -p 6379
192.168.200.129:6379> cluster nodes
The execution results are as follows, 6379 has become a slave, and it is hung under 6389
If the master and slave of a certain slot are down, can the redis service continue?
At this time, it depends on the value of the cluster-require-full-coverage parameter
- yes (default value): The entire cluster is unable to provide services
- no: All the slot data of the downtime cannot be used, and the other slots are normal
9. SpringBoot integrates redis cluster
9.1. Introduce the maven configuration of redis
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
9.2. Configure redis cluster information in application.properties
# 集群节点(host:port)，多个之间用逗号隔开
spring.redis.cluster.nodes=192.168.200.129:6379,192.168.200.129:6380,192.168.200.129:6381,192.168.200.129:6389,192.168.200.129:6390,192.168.200.129:6391
# 连接超时时间（毫秒）
spring.redis.timeout=60000
9.3. Use the RedisTemplate tool class to operate redis
RedisTemplate is used in springboot to operate redis, and this object needs to be injected into our bean. The code is as follows:
@Autowired
private RedisTemplate<String, String> redisTemplate;
// 用下面5个对象来操作对应的类型
this.redisTemplate.opsForValue(); //提供了操作string类型的所有方法
this.redisTemplate.opsForList(); // 提供了操作list类型的所有方法
this.redisTemplate.opsForSet(); //提供了操作set的所有方法
this.redisTemplate.opsForHash(); //提供了操作hash表的所有方法
this.redisTemplate.opsForZSet(); //提供了操作zset的所有方法
9.4, RedisTemplate sample code
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.RedisCallback;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.http.MediaType;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
/**
* @className RedisController
* @date 2022/6/21
**/
@RestController
@RequestMapping("/redis")
public class RedisController {
@Autowired
private RedisTemplate<String, String> redisTemplate;
/**
* string测试
* @return
*/
@RequestMapping("/stringTest")
public String stringTest() {
this.redisTemplate.delete("name");
this.redisTemplate.opsForValue().set("name", "路人");
String name = this.redisTemplate.opsForValue().get("name");
return name;
}
/**
* list测试
* @return
*/
@RequestMapping("/listTest")
public List<String> listTest() {
this.redisTemplate.delete("names");
this.redisTemplate.opsForList().rightPushAll("names", "刘德华", "张学友", "郭富城", "黎明");
List<String> courses = this.redisTemplate.opsForList().range("names", 0, -1);
return courses;
}
/**
* set类型测试
* @return
*/
@RequestMapping("setTest")
public Set<String> setTest() {
this.redisTemplate.delete("courses");
this.redisTemplate.opsForSet().add("courses", "java", "spring", "springboot");
Set<String> courses = this.redisTemplate.opsForSet().members("courses");
return courses;
}
/**
* hash类型测试
* @return
*/
@RequestMapping("hashTest")
public Map<Object, Object> hashTest() {
this.redisTemplate.delete("userMap");
Map<String, String> map = new HashMap<>();
map.put("name", "路人");
map.put("age", "30");
this.redisTemplate.opsForHash().putAll("userMap", map);
Map<Object, Object> userMap = this.redisTemplate.opsForHash().entries("userMap");
return userMap;
}
/**
* zset类型测试
* @return
*/
@RequestMapping("zsetTest")
public Set<String> zsetTest() {
this.redisTemplate.delete("languages");
this.redisTemplate.opsForZSet().add("languages", "java", 100d);
this.redisTemplate.opsForZSet().add("languages", "c", 95d);
this.redisTemplate.opsForZSet().add("languages", "php", 70);
Set<String> languages = this.redisTemplate.opsForZSet().range("languages", 0, -1);
return languages;
}
/**
* 查看redis机器信息
* @return
*/
@RequestMapping(value = "/info", produces = MediaType.TEXT_PLAIN_VALUE)
public String info() {
return this.redisTemplate.execute((RedisCallback<String>) connection ->
String.valueOf(connection.execute("info")));
}
}