Series column: Redis series column

Slice cluster

Data extension mode

If the total amount of data to be cached by Redis is not very large, such as 5GB of data, it is generally sufficient to use the master-slave model + sentinel cluster to ensure high availability. However, if the total amount of data to be cached by Redis is relatively large, or may increase in the future, such as 20GB or 50GB of data, then one main database cannot be satisfied. At this time, there are generally two modes for expansion: vertical expansion and horizontal expansion.

Scale up

Scale-up is to upgrade the configuration of a single Redis instance, increase the server memory capacity, disk capacity, and use a higher-configured CPU. For example, the previously used 4C 8GB 50GB, upgraded to 8C 32GB 100GB, and the amount of data increases, continue to add configuration.

The benefit of scaling up is simplicity to implement, but there are potential problems:

If RDB data persistence is used, as the amount of data increases, on the one hand, the memory required will also increase; on the other hand, when RDB is persisted, the main thread will fork the child process, and the time consuming of the fork operation and the amount of data in Redis It is positively correlated. The larger the amount of data, the longer the main thread is blocked by the fork operation, which will cause Redis to sometimes slow down the response. Of course, this problem does not exist if RDB persistence is not required.
Another problem is that vertical expansion will be limited by hardware and cost. It is easy to expand memory from 32GB to 64GB, but if you want to expand to 1TB, you will face hardware capacity and cost constraints.

Scale out

Horizontal expansion is to increase the number of Redis instances and distribute data to each instance. For example, to cache 15GB of data, you can use three 8GB servers, and one instance only needs to cache 5GB of data.

This is Redis 切片集群, also called a sharded cluster, which is to start multiple Redis instances to form a cluster, and then divide the received data into each instance according to certain rules. This solution only needs to increase the number of Redis instances, without worrying about the hardware and cost constraints of a single instance.

Although it is troublesome to set up a sliced cluster, it can save a large amount of data. Since the amount of data saved by a single instance is small, the blocking effect on the main thread during RDB persistence is also relatively small. And with the expansion of users or business scale, it is usually unavoidable to save a large amount of data, then sliced cluster is a very good solution.

Cluster Architecture

The deployment of Redis generally has different solutions according to different scenarios. Here is a simple comparison:

1. Stand-alone mode

If the amount of data is small, it is mainly used for scenarios with high concurrency and high performance. For example, the cache is generally only a few gigabytes. It is enough to deploy a single-machine Redis instance.

2. Master-slave replication + sentinel

This mode is mainly to ensure high concurrent reading and high availability scenarios. The master-slave mode is a master library + multiple slave libraries. Deploying several slave libraries is related to read throughput, and then deploying a sentinel cluster to ensure the master High availability from the cluster. Generally, this mode can be used in scenarios where the amount of data is not particularly large.

3. Cluster mode

In the master-slave architecture mode of Redis, all data is written to one master library, and there are some problems such as the upper limit of single-machine capacity and high concurrent write performance. The Redis cluster has multiple main libraries. The data is distributed to each main library according to certain rules. The number of main libraries depends on the size of the data volume. The main library nodes can be dynamically added or deleted according to the data volume.

A Redis cluster requires at least 3 master nodes to operate normally. It is recommended to configure at least one slave node for each master node for master-standby switchover. When the master library is unavailable, the slave library can be installed. In the production environment, it is recommended to use 6 servers and deploy 3 master libraries and 3 slave libraries respectively, which can better ensure the high availability of the cluster.

Different from the master-slave architecture, the slave nodes in the Redis cluster are mainly used for high availability. Each master database is hung with one or two slave nodes, which are used for data hot backup and master-standby switching to achieve high availability. Therefore, in the cluster mode The master-slave nodes do not need sentries to ensure high availability.

In addition, if you want to improve the performance of high concurrent reading, you can generally increase the read throughput by increasing the main library node. Of course, you can also configure the slave library to be readable by modifying the configuration, and do read-write separation mode, but this may be complicated. At the same time, the Redis client also supports read-write separation.

Cluster deployment

First stop the previously deployed Redis master-slave and sentinel clusters. The deployment method of the Redis cluster is different from before. Then according to the above cluster architecture diagram, we use three servers to deploy 6 instances, that is, three masters and three slaves, and try to keep the master and slave instances not on the same machine.

规划6个实例部署到三台服务器上，6个实例的端口依次为 7001 ~ 7006，部署架构如下：

服务器	Redis 实例
172.17.0.2	7001、7002
172.17.0.3	7003、7004
172.17.0.4	7005、7006

1、创建数据目录

首先在第一台服务器上操作，先创建几个目录：

# mkdir -p /var/redis/log
# mkdir -p /var/redis/7001
# mkdir -p /var/redis/7002
复制代码

2、配置文件

将Redis配置文件拷贝两份到 /etc/redis 下，分别为 7001.conf、7002.conf：

# cp /usr/local/src/redis-6.2.5/redis.conf /etc/redis/7001.conf
# cp /usr/local/src/redis-6.2.5/redis.conf /etc/redis/7002.conf
复制代码

修改配置文件中的如下配置，7001 与文件端口一致即可：

配置	值	说明
bind	172.17.0.2	绑定本机IP
port	7001	端口
cluster-enabled	yes	开启集群模式
cluster-config-file	/etc/redis/node-7002.conf	集群配置文件
cluster-node-timeout	15000	节点存活超时时长
daemonize	yes	守护进程
pidfile	/var/run/redis_7001.pid	PID文件
dir	/var/redis/7001	数据目录
logfile	/var/redis/log/7001.log	日志文件
appendonly	yes	开启AOF持久化

注意不要在文件中配置 replicaof <masterip> <masterport>

3、启动脚本

将Redis启动脚本复制两份到 /etc/init.d/下：

# cp /usr/local/src/redis-6.2.5/utils/redis_init_script /etc/init.d/redis_7001
# cp /usr/local/src/redis-6.2.5/utils/redis_init_script /etc/init.d/redis_7002
复制代码

然后修改脚本中的端口号：

并在脚本开头添加如下两行注释做开机启动：

# chkconfig: 2345 90 10 
# description: Redis is a persistent key-value database
复制代码

设置开机启动：

# chkconfig redis_7001 on
# chkconfig redis_7002 on
复制代码

4、启动Redis实例

分别安装上面的方式在三台服务器上加好配置文件和启动脚本，然后分别启动各个实例。

# cd /etc/init.d

# ./redis_7001 start
# ./redis_7002 start
复制代码

查看Redis进程：

如果启动不成功可以查看日志：

5、创建集群

集群管理可以使用官方提供的 redis-cli --cluster 工具，命令格式为：

redis-cli --cluster SUBCOMMAND [ARGUMENTS] [OPTIONS]
复制代码

创建集群的命令格式如下，--cluster-replicas <arg> 表示每个 master 有几个 slave：

redis-cli --cluster create host1:port1 ... hostN:portN --cluster-replicas <arg>
复制代码

下面将6个实例创建一个集群，它会自动帮我们分配哪些实例作为 master，哪些作为 slave，并尽量让 master 和 slave 不在同一个服务器上。通过输出结果可以看到集群创建的过程。

[root@centos-01 init.d]# redis-cli --cluster create 
    172.17.0.2:7001 172.17.0.2:7002 
    172.17.0.3:7003 172.17.0.3:7004 
    172.17.0.4:7005 172.17.0.4:7006 
    --cluster-replicas 1
    
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 172.17.0.3:7004 to 172.17.0.2:7001
Adding replica 172.17.0.4:7006 to 172.17.0.3:7003
Adding replica 172.17.0.2:7002 to 172.17.0.4:7005
M: 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 172.17.0.2:7001
   slots:[0-5460] (5461 slots) master
S: 9b0751a4b6ea227658ed26576d9bda1f3b01fc04 172.17.0.2:7002
   replicates 26776d9aef1c511d77efee79a45a63eff18fb4f9
M: 4e2f6532200991c93bbf23f7d4996fbd2aff02ac 172.17.0.3:7003
   slots:[5461-10922] (5462 slots) master
S: 9a627cad72e4192802237c00e4e570b37241feda 172.17.0.3:7004
   replicates 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646
M: 26776d9aef1c511d77efee79a45a63eff18fb4f9 172.17.0.4:7005
   slots:[10923-16383] (5461 slots) master
S: fc25b50d6afe39211a47609287afd250c9e3183b 172.17.0.4:7006
   replicates 4e2f6532200991c93bbf23f7d4996fbd2aff02ac
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 172.17.0.2:7001)
M: 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 172.17.0.2:7001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: 26776d9aef1c511d77efee79a45a63eff18fb4f9 172.17.0.4:7005
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 9a627cad72e4192802237c00e4e570b37241feda 172.17.0.3:7004
   slots: (0 slots) slave
   replicates 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646
S: 9b0751a4b6ea227658ed26576d9bda1f3b01fc04 172.17.0.2:7002
   slots: (0 slots) slave
   replicates 26776d9aef1c511d77efee79a45a63eff18fb4f9
S: fc25b50d6afe39211a47609287afd250c9e3183b 172.17.0.4:7006
   slots: (0 slots) slave
   replicates 4e2f6532200991c93bbf23f7d4996fbd2aff02ac
M: 4e2f6532200991c93bbf23f7d4996fbd2aff02ac 172.17.0.3:7003
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
复制代码

创建完成后，可以在其它实例上用 redis-cli --cluster check 检查集群状态：

[root@centos-02 init.d]# redis-cli --cluster check 172.17.0.3:7003
172.17.0.3:7003 (4e2f6532...) -> 0 keys | 5462 slots | 1 slaves.
172.17.0.4:7005 (26776d9a...) -> 0 keys | 5461 slots | 1 slaves.
172.17.0.2:7001 (9b4f3c2c...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 172.17.0.3:7003)
M: 4e2f6532200991c93bbf23f7d4996fbd2aff02ac 172.17.0.3:7003
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
M: 26776d9aef1c511d77efee79a45a63eff18fb4f9 172.17.0.4:7005
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: fc25b50d6afe39211a47609287afd250c9e3183b 172.17.0.4:7006
   slots: (0 slots) slave
   replicates 4e2f6532200991c93bbf23f7d4996fbd2aff02ac
S: 9a627cad72e4192802237c00e4e570b37241feda 172.17.0.3:7004
   slots: (0 slots) slave
   replicates 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646
S: 9b0751a4b6ea227658ed26576d9bda1f3b01fc04 172.17.0.2:7002
   slots: (0 slots) slave
   replicates 26776d9aef1c511d77efee79a45a63eff18fb4f9
M: 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 172.17.0.2:7001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
复制代码

至此，一个三台服务器三主三从的Redis集群就搭建好了。

集群原理

数据分布算法

切片集群是一种保存大量数据的通用机制，这个机制可以有不同的实现方案，Redis 官方提供的集群化方案为 Redis Cluster，Redis Cluster 方案中规定了数据和实例的对应规则。

哈希槽

Redis Cluster 采用哈希槽（Hash Slot）来处理数据和实例之间的映射关系，Redis Cluster 将集群划分为 16384 个槽，每个键值对都会根据它的 key，被映射到一个哈希槽中。

在创建集群时，redis-cli --cluster create 命令会自动把 16384 个槽平均分布在集群实例上，如果集群有 N 个Master，那每个Master上槽的个数为 16384 / N。

通过前面创建集群的日志可以得知，每个Master对应了哪些 Slot，以及 Master -> Slave 的关系。

Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 172.17.0.3:7004 to 172.17.0.2:7001
Adding replica 172.17.0.4:7006 to 172.17.0.3:7003
Adding replica 172.17.0.2:7002 to 172.17.0.4:7005
M: 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 172.17.0.2:7001
   slots:[0-5460] (5461 slots) master
S: 9b0751a4b6ea227658ed26576d9bda1f3b01fc04 172.17.0.2:7002
   replicates 26776d9aef1c511d77efee79a45a63eff18fb4f9
M: 4e2f6532200991c93bbf23f7d4996fbd2aff02ac 172.17.0.3:7003
   slots:[5461-10922] (5462 slots) master
S: 9a627cad72e4192802237c00e4e570b37241feda 172.17.0.3:7004
   replicates 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646
M: 26776d9aef1c511d77efee79a45a63eff18fb4f9 172.17.0.4:7005
   slots:[10923-16383] (5461 slots) master
S: fc25b50d6afe39211a47609287afd250c9e3183b 172.17.0.4:7006
   replicates 4e2f6532200991c93bbf23f7d4996fbd2aff02ac
复制代码

当然，如果某台机器内存、CPU等配置较高，我们可以给这台机器的Redis实例分配更多的槽。重新分配槽的命令为 redis-cli --cluster reshard，例如下面将 172.17.0.2:7001（..2f5ca7bf646）上500个 Slots 移到 172.17.0.2:7003（..fbd2aff02ac）去。

[root@centos-01 /]# redis-cli --cluster reshard 172.17.0.2:7001 
    --cluster-from 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 
    --cluster-to 4e2f6532200991c93bbf23f7d4996fbd2aff02ac 
    --cluster-slots 500
复制代码

重新分配后，可以再用 redis-cli --cluster check 命令检查槽位配置。

槽位定位算法

Redis Cluster 默认会对 key 值使用 CRC16 算法计算得到一个 16 bit 的hash整数，然后用这个整数值对 16384 取模，得到一个 0~16383 范围内的模数，每个模数就代表一个相应编号的哈希槽，就得到了具体的槽位。

Redis Cluster 还允许用户强制某个 key 挂在特定槽位上，通过在 key 字符串里面嵌入 {tag} 标记，CRC16 就会只计算 tag 的 hash 值。例如：user_{system}、role_{system}，这两个 key 都只对 system 计算 hash 值，那最终这两个 key 都会落到同一个槽上。

哈希槽的优点

Redis Cluster 不是直接维护key与实例的映射关系，而是维护了哈希槽和实例的映射关系，这就需要客户端先用 CRC16 计算key的hash值，那为什么不直接维护key与实例的映射关系呢？

首先，整个集群存储key的数量是无法预估的，直接存储key与实例的映射关系，这个映射表可能会非常庞大，这个映射表无论是存储在服务端还是客户端都会占用非常大的内存空间。
Redis Cluster 采用去中心化的模式，每个实例会与其它实例交换自己的槽位配置，存储了整个集群完整的映射关系，如果是存储key与实例的关系，节点之间交换信息会变得庞大，消耗过多的网络资源。
当集群在扩容、缩容、数据均衡时，节点之间会发生数据迁移，迁移时需要修改每个key的映射关系，维护成本高。

通过在中间增加一层哈希槽，可以把数据和节点解耦，key通过hash计算，只需要关心映射到了哪个哈希槽，然后再通过哈希槽和节点的映射表找到节点，相当于消耗了很少的CPU资源，不但让数据分布更均匀，还可以让这个映射表变得很小，利于客户端和服务端保存，节点之间交换信息时也变得轻量。在数据迁移时，都以哈希槽为基本单位进行操作，也简化了节点扩容、缩容的难度，便于集群的维护和管理。

槽位配置信息

Redis 集群中的客户端（jedis、lettuce等）连接到集群实例时，它会得到一份集群的槽位配置信息，然后客户端会把哈希槽信息缓存在本地。当客户端请求键值对时，会先计算键所对应的哈希槽，然后就可以给哈希槽对应的实例发送请求了。

那每个实例怎么知道其它实例负责的槽位呢？这是因为集群中的实例相互连接时，会将自己的哈希槽信息发送给对方，当实例之间相互连接后，每个实例就有所有哈希槽的映射关系了。这样客户端在连接到某个实例时，就可以得到集群的整个哈希槽映射关系了。

集群的槽位信息存储于每个节点中，它不需要另外的分布式存储来存储节点槽位信息。前面在创建集群时，我们通过 cluster-config-file 指定了集群配置文件，集群的每个节点会将集群的哈希槽信息持久化到这个配置文件中，所以必须确保配置文件是可写的，而且不要人工修改配置文件。有了这个文件，实例宕机重启也能恢复槽位信息配置。

例如下面的两个配置文件：

槽位重定向

前面说了，Redis集群是去中心化的模式，集群中每个实例都有一份整个集群完整的槽位映射表，这就为客户端提供了纠错能力，因为客户端访问的槽位可能并不在目标实例上。

Redis集群的客户端连接集群时，它也会得到一份集群的槽位配置信息，然后缓存在本地。但如果集群槽位信息发生变更，比如扩容、缩容，节点之间可以通过相互传递消息来获得最新的哈希槽分配信息，但是客户端是无法主动感知这些变化的。

Redis Cluster 提供了一种重定向机制，在客户端给一个实例发送指令时，该实例发现指令的 key 所在的槽位并不归自己管理，这时它会向客户端返回一个特殊的跳转指令 MOVED，其中就包含了这个 key 所在的槽位和实例，然后客户端再重新去访问这个新实例。

例如下面在 7001 上访问返回了 MOVED 命令，说明了请求的键值对所在的哈希槽为 5798，实际是在 172.17.0.3:7003 这个实例上。这时客户端会更新本地缓存的槽位信息，然后向新实例重新发送指令。

数据迁移

在集群中，我们可能会进行扩容、缩容，或是负载均衡等操作，Redis Cluster 就会把哈希槽重新分布一遍，然后就会迁移数据。

实例之间进行数据迁移都是以哈希槽为基本单位进行操作，当一个槽正在迁移时，这个槽就处于中间过渡状态。那么就有可能这个槽的一部分数据已迁移到目标节点，一部分数据还在原节点。

客户端在访问原节点时，如果对应的数据还在原节点里，那么原节点正常处理。如果对应的数据不在原节点里，那么有两种可能，要么该数据在新节点里，要么根本就不存在。旧节点不知道是哪种情况，所以它会向客户端返回一个 ASK 的重定向指令，返回的 ASK 指令告诉了客户端这个槽位的数据正在迁移中，并且告诉客户端这个槽位的新实例地址。客户端收到这个重定向指令后，先去目标节点执行一个不带任何参数的 ASKING 指令，然后在目标节点再重新执行原先的操作指令。

GET hello:key
(error) ASK 13320 172.17.0.3:7003
复制代码

之所以要先发送 ASKING 指令，是因为在迁移没有完成之前，这个槽位不归新节点管理，如果这个时候向新节点发送该槽位的指令，它会向客户端返回一个 MOVED 重定向指令告诉它去原节点去执行，这样就会导致循环重定向。而 ASKING 指令就是告诉新节点强制执行下一条指令，把下一条指令当成自己的槽位来处理。

和 MOVED 命令不同，ASK 命令并不会更新客户端缓存的哈希槽映射信息。所以，如果客户端再次请求这个槽中的数据，它还是会给原节点发送请求。

另外，Redis Cluster 的数据迁移是同步的，迁移一个key会同时阻塞原节点和目标节点，迁移过程中会有一定的性能问题。

主备切换

Redis 集群中的 Slave 节点主要是用于做数据的热备、主备切换，实现集群的高可用的，主库宕机后，集群就会从主库的 Slave 节点中选举出一个节点切换为新主库。

主备切换选举的过程与哨兵模式的原理非常类似，可参考：Redis系列（3） — 哨兵高可用，下面简单说明下集群模式中主备切换的原理。

集群中的 Master 节点之间会不断发送 PING 消息来交换信息，并确认节点间的通信是否正常等。

如果一个节点 PING 另一个节点，在 cluster-node-timeout 内都没有返回，那么就认为另一个节点宕机，这个节点的状态就是主观下线(sdown)，如果超过半数节点都认为另一个节点宕机了，那就是客观下线(odown)。

判断节点客观下线后，就要从节点的Slave节点中重选一个，这时会按如下规则进行选举：

Slave 节点与 Master 节点断开连接的时间如果超过了 cluster-node-timeout * cluster-replica-validity-factor，就会被剔除
根据Slave节点对Master节点复制数据的 offset 排序，offset 越大，优先选举
如果 offset 一样，所有 Master 节点对这些 Slave 节点投票，得票超过半数（N/2+1）的就选举通过。

cluster-node-timeout 15000
cluster-replica-validity-factor 10
复制代码

选举出新的节点后，就执行主备切换的过程。整个流程跟哨兵非常类似，可以看出 Redis Cluster 功能更强大，集成了主从复制和哨兵的功能。

集群实验

主从切换

下面通过实验来验证下主库宕机后，从库是否会自动切换为主库。

通过前面的部署可以知道集群节点的关系如下，以 7003 为例做测试：

172.17.0.2:7001（Master） -> 172.17.0.3:7004(Slave)
172.17.0.4:7005（Master） -> 172.17.0.2:7002(Slave)
172.17.0.3:7003（Master） -> 172.17.0.4:7006(Slave)
复制代码

先在 7003（Master）上验证操作可行：

在对应的 7006（Slave）上不可操作：

此时将 7003 kill 掉：

这时再检查集群状态，会发现之前的 7006（Slave）已经切换为 Master 了：

[root@centos-01 ~]# redis-cli --cluster check 172.17.0.2:7001
Could not connect to Redis at 172.17.0.3:7003: Connection refused
172.17.0.2:7001 (9b4f3c2c...) -> 0 keys | 5461 slots | 1 slaves.
172.17.0.4:7005 (26776d9a...) -> 0 keys | 5461 slots | 1 slaves.
172.17.0.4:7006 (fc25b50d...) -> 1 keys | 5462 slots | 0 slaves.
[OK] 1 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 172.17.0.2:7001)
M: 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 172.17.0.2:7001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: 26776d9aef1c511d77efee79a45a63eff18fb4f9 172.17.0.4:7005
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 9a627cad72e4192802237c00e4e570b37241feda 172.17.0.3:7004
   slots: (0 slots) slave
   replicates 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646
S: 9b0751a4b6ea227658ed26576d9bda1f3b01fc04 172.17.0.2:7002
   slots: (0 slots) slave
   replicates 26776d9aef1c511d77efee79a45a63eff18fb4f9
M: fc25b50d6afe39211a47609287afd250c9e3183b 172.17.0.4:7006
   slots:[5461-10922] (5462 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
复制代码

再连接到 7006 上，这时发现可以进行操作了：

接下来再把之前的 7003 重新启动：

再检查集群的状态，可以看到 7003（...2aff02ac）自动变为了 7006（...50c9e3183b）的从库。

[root@centos-01 ~]# redis-cli --cluster check 172.17.0.2:7001
172.17.0.2:7001 (9b4f3c2c...) -> 0 keys | 5461 slots | 1 slaves.
172.17.0.4:7005 (26776d9a...) -> 0 keys | 5461 slots | 1 slaves.
172.17.0.4:7006 (fc25b50d...) -> 1 keys | 5462 slots | 1 slaves.
[OK] 1 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 172.17.0.2:7001)
M: 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 172.17.0.2:7001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: 26776d9aef1c511d77efee79a45a63eff18fb4f9 172.17.0.4:7005
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 9a627cad72e4192802237c00e4e570b37241feda 172.17.0.3:7004
   slots: (0 slots) slave
   replicates 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646
S: 9b0751a4b6ea227658ed26576d9bda1f3b01fc04 172.17.0.2:7002
   slots: (0 slots) slave
   replicates 26776d9aef1c511d77efee79a45a63eff18fb4f9
M: fc25b50d6afe39211a47609287afd250c9e3183b 172.17.0.4:7006
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 4e2f6532200991c93bbf23f7d4996fbd2aff02ac 172.17.0.3:7003
   slots: (0 slots) slave
   replicates fc25b50d6afe39211a47609287afd250c9e3183b
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
复制代码

通过上面的测试说明主库宕机时，从库会自动切换为主库顶上去，原主库重启后，会自动变为新主库的从库。

集群扩容

如果要提高集群的读写吞吐量、或支撑更多的数据，这时就需要做集群扩容，增加节点，下面就来实验一下。

增加一台 172.17.0.5 的服务器，同样的，新增一主一从两个节点，端口分别为 7007、7008。

先按照前面集群搭建的步骤把基础环境装好：

# 创建目录
# mkdir -p /var/redis/log
# mkdir -p /var/redis/7007
# mkdir -p /var/redis/7008

# 拷贝配置文件，并修改相关配置
# cp /usr/local/src/redis-6.2.5/redis.conf /etc/redis/7007.conf
# cp /usr/local/src/redis-6.2.5/redis.conf /etc/redis/7008.conf

# 拷贝脚本，并修改相关配置
# cp /usr/local/src/redis-6.2.5/utils/redis_init_script /etc/init.d/redis_7007
# cp /usr/local/src/redis-6.2.5/utils/redis_init_script /etc/init.d/redis_7008
复制代码

将两个节点启动起来：

集群中增加节点的命令格式如下：

redis-cli --cluster add-node new_host:new_port existing_host:existing_port
    --cluster-slave --cluster-master-id <arg>
复制代码

下面将 7007 添加为集群的 Master：

[root@centos-01 ~]# redis-cli --cluster add-node 172.17.0.5:7007 172.17.0.2:7001
>>> Adding node 172.17.0.5:7007 to cluster 172.17.0.2:7001
......
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 172.17.0.5:7007 to make it join the cluster.
[OK] New node added correctly.
复制代码

此时检查集群状态，发现 7007 没有 slave（0 slaves），也没有分配槽位（0 slots），7007 的实例ID 为 bca31339ca8d8177e9d80831d72b64bc52fdbb87。

[root@centos-01 ~]# redis-cli --cluster check 172.17.0.2:7001
172.17.0.2:7001 (9b4f3c2c...) -> 0 keys | 5461 slots | 1 slaves.
172.17.0.4:7005 (26776d9a...) -> 0 keys | 5461 slots | 1 slaves.
172.17.0.5:7007 (bca31339...) -> 0 keys | 0 slots | 0 slaves.
172.17.0.4:7006 (fc25b50d...) -> 1 keys | 5462 slots | 1 slaves.
[OK] 1 keys in 4 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 172.17.0.2:7001)
M: 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 172.17.0.2:7001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: 26776d9aef1c511d77efee79a45a63eff18fb4f9 172.17.0.4:7005
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: bca31339ca8d8177e9d80831d72b64bc52fdbb87 172.17.0.5:7007
   slots: (0 slots) master
S: 9a627cad72e4192802237c00e4e570b37241feda 172.17.0.3:7004
   slots: (0 slots) slave
   replicates 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646
S: 9b0751a4b6ea227658ed26576d9bda1f3b01fc04 172.17.0.2:7002
   slots: (0 slots) slave
   replicates 26776d9aef1c511d77efee79a45a63eff18fb4f9
M: fc25b50d6afe39211a47609287afd250c9e3183b 172.17.0.4:7006
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 4e2f6532200991c93bbf23f7d4996fbd2aff02ac 172.17.0.3:7003
   slots: (0 slots) slave
   replicates fc25b50d6afe39211a47609287afd250c9e3183b
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
复制代码

接下来将 7008 添加为 7007 的 Slave：

[root@centos-01 ~]# redis-cli --cluster add-node 172.17.0.5:7008 172.17.0.2:7001 
        --cluster-slave --cluster-master-id bca31339ca8d8177e9d80831d72b64bc52fdbb87
>>> Adding node 172.17.0.5:7008 to cluster 172.17.0.2:7001
>>> Performing Cluster Check (using node 172.17.0.2:7001)
......
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 172.17.0.5:7008 to make it join the cluster.
Waiting for the cluster to join

>>> Configure node as replica of 172.17.0.5:7007.
[OK] New node added correctly.
复制代码

此时再检查集群状态，可以看到 7008 已经变为 7007 的从库了：

[root@centos-01 ~]# redis-cli --cluster check 172.17.0.2:7001
172.17.0.2:7001 (9b4f3c2c...) -> 0 keys | 5461 slots | 1 slaves.
172.17.0.4:7005 (26776d9a...) -> 0 keys | 5461 slots | 1 slaves.
172.17.0.5:7007 (bca31339...) -> 0 keys | 0 slots | 1 slaves.
172.17.0.4:7006 (fc25b50d...) -> 1 keys | 5462 slots | 1 slaves.
[OK] 1 keys in 4 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 172.17.0.2:7001)
M: 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 172.17.0.2:7001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: 26776d9aef1c511d77efee79a45a63eff18fb4f9 172.17.0.4:7005
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: bca31339ca8d8177e9d80831d72b64bc52fdbb87 172.17.0.5:7007
   slots: (0 slots) master
   1 additional replica(s)
S: 5bfc64bbce07a872d6d6048c62e80f7a8e56990a 172.17.0.5:7008
   slots: (0 slots) slave
   replicates bca31339ca8d8177e9d80831d72b64bc52fdbb87
S: 9a627cad72e4192802237c00e4e570b37241feda 172.17.0.3:7004
   slots: (0 slots) slave
   replicates 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646
S: 9b0751a4b6ea227658ed26576d9bda1f3b01fc04 172.17.0.2:7002
   slots: (0 slots) slave
   replicates 26776d9aef1c511d77efee79a45a63eff18fb4f9
M: fc25b50d6afe39211a47609287afd250c9e3183b 172.17.0.4:7006
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 4e2f6532200991c93bbf23f7d4996fbd2aff02ac 172.17.0.3:7003
   slots: (0 slots) slave
   replicates fc25b50d6afe39211a47609287afd250c9e3183b
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
复制代码

但此时新加入的节点还没有分配哈希槽，我们可以使用 reshard 或 rebalance 命令分配一些槽位到新节点上。例如从 7001（...2f5ca7bf646）上移动2000个哈希槽到 7007（...c52fdbb87）上：

[root@centos-01 ~]# redis-cli --cluster reshard 172.17.0.2:7001 
        --cluster-from 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 
        --cluster-to bca31339ca8d8177e9d80831d72b64bc52fdbb87 
        --cluster-slots 2000
复制代码

再检查集群状态，发现 7007 上已经有哈希槽了：

[root@centos-01 ~]# redis-cli --cluster check 172.17.0.2:7001
172.17.0.2:7001 (9b4f3c2c...) -> 0 keys | 3461 slots | 1 slaves.
172.17.0.4:7005 (26776d9a...) -> 0 keys | 5461 slots | 1 slaves.
172.17.0.5:7007 (bca31339...) -> 0 keys | 2000 slots | 1 slaves.
172.17.0.4:7006 (fc25b50d...) -> 1 keys | 5462 slots | 1 slaves.
[OK] 1 keys in 4 masters.
...
[OK] All 16384 slots covered.
复制代码

集群缩容

下面演示删除 172.17.0.5:7007（bca31339ca8d8177e9d80831d72b64bc52fdbb87）节点。

要删除集群中某个节点，需先将集群上所有的哈希槽移到其它节点后才可以删除，否则会报错：

First use the reshardcommand to move the hash slot of the 7007(...c52fdbb87) node to 7001(...2f5ca7bf646):

[root@centos-01 ~]# redis-cli --cluster reshard 172.17.0.2:7001 
        --cluster-from bca31339ca8d8177e9d80831d72b64bc52fdbb87
        --cluster-to 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 
        --cluster-slots 2000
复制代码

At this point, deleting the 7007 node can be successfully executed:

[root@centos-01 ~]# redis-cli --cluster del-node 172.17.0.2:7002 bca31339ca8d8177e9d80831d72b64bc52fdbb87
>>> Removing node bca31339ca8d8177e9d80831d72b64bc52fdbb87 from cluster 172.17.0.2:7002
>>> Sending CLUSTER FORGET messages to the cluster...
>>> Sending CLUSTER RESET SOFT to the deleted node.
复制代码

The original slave node 7008 will be automatically mounted to other masters. You can see that 7008 becomes a slave of 7001:

[root@centos-01 ~]# redis-cli --cluster check 172.17.0.2:7001
172.17.0.2:7001 (9b4f3c2c...) -> 0 keys | 5461 slots | 2 slaves.
172.17.0.4:7005 (26776d9a...) -> 0 keys | 5461 slots | 1 slaves.
172.17.0.4:7006 (fc25b50d...) -> 1 keys | 5462 slots | 1 slaves.
[OK] 1 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 172.17.0.2:7001)
M: 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 172.17.0.2:7001
   slots:[0-5460] (5461 slots) master
   2 additional replica(s)
M: 26776d9aef1c511d77efee79a45a63eff18fb4f9 172.17.0.4:7005
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 5bfc64bbce07a872d6d6048c62e80f7a8e56990a 172.17.0.5:7008
   slots: (0 slots) slave
   replicates 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646
S: 9a627cad72e4192802237c00e4e570b37241feda 172.17.0.3:7004
   slots: (0 slots) slave
   replicates 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646
S: 9b0751a4b6ea227658ed26576d9bda1f3b01fc04 172.17.0.2:7002
   slots: (0 slots) slave
   replicates 26776d9aef1c511d77efee79a45a63eff18fb4f9
M: fc25b50d6afe39211a47609287afd250c9e3183b 172.17.0.4:7006
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 4e2f6532200991c93bbf23f7d4996fbd2aff02ac 172.17.0.3:7003
   slots: (0 slots) slave
   replicates fc25b50d6afe39211a47609287afd250c9e3183b
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
复制代码

Of course we can also delete the 7008 node:

[root@centos-01 ~]# redis-cli --cluster del-node 172.17.0.2:7002 5bfc64bbce07a872d6d6048c62e80f7a8e56990a
>>> Removing node 5bfc64bbce07a872d6d6048c62e80f7a8e56990a from cluster 172.17.0.2:7002
>>> Sending CLUSTER FORGET messages to the cluster...
>>> Sending CLUSTER RESET SOFT to the deleted node.
复制代码

Redis series (4) - sliced cluster