redis cluster expansion

background introduction

During the double 11 period of e-commerce projects, there are usually more concurrency, and the pressure on redis requests surges, requiring more redis to provide external services. However, after double 11, the pressure on redis drops sharply. In order to save costs, some parts need to be withdrawn redis service! We can realize the dynamic expansion and shrinkage of the redis cluster by adding and deleting redis cluster nodes, and increase the high availability of the redis cluster.
The original cluster (see the figure below) consists of 6 nodes, and the 6 nodes are distributed on three machines (this case is only demonstrated on one machine!), adopting the mode of three masters and three slaves.

In order to simulate cluster expansion, we add a master (8007) and a slave (8008) on the basis of the original cluster. See the figure below for the cluster after adding nodes

. Cluster, and observe whether the cluster status is normal. If it is normal, the capacity can be expanded or reduced.
After the startup is successful , cluster nodescheck the node status through the command

. As can be seen from the above figure, the entire cluster is running normally. There are three master nodes and three slave nodes. The instance nodes on port 8001 store hash slots from 0 to 5460, and the instance nodes on port 8002 store 5461 -10922 these hash slots, the instance node on port 8003 stores these hash slots 10923-16383, all the hash slots stored by these three master nodes form the storage slots of the redis cluster, and the slave point is the backup slave node of each master node, not Show storage slots!

2. Redis cluster expansion

After the original cluster is started and confirmed to be healthy, we add a master (8007) and a slave (8008) to the original cluster to achieve cluster expansion. The steps are as follows:

2.1 Add redis instance

Create folders 8007 and 8008 under /usr/local/redis-cluster, and copy the redis.conf file under folder 8001 to folders 8007 and 8008. The specific steps are as follows:

mkdir 8007 8008
cd 8001
cp redis.conf /usr/local/redis-cluster/8007/
cp redis.conf /usr/local/redis-cluster/8008/

# 修改8007文件夹下的redis.conf配置文件
vim /usr/local/redis-cluster/8007/redis.conf
# 修改如下内容：
port:8007
dir /usr/local/redis-cluster/8007/
cluster-config-file nodes-8007.conf
pidfile /var/run/redis_8007.pid


# 修改8008文件夹下的redis.conf配置文件
vim /usr/local/redis-cluster/8008/redis.conf
修改内容如下：
port:8008
dir /usr/local/redis-cluster/8008/
cluster-config-file nodes-8008.conf
pidfile /var/run/redis_8008.pid

# 启动8007和8008俩个服务并查看服务状态
/usr/local/redis-5.0.3/src/redis-server /usr/local/redis-cluster/8007/redis.conf
/usr/local/redis-5.0.3/src/redis-server /usr/local/redis-cluster/8008/redis.conf
ps -el | grep redis

After the startup is complete, the two nodes 8007 and 8008 are still in a free state and have not yet joined the redis cluster!

2.2 Configure 8007 as the master node

Use the add-node command of redis-cli to add a new master node 8007 (master), the previous ip:port is the new node, and the latter ip:port is the existing node in the cluster.

src/redis-cli --cluster add-node 192.168.100.100:8007 192.168.100.100:8001

This process is operated through the meet command of the gossip protocol!

meet：某个节点发送meet给新加入的节点，让新节点加入集群中，然后新节点就会开始与其他节点进行通信；

Finally, you can see that there is a prompt of "[OK] New node added correctly" at the end of the log, which means that the new node has been added

successfully. After the addition is successful, check the node status through the cluster nodes command as follows

Note: When the node is added successfully, the newly added node will not have any data. Since it has not allocated any slots (hash slots), we need to manually allocate hash slots for new nodes. Use the rehash command of redis-cli to allocate hash slots for 8007, find any master node in the cluster, and re-shard it.

src/redis-cli --cluster reshard 192.168.100.100:8001

After executing the above command, it will enter the process of manually allocating slots. The allocation details are as follows:

... ...
How many slots do you want to move (from 1 to 16384)? 600
(ps:需要多少个槽移动到新的节点上，自己设置，比如600个hash槽)
What is the receiving node ID? 2728a594a0498e98e4b83a537e19f9a0a3790f38
(ps:把这600个hash槽移动到哪个节点上去，需要指定节点id)
Please enter all the source node IDs.(Please enter all the source node IDs)
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.(输入'all'以使用所有节点作为散列槽的源节点。输入所有源节点id后，键入'done'。)
Source node 1:all
(ps:输入all为从所有主节点(8001,8002,8003)中分别抽取相应的槽数指定到新节点中，抽取的总槽数为600个)
 ... ...
Do you want to proceed with the proposed reshard plan (yes/no)? yes
(ps:输入yes确认开始执行分片任务)
... ...

After the slot allocation is complete, run the cluster nodes command again to view the node status as follows:

Note: After the slot is migrated, the data in the corresponding slot will also be migrated!
So far! 8007Master node joined the cluster successfully!

2.3 Configure 8008 as the slave node of 8007

Repeat the 8008 node to join the cluster command add-node

src/redis-cli --cluster add-node 192.168.100.100:8008 192.168.100.100:8001

Use the cluster nodes command to view the node status

. You can see that 8008 also has no slot allocation, but since the 8008 node is to be the slave node of the 8007, there is no need to allocate slots to the 8008 node. To configure the master-slave relationship for the 8008 node, you need to connect to the client of the 8008 node, and then use the cluster command replicate to assign the current 8008 (slave) node to a master node (the previously created 8007 master node is used here), the command is as follows :

# 进入8008的客户端
[root@CentOS7 redis-6.0.9]# src/redis-cli  -p 8008

# 在8008客户端下指定8008节点的主从关系
# 4b339ad25b4884c2ff6de8a8ec2bc8766f8faf0b 是8007节点的id
127.0.0.1:8008> cluster replicate 4b339ad25b4884c2ff6de8a8ec2bc8766f8faf0b

Then check the node status through the cluster nodes command,

you can see that 8008 has become the slave node of 8007, and the expansion of the cluster nodes has been completed!

3. Redis cluster shrink

Redis cluster shrinking is actually deleting some cluster nodes. Here we test cluster shrinking by deleting the cluster nodes composed of 8007 and 8008 added above!

3.1 Return slot data first

Because the master node 8007 has allocated hash slots, we must first put the hash slots in 8007 into other available master nodes, and then remove the node, otherwise data loss will occur (Currently, the data of the master can only be migrated to one node, and the average distribution function cannot be performed temporarily), and the execution command is as follows:

src/redis-cli  --cluster reshard 192.168.100.100:8007

 ... ...
How many slots do you want to move (from 1 to 16384)? 600
What is the receiving node ID? baf0c2f3afde2410e34351a8261a703f1394cee9
(ps:这里是需要把数据移动到哪？8001的主节点id)
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node 1:4b339ad25b4884c2ff6de8a8ec2bc8766f8faf0b
(ps:这里是需要数据源，也就是我们的8007节点id)
Source node 2:done
(ps:这里直接输入done 开始生成迁移计划)
 ... ...
Do you want to proceed with the proposed reshard plan (yes/no)? Yes
(ps:这里输入yes开始迁移)

So far, we have successfully migrated the data of the 8007 master node to 8001. We can take a look at the current cluster status as shown in the figure below, and you will find that there is no hash slot under 8007, which proves that the migration was successful!

3.2 Delete node 8007

Finally, we can directly use the del-node command to delete the 8007 master node

src/redis-cli  --cluster del-node 192.168.100.100:8007 4b339ad25b4884c2ff6de8a8ec2bc8766f8faf0b

Finally, use the cluster nodes command to check the node status. If it is restored to the original state, it means that the shrinkage is successful!

4. Redis - Solve the bug in reshard: Syntax error, try CLIENT (LIST|KILL|GETNAME|SETNAME|PAUSE|REPLY）

When migrating a slot with a key-value, redis-trib.rb reshardan error occurs when the command is executed:
the content of the error is: Syntax error, try CLIENT (LIST|KILL|GETNAME|SETNAME|PAUSE|REPLY)
but when migrating a slot without a key-value will execute successfully. This shows that the problem lies in the existence of key-value.

solution

Just modify the migration statement in the rb file to:
source.r.call(["migrate",target.info[:host],target.info[:port],"",0,@timeout,"replace",: keys,*keys])
source.r.call(["migrate",target.info[:host],target.info[:port],"",0,@timeout,:replace,:keys,*keys] )
that is, execute migrateCommand directly without executing clientCommand.