Redis5.0+ Redis cluster horizontal expansion | node deletion

1. Background

I built a Redis cluster environment before, with 3 masters and 3 slaves, a total of six nodes (three masters and three slaves to build a redis cluster: https://blog.csdn.net/craftsman2020/article/details/128175886) Now I want to
expand horizontally and delete 1 master and 1 slave.

2. Redis cluster horizontal expansion

2.1 Start the cluster

./redis-server ../redis-cluster/8001/redis-8001.conf
./redis-server ../redis-cluster/8002/redis-8002.conf
./redis-server ../redis-cluster/8003/redis-8003.conf
./redis-server ../redis-cluster/8004/redis-8004.conf
./redis-server ../redis-cluster/8005/redis-8005.conf
./redis-server ../redis-cluster/8006/redis-8006.conf

2.2 Connect to one of the nodes

redis-cli -p 8003 -h 192.168.1.51 -c -a root

# 连接成功后，查看集群节点信息
192.168.1.51:8003> CLUSTER NODES

Message as follows:

192.168.1.51:8003> cluster nodes
af4c1669c8cf1de35836ebeeec294227ab567120 192.168.1.52:8005@18005 master - 0 1685081492000 5 connected 10923-16383
96c7ce1ea3a57ad49a75e857c676aee8f2fb6011 192.168.1.50:8001@18001 slave 06683de7cf7c415b907913e5d4aece5a8091e9ed 0 1685081493000 9 connected
06683de7cf7c415b907913e5d4aece5a8091e9ed 192.168.1.51:8004@18004 master - 0 1685081493000 9 connected 0-5460
ed604208cf43e7b2490ff2640ae17e0ff590ef5e 192.168.1.51:8003@18003 myself,slave bd311076d5e70d5b8e4cfef073c6298a736c7880 0 1685081492000 3 connected
bd311076d5e70d5b8e4cfef073c6298a736c7880 192.168.1.52:8006@18006 master - 0 1685081493812 8 connected 5461-10922
72999f45b138940d65692e8f9c9f52d75df64fff 192.168.1.50:8002@18002 slave af4c1669c8cf1de35836ebeeec294227ab567120 0 1685081491806 5 connected

From the above information, it can be seen that the whole cluster is running normally. There are three master nodes and three slave nodes. The instance nodes on port 8004 store the hash slots 0-5460, the instance nodes on port 8006 store the hash slots 5461-10922, and port 8005 The instance nodes store the hash slots 10923-16383. All the hash slots stored by the three master nodes form the storage slots of the redis cluster. The slave point is the backup slave node of each master node, and the storage slots are not displayed.

Or check the cluster status with the following command:

../src/redis-cli --cluster info 192.168.1.50:8001

show:

[root@192 cluster]# ../src/redis-cli --cluster info 192.168.1.50:8001
192.168.1.52:8006 (a9e25213...) -> 0 keys | 5461 slots | 1 slaves.
192.168.1.51:8004 (579d2e92...) -> 0 keys | 5462 slots | 1 slaves.
192.168.1.52:8005 (78b51db2...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.

2.3 Configure new nodes

Here I created the 8007 and 8008 folders under the same path as the cluster configuration file on the 192.168.1.53 service, and copied the 8001 configuration file to 8007/redis-8007.conf and 8008/redis-8008.conf, and related to 800x The configuration is modified.
The modification of the configuration file here is the same as that of the previous cluster configuration file, and the related configuration of the cluster configuration file 800x can be modified.

2.4 Starting a new node

./redis-server ../redis-cluster/8007/redis-8007.conf
./redis-server ../redis-cluster/8008/redis-8008.conf

# 查看redis进程
ps -ef|grep redis

The redis process of the server where 8007 and 8008 are located

[[email protected] ~]$ ps -ef|grep redis
root     3166220       1  0 May25 ?        00:01:46 redis-server *:8008 [cluster]
root     3234303       1  0 May25 ?        00:02:42 redis-server *:8007 [cluster]

2.5 Configure node 8007 to join the cluster and become the master node

Add the master node with the following command.

redis-cli --cluster add-node new_host:new_port existing_host:existing_port --cluster-master-id node_id

example

/usr/local/redis/bin/redis-cli -a 123456 --cluster add-node 192.168.1.53:8007 192.168.1.52:8005 --cluster-master-id 4568a2560a688898a5d2337bce3a288f12355ae8

• new_host:new_port: IP and port of the master node to be newly added
• existing_host:existing_port: Indicates the IP and port of the last master node that exists in the environment. This can be known by viewing the node information. According to the number of slots, the number of node slots corresponding to 192.168.1.52:8005 is 10923-16383. 16383 represents the last slot number
• –cluster-master-id: Indicates the node ID of the last master node, indicating that the newly added master node will be behind this node

Redis cluster command help reference【4.redis cluster command help】

# 将8007加入集群，并成为主节点
./redis-cli --cluster add-node 192.168.1.53:8007 192.168.1.53:8001 -a root

Join successfully!
Since I didn’t save screenshots one by one when I added nodes, I found similar results on the Internet as follows (quoted from the Internet): From the
above information, it can be seen that the hash slots are: 0-5460, 5464-10922, 10923- 16380 (2 to the 14th power).
View cluster status:

./redis-cli -h 192.168.1.50 -p 8001 -a root -c

# 查看集群状态
cluster nodes

Since I did not save screenshots one by one when I added nodes, I found similar results on the Internet as follows (quoted from the Internet):
Note: After the node is successfully added, the newly added node will not have any data, because it has not been assigned any slot (hash slot), we need to manually assign hash slots for new nodes. The newly added node also has no child nodes. We can assign hash slots to it first, or add child nodes to it first.

2.6 Add child nodes

Order

redis-cli --cluster add-node new_host:new_port existing_host:existing_port --cluster-slave --cluster-master-id node_id

example

/usr/local/redis/bin/redis-cli -a 123456 --cluster add-node 192.168.1.53:8007 192.168.1.53:8008 --cluster-slave --cluster-master-id 66f3cbc063a84ff3c209fa2db7104e4666d82e9c

• new_host:new_port: Indicates the IP and port of the slave node to be added
• existing_host:existing_port: Indicates which master node to add a slave node to
• –cluster-slave: means to add a slave node
• –cluster-master-id node_id: Indicates which master node to add a slave node to, the master node node ID

66f3cbc063a84ff3c209fa2db7104e4666d82e9c displayed after –cluster-master-id is the ID of 8007, which can be obtained from the information in the previous step.

After adding child nodes, check the redis cluster again

../src/redis-cli --cluster check 192.168.1.50:8001

Or view it with the following command

# 连接8007主节点
./redis-cli -h 192.168.1.53 -p 8007 -a root -c
# 查看集群信息
cluster nodes

Since I did not save screenshots one by one when adding nodes, I found similar results on the Internet as follows (quoted from the Internet)

2.7 Assign hash slot

Next we need to allocate hash slots for 8007

../redis-cli -a root --cluster reshard 192.169.1.53:8007

[root@192 cluster]# ../src/redis-cli --cluster reshard 192.168.1.53:8007
>>> Performing Cluster Check (using node 192.168.1.53:8007)
M: 3067f715eaa5c88c260fb2e7f21edc06112208d3 192.168.1.53:8007
   slots: (0 slots) master
   1 additional replica(s)
M: 579d2e9233faa12d6e13dfba8fe37f7e1c9983f8 192.168.1.50:8002
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 9e266605685ab453eccdb6042327a7e187cee7c3 192.168.1.51:8004
   slots: (0 slots) slave
   replicates 579d2e9233faa12d6e13dfba8fe37f7e1c9983f8
S: 1780235a26ce1ed943fcdff24c509ebb27c2d03e 192.168.1.52:8005
   slots: (0 slots) slave
   replicates 78b51db2871be7fa265db054dd79cded9e97c4ff
S: b6f7f1df8c16039c764d4c259182515139263e84 192.168.1.52:8006
   slots: (0 slots) slave
   replicates a9e25213d46b7d19bc46931b0e2490a93c8237ac
S: 5ef427a8a8c752301826e3f88e3a7282ff75a537 192.168.1.53:8008
   slots: (0 slots) slave
   replicates 3067f715eaa5c88c260fb2e7f21edc06112208d3
M: 78b51db2871be7fa265db054dd79cded9e97c4ff 192.168.1.51:8003
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: a9e25213d46b7d19bc46931b0e2490a93c8237ac 192.168.1.50:8001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 

Input (4 master nodes, 16384 hash slots in total, average per master node: 4096): 4096
Display:

[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 4096
What is the receiving node ID? 

Input (id of 6007): 3067f715eaa5c88c260fb2e7f21edc06112208d3
shows:

[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 4096
What is the receiving node ID? 3067f715eaa5c88c260fb2e7f21edc06112208d3
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1: 

Input (from which nodes will be assigned to 8007): all
shows:

    ***
    Moving slot 1359 from a9e25213d46b7d19bc46931b0e2490a93c8237ac
    Moving slot 1360 from a9e25213d46b7d19bc46931b0e2490a93c8237ac
    Moving slot 1361 from a9e25213d46b7d19bc46931b0e2490a93c8237ac
    Moving slot 1362 from a9e25213d46b7d19bc46931b0e2490a93c8237ac
    Moving slot 1363 from a9e25213d46b7d19bc46931b0e2490a93c8237ac
    Moving slot 1364 from a9e25213d46b7d19bc46931b0e2490a93c8237ac
Do you want to proceed with the proposed reshard plan (yes/no)?

Input: yes
So far, the allocation of hash slots has been completed.

2.8 Check Redis Cluster

redis-cli --cluster check 192.168.1.53：8007

Display:
8007 has 4096 hash slots and has one child node.

(base)[root@192-cluster]$redis-cli -a 123456 --cluster check 10.12.1.53：8007
10.12.1.53：8007 (84d4bc3e...) -> 127 keys | 4096 slots | 1 slaves.
10.12.1.51:8004 (06683de7...) -> 128 keys | 4096 slots | 1 slaves.
192.168.1.52:8005 (af4c1669...) -> 151 keys | 4096 slots | 1 slaves.
192.168.1.52:8006 (bd311076...) -> 135 keys | 4096 slots | 1 slaves.
[OK] 541 keys in 4 masters.
0.03 keys per slot on average.
>>> Performing Cluster Check (using node 10.12.1.53:8007)
M: 84d4bc3e46a829df205f645867468f5ec44682e9 10.12.1.53:8007
   slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master
   1 additional replica(s)
M: 06683de7cf7c415b907913e5d4aece5a8091e9ed 10.12.1.51:8004
   slots:[1365-5460] (4096 slots) master
   1 additional replica(s)
M: af4c1669c8cf1de35836ebeeec294227ab567120 192.168.1.52:8005
   slots:[12288-16383] (4096 slots) master
   1 additional replica(s)
S: 72999f45b138940d65692e8f9c9f52d75df64fff 10.12.1.50:8002
   slots: (0 slots) slave
   replicates af4c1669c8cf1de35836ebeeec294227ab567120
S: ed604208cf43e7b2490ff2640ae17e0ff590ef5e 10.12.1.51:8003
   slots: (0 slots) slave
   replicates bd311076d5e70d5b8e4cfef073c6298a736c7880
S: bf352d47f8c4559b3bf54aecc0f2f75687eac4e2 10.12.1.53:8008
   slots: (0 slots) slave
   replicates 84d4bc3e46a829df205f645867468f5ec44682e9
M: bd311076d5e70d5b8e4cfef073c6298a736c7880 192.168.1.52:8006
   slots:[6827-10922] (4096 slots) master
   1 additional replica(s)
S: 96c7ce1ea3a57ad49a75e857c676aee8f2fb6011 10.12.1.50:8001
   slots: (0 slots) slave
   replicates 06683de7cf7c415b907913e5d4aece5a8091e9ed
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

3. Node deletion

Here we mainly introduce the deletion of the master node. Deleting the master node is a little troublesome, because the master node has allocated slots, so you must put the slots in other available nodes first, and then remove the node, otherwise data loss will occur. The deletion of the slave node is very simple. In the process of deleting the master node, after the hash slot is transferred to other master nodes, the master node without the hash slot is finally deleted. The operation of deleting a slave node is the same as deleting a master node without a hash slot.

3.1 Resharding

Move the data to other primary nodes and execute the re-sharding command.

redis-cli -a 111111--cluster reshard xx.xx.xx.xx:8005

xx.xx.xx.xx:8005 has allocated 4096 slots, just enter 4096 here.
After pressing Enter, what is the receiving node ID? appears, which means you want to move to that node.
I want to move to node 8003, so enter the ID of node 8003 here.

After pressing Enter, you can continue to select other source nodes, but here I just want to assign node 8005 to 8003, just enter done here, otherwise enter the ID of other nodes, and finally enter done.
Note: Other source nodes must be master nodes.

Finally, enter yes to wait for the transfer to complete.

# 再次查看集群信息
redis-cli -c -a 111111-h xx.xx.xx.xx -p 8001cluster nodes
# 发现 8005已经没有 slots 槽分配了

The reason why it is fail in the picture is because I have killed the redis process of this node. It doesn't matter if you don't need to kill.

3.2 Delete node

Next, call the way to delete the slave node to delete the master node.

redis-cli -a 111111 --cluster del-node xx.xx.xx.xx:8005 af4c1669c8cf1de35836ebeeec294227ab567120

# 再次查看集群信息
redis-cli -c -a 111111 -h xx.xx.xx.xx -p 8001 cluster nodes
# 环境中 8005主节点已被移除

4. Redis cluster command help

# 查看redis集群的命令帮助
./redis‐cli ‐‐cluster help

[root@192]$ redis-cli --cluster help
Cluster Manager Commands:
  create         host1:port1 ... hostN:portN
                 --cluster-replicas <arg>
  check          host:port
                 --cluster-search-multiple-owners
  info           host:port
  fix            host:port
                 --cluster-search-multiple-owners
  reshard        host:port
                 --cluster-from <arg>
                 --cluster-to <arg>
                 --cluster-slots <arg>
                 --cluster-yes
                 --cluster-timeout <arg>
                 --cluster-pipeline <arg>
                 --cluster-replace
  rebalance      host:port
                 --cluster-weight <node1=w1...nodeN=wN>
                 --cluster-use-empty-masters
                 --cluster-timeout <arg>
                 --cluster-simulate
                 --cluster-pipeline <arg>
                 --cluster-threshold <arg>
                 --cluster-replace
  add-node       new_host:new_port existing_host:existing_port
                 --cluster-slave
                 --cluster-master-id <arg>
  del-node       host:port node_id
  call           host:port command arg arg .. arg
  set-timeout    host:port milliseconds
  import         host:port
                 --cluster-from <arg>
                 --cluster-copy
                 --cluster-replace
  help           

For check, fix, reshard, del-node, set-timeout you can specify the host and port of any working node in the cluster.

• create: create a cluster environment host1:port1 ... hostN:portN
• call: can execute redis command
• add-node: Add a node to the cluster, the first parameter is the ip:port of the new node, and the second parameter is the ip:port of any existing node in the cluster
• del-node: remove a node
• reshard: reshard
• check: Check the status of the cluster

5. References

• CentOS 1908 configuration Redis 5.0.7 five (cluster new master-slave nodes) https://blog.csdn.net/qq_32596527/article/details/104402130
• Redis cluster horizontal expansion and node deletion https://blog.csdn.net/weixin_40955398/article/details/122666132
• Redis5.0+——cluster construction, horizontal expansion https://blog.csdn.net/G_change_/article/details/128448636
• Redis cluster node management (cluster) https://blog.csdn.net/weixin_41753567/article/details/124110749
• [Redis] Analysis of Redis Cluster Architecture (4): Slot Migration and Redistribution
• Expansion and contraction of redis cluster cluster nodes, reallocation of slots
• Three machines build redis cluster process and problem records
• Redis installation, configuration, use and uninstallation in windows environment
• Linux installation and deployment of Redis
• Redis5.0 cluster construction (two servers)
• redis official website