Problem phenomenon
After starting the cluster, the log file displays the following information:
1 |
Unrecoverable error: corrupted cluster config file. |
In this case, the node cannot be started because the cluster-config-file file of the cluster is damaged.
Repair plan
- First remove the faulty node on each node
- Delete the cluster-config-file file (node file)
- Restart the node
- Add the node to the cluster
- Specify the master of the node, and join the node to the cluster as a slave (this repair has not been performed, and the cluster will automatically return to normal)
Implementation steps
1. First, remove the faulty node on each node
Execute the following script to remove the fail node out of the cluster. This operation only removes the useless node information in one machine. If there are multiple machines, please add multiple machine IPs in host_array
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
#!/bin/bash
host_array=(192.168.0.101)
for var in ${host_array[@]};
do
for i in $(seq 7000 7012)
do
nodeids=`src/redis-cli -c -h $var -p $i cluster nodes|grep fail|awk '{print $1}'`
for d in $nodeids
do
echo $d
src/redis-cli -c -h $var -p $i cluster forget $d
done
done
done
|
2. Delete the cluster-config-file file
1 |
mv nodes-7004.conf nodes-7004.conf.bak |
Three, restart the node
1 |
redis-server redis.conf |
Fourth, add the node to the cluster
Execute the following commands in any instance of the cluster
1 |
CLUSTER MEET <ip> <port> //Add the node specified by ip and port to the cluster and make it a part of the cluster |
Five, specify the master of the node, and join the node to the cluster as a slave
In this repair scenario, the fifth step is not performed, and the cluster is automatically assigned to become the slave of the unbalanced node
If the cluster is not automatically allocated, execute it on the instance that needs to be added to the cluster, and specify it as the slave of a certain master
1 |
CLUSTER REPLICATE <node_id> //Set the current node as the slave node of the node specified by node_id. |
--This article is reproduced from http://trumandu.github.io/2017/06/01/RedisCluster-%E9%9B%86%E7%BE%A4%E9%85%8D%E7%BD%AE%E6 %96%87%E4%BB%B6%E6%8D%9F%E5%9D%8F%E4%BF%AE%E5%A4%8D/