Three machines build redis cluster process and problem records

Article directory

1 Introduction

Redis version 5.0.4
server version Linux CentOS 6; CentOS 7; CentOS 9;
redis cluster requires at least three master nodes, we build three master nodes here, and build a slave node for each master, a total of 6 redis Node, three masters and three slaves.

2. Build a cluster

Install redis Linux on all three machines to install and deploy Redis
https://blog.csdn.net/craftsman2020/article/details/122851974
Create a redis-cluster directory
Create a redis-cluster directory in the redis installation directory (/usr/local/redis/) of each server

cd  /usr/local/redis/
mkdir  redis-cluster

Create 8001 and 8002 respectively under redis-cluster

cd /usr/local/redis/redis-cluster
mkdir 8001 8002

The same operation as the other two machines, the ports are 8003 8004, 8005 8006

2.3 Copy the configuration file
Copy the previous redis.conf to the 8001 directory

cp /usr/local/redis/redis.conf  /usr/local/redis/redis-cluster/8001/

Do the same for the other 5 directories

Modify redis.conf

1）daemonize yes
2）port 8001（分别对每个机器的端口号进行设置）
3）dir /usr/local/redis/redis-cluster/8001/（指定数据文件存放位置，必须要指定不同的目录位置，不然会丢失数据）
4）cluster-enabled yes（启动集群模式）
5）cluster-config-file nodes-8001.conf（集群节点信息文件，这里800x最好和port对应上）
6）cluster-node-timeout 5000
7) bind 127.0.0.1（去掉bind绑定访问ip信息, 注释掉，或改为0.0.0.0也可）
8) protected-mode no （关闭保护模式）
9）appendonly yes
如果要设置密码需要增加如下配置：
10）requirepass xxx (设置redis访问密码)
11）masterauth xxx (设置集群节点间访问密码，跟上面一致)

Copy the modified redis.conf configuration file of 8001 to 8002-8006 respectively, and modify the corresponding port and file name

cp /usr/local/redis/redis-cluster/8001/redis.conf /usr/local/redis/redis-cluster/8002/
scp /usr/local/redis/redis-cluster/8001/redis.conf [email protected]:/usr/local/redis/redis-cluster/8003/
scp /usr/local/redis/redis-cluster/8001/redis.conf [email protected]:/usr/local/redis/redis-cluster/8004/
scp  /usr/local/redis/redis-cluster/8001/redis.conf [email protected]:/usr/local/redis/redis-cluster/8005/
scp  /usr/local/redis/redis-cluster/8001/redis.conf [email protected]:/usr/local/redis/redis-cluster/8006/

The newly copied five files only modify the configuration items: port, dir, cluster-config-file

Start 6 redis instances separately

redis-server /usr/local/redis/redis-cluster/8001/redis.conf
redis-server /usr/local/redis/redis-cluster/8002/redis.conf
redis-server /usr/local/redis/redis-cluster/8003/redis.conf
redis-server /usr/local/redis/redis-cluster/8004/redis.conf
redis-server /usr/local/redis/redis-cluster/8005/redis.conf
redis-server /usr/local/redis/redis-cluster/8006/redis.conf

create cluster

redis-cli -a 123456 --cluster create --cluster-replicas 1 192.168.2.58:8001 192.168.2.58:8002 192.168.2.59:8003 192.168.2.59:8004 192.168.1.60:8005 192.168.1.60:8006

Verify the cluster
Connect to any node of machine A

redis-cli -a 123456 -c -h 192.168.2.58 -p 8001  -a 123456

insert image description here

View cluster information
View cluster nodes
Shut down the cluster

redis-cli -a 123456 -c -h 192.168.2.58 -p 8001 shutdown
redis-cli -a 123456 -c -h 192.168.2.58-p 8002 shutdown
redis-cli -a 123456 -c -h 192.168.2.59 -p 8003 shutdown
redis-cli -a 123456 -c -h 192.168.2.59 -p 8004 shutdown
redis-cli -a 123456 -c -h 192.168.1.60 -p 8005 shutdown
redis-cli -a 123456 -c -h 192.168.1.60 -p 8006 shutdown

Shutting down and starting the cluster with scripts
Shut down the cluster with a script

#!/bin/bash
#所有服务器节点的hostname
allnodes=('10.12.2.59' '10.12.2.58' '192.168.1.60')
local_ip=`ifconfig -a|grep inet|grep -v 127.0.0.1|grep -v inet6|awk '{print $2}'|tr -d "addr:"`
echo "### " date
echo "### local_ip: ${local_ip}"
PORT=8001
ENDPORT=8007
PASSWROD=123456
while [ $((PORT < ENDPORT)) != "0" ]; do
    for ip in ${
    
    allnodes[@]};
    do
        if [ "$ip" = "$local_ip" ];
        then 
            local_count=`ps -ef|grep redis |grep $PORT | wc -l`
            if [ $local_count -gt 0 ];
            then
                echo "### Stoping Local Redis $ip:$PORT"
                redis-cli -p $PORT -a $PASSWROD shutdown
            else 
                echo "no Local redis $ip:$PORT"
            fi
        else
            #判断某个端口是否已被占用，如果是，则执行关闭命令
            count=`ssh root@$ip lsof -i:$PORT | wc -l`
            if [ $count -gt 0 ];
            then
                echo "### Stopping Redis $ip:$PORT"
                ssh root@$ip redis-cli -p $PORT -a $PASSWROD shutdown 2>/dev/null
            else
                echo "no redis $ip:$PORT"
            fi
        fi
    done
    PORT=$((PORT+1))
done
exit 0

Start the cluster with a script

#!/bin/bash
#所有服务器节点的hostname
declare -A dic
dic=([8001]="10.12.2.58" [8002]="10.12.2.58" [8003]="10.12.2.59" [8004]="10.12.2.59" [8005]="192.168.1.60" [8006]="192.168.1.60")

local_ip=`ifconfig -a|grep inet|grep -v 127.0.0.1|grep -v inet6|awk '{print $2}'|tr -d "addr:"`
echo "### " date
echo "### local_ip: ${local_ip}"

for port in ${
    
    !dic[*]}
do 
    ip=${
    
    dic[${
    
    port}]}
    #echo ip: ${ip} ${port}
    if [ "$ip" = "$local_ip" ];
    then 
        local_count=`ps -ef|grep redis |grep $port | wc -l`
        if [ $local_count -eq 0 ];
        then
            echo "Start Local Redis Server $ip $port"
            redis-server /usr/local/redis/redis-cluster/$port/redis.conf
        else
            echo "Local Redis Server $ip $port already exists!"
        fi
    else
        count=`ssh root@$ip ps -ef|grep redis-server|grep -v grep|grep $port| wc -l`
        if [ $count -eq 0 ];
        then
            echo "Start Redis Server $ip $port"
            ssh root@$ip redis-server /usr/local/redis/redis-cluster/$port/redis.conf
        else
            echo "Redis Server $ip $port already exists!"
        fi
    fi    
done
exit 0

Notice:

Each cluster is set to start up. For the setting method, please refer to https://blog.csdn.net/craftsman2020/article/details/128130348?spm=1001.2014.3001.5502
In order to facilitate the start and termination of the program, use shell scripts.
The operating system versions of each machine in the cluster should be coordinated as much as possible to avoid the problem that Centos6 and Centos9 cannot be accessed by ssh.

3. Problems encountered

The three machines set up password-free mutual visits
. However, due to the large differences in the centos versions of the three machines, Centos9 and Centos6 cannot set secret-free mutual visits.
If you set password-free and report an error, you will also report an error if you connect directly with the ssh command:

try to create a config file (with no suffix) in the folder where the public key is generated (usually in the .ssh file under the current user directory), and open it in text document format , add the following

Host *
HostkeyAlgorithms +ssh-rsa
PubkeyAcceptedKeyTypes +ssh-rsa

Reference:
https://blog.csdn.net/weixin_51443484/article/details/125944815
https://weibo.com/ttarticle/p/show?id=2309404806141325738316
But it has not been resolved.

Then set password-free for centos9 on centos6:

ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected]

The error is as follows:

no hostkey alg

On centos9, set the password-free setting for centos6 and report the error as follows:

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: ERROR: Unable to negotiate with 192.168.1.30 port 22: no matching host key type found. Their offer: ssh-rsa,ssh-dss

After that, a compromise was adopted, because Centos7 can set password-free for the other two scripts, so the scripts for starting and terminating the redis cluster were placed on Centos7.

Error when connecting to redis cluster: (error) MOVED solution

./redis-cli -h 192.168.2.58 -p 8001-a 123456

192.168.2.58 :8001 > get name

(error) MOVED 5798 192.168.2.58 :8001

This situation is generally caused by not setting the cluster mode when starting redis-cli.

Use the -c parameter to start the cluster mode at startup, the command is as follows:

./redis-cli -h 192.168.2.58 -p 8001 -a 123456 -c

192.168.2.58:8001 > get name

-> Redirected to slot [5798] located at 192.168.2.58 :8001

(error) READONLY You can't write against a read only replica.
If the above error occurs, it means that the current redis service is read-only and has no write permission. It is estimated that the service is used as a slave database.
When doing write operations on the slave nodes of the redis cluster, the above error will be reported.
Redis (error) CROSSSLOT Keys in request don’t hash to the same slot
参考 https://blog.csdn.net/TreeShu321/article/details/102766814