Redis distributed cluster environment construction

        Redis distributed clusters can be divided into master- slave clusters , sentinel clusters , and fragmented clusters . In this article, we will introduce the construction of these three cluster environments.

        This build environment is a Linux virtual machine system.

        Before building a redis cluster, you need to install and deploy a stand-alone redis in the linux system (the installation directory is: /usr/local/src ), and the detailed steps can be clicked on the following link: Linux cloud server installation and deployment redis detailed steps

Ⅰ. Redis master-slave cluster construction

1.1 Cluster structure 

The master-slave cluster structure we built is shown in the figure:

There are three nodes in total, one master node and two slave nodes (the master node is used for write operations, and the slave nodes can only perform read operations).

Here we will open 3 redis instances in the same virtual machine to simulate a master-slave cluster. The information is as follows:

IP	PORT	Role
192.168.70.130	7001	master
192.168.70.130	7002	slave
192.168.70.130	7003	slave

1.2 Prepare and configure the instance

To start three instances on the same virtual machine, three different configuration files and directories must be prepared, and the directory where the configuration files are located is also the working directory.

1) Create a directory

We create three folders named 7001, 7002, and 7003:

# 进入 /usr/local/src 目录
cd /usr/local/src
# 创建目录
mkdir 7001 7002 7003

As shown in the picture:

2) Turn on RDB and turn off AOF (by default)

Modify the redis-6.2.4/redis.conf file, change the persistence mode to the default RDB mode, and keep AOF off

# 开启RDB
# save ""
save 3600 1
save 300 100
save 60 10000

# 关闭AOF
appendonly no

3) Copy the configuration file to each instance directory

# 方式一：逐个拷贝
cp redis-6.2.4/redis.conf 7001
cp redis-6.2.4/redis.conf 7002
cp redis-6.2.4/redis.conf 7003

# 方式二：管道组合命令，一键拷贝
echo 7001 7002 7003 | xargs -t -n 1 cp redis-6.2.4/redis.conf

4) Modify the port and working directory of each instance

Modify the configuration files in each folder, modify the ports to 7001, 7002, and 7003 respectively, and modify the storage location of rdb files to your own directory (execute the following commands in the /usr/local/src directory):

sed -i -e 's/6379/7001/g' -e 's/dir .\//dir \/usr\/local\/src\/7001\//g' 7001/redis.conf
sed -i -e 's/6379/7002/g' -e 's/dir .\//dir \/usr\/local\/src\/7002\//g' 7002/redis.conf
sed -i -e 's/6379/7003/g' -e 's/dir .\//dir \/usr\/local\/src\/7003\//g' 7003/redis.conf

5) Modify the declared IP of each instance

The virtual machine itself has multiple IPs. In order to avoid confusion in the future, we need to specify the binding ip information of each instance in the redis.conf file. The format is as follows:

# redis实例的声明 IP
replica-announce-ip 192.168.150.101

Each directory needs to be changed, and we can complete the modification with one click (execute the following command in the /usr/local/src directory):

# 逐一执行
sed -i '1a replica-announce-ip 192.168.70.130' 7001/redis.conf
sed -i '1a replica-announce-ip 192.168.70.130' 7002/redis.conf
sed -i '1a replica-announce-ip 192.168.70.130' 7003/redis.conf

# 或者一键修改
printf '%s\n' 7001 7002 7003 | xargs -I{} -t sed -i '1a replica-announce-ip 192.168.70.130' {}/redis.conf

If the password requirepass is enabled on both master and slave, resulting in failure to verify successfully, just add masterauth 123321 in the configuration file (redis.conf) of the redis slave server (execute the following command in the /usr/local/src directory):

# 逐一执行
sed -i 'masterauth 123321' 7002/redis.conf
sed -i 'masterauth 123321' 7003/redis.conf

# 或者意见修改
printf '%s\n' 7002 7003 | xargs -I{} -t sed -i 'masterauth 123321' {}/redis.conf

(123321 is the password of my main redis, this parameter is the password verification when connecting with the main)

1.3 start

In order to view the logs conveniently, we can open 3 ssh windows, start 3 redis instances respectively, and start the command (execute the following command in the /usr/local/src directory):  

# 第1个
redis-server 7001/redis.conf
# 第2个
redis-server 7002/redis.conf
# 第3个
redis-server 7003/redis.conf

If you want to stop with one key, you can run the following command:

printf '%s\n' 7001 7002 7003 | xargs -I{} -t redis-cli -p {} shutdown

1.4 Open the master-slave relationship

Now the three instances have nothing to do with each other. To configure the master-slave, you can use the replicaof or slaveof (before 5.0) command.

There are two modes, temporary and permanent:

• Modify the configuration file (permanent)

  • Add a line of configuration to redis.conf:slaveof <masterip> <masterport>

• Use the redis-cli client to connect to the redis service and execute the slaveof command (it will fail after restarting):slaveof <masterip> <masterport>

Note : The command replicaof is added after 5.0, which has the same effect as salveof.

Here we use the second method for demonstration:

1) Connect to 7002 through the redis-cli command, and execute the following command:

# 连接 7002
redis-cli -p 7002
# 执行slaveof
slaveof 192.168.150.101 7001

2) Connect to 7003 through the redis-cli command, and execute the following command:  

# 连接 7003
redis-cli -p 7003
# 执行slaveof
slaveof 192.168.150.101 7001

3) Then connect to node 7001 to check the cluster status:

# 连接 7001
redis-cli -p 7001
# 查看状态
info replication

result:

1.5 Test

Do the following to test:

• Use redis-cli to connect to 7001, executeset num 123

• Use redis-cli to connect to 7002, execute get num, and then executeset num 666

• Use redis-cli to connect to 7003, execute get num, and then executeset num 888

It can be found that only the master node 7001 can perform write operations, and the two slave nodes 7002 and 7003 can only perform read operations.

Ⅱ. Redis sentinel cluster construction

2.1 Cluster structure

Here we build a Sentinel cluster formed by three nodes to supervise the previous Redis master-slave cluster. As shown in the picture:  

 

The information of the three sentinel instances is as follows:

node	IP	PORT
s1	192.168.70.130	27001
s2	192.168.70.130	27002
s3	192.168.70.130	27003

2.2 Prepare and configure the instance

To start three instances on the same virtual machine, three different configuration files and directories must be prepared, and the directory where the configuration files are located is also the working directory.

We create three folders named s1, s2, s3:

# 进入/usr/local/src目录
cd /usr/local/src
# 创建目录
mkdir s1 s2 s3

As shown in the picture:

Then we create a sentinel.conf file in the s1 directory and add the following content:

port 27001
sentinel announce-ip 192.168.70.130
sentinel monitor mymaster 192.168.70.130 7001 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 60000
dir "/usr/local/src/s1"

explain:

• port 27001: is the port of the current sentinel instance

• sentinel monitor mymaster 192.168.70.130 7001 2: Specify master node information

  • mymaster: master node name, user-defined, arbitrary write

  • 192.168.70.130 7001: the ip and port of the master node

  • 2: The quorum value when the master is elected (that is, if more than 2 sentinels think that the node is offline subjectively, it is considered that the node is objectively offline)

Then copy the s1/sentinel.conf file to the s2 and s3 directories (execute the following command in the /usr/local/src directory):

# 方式一：逐个拷贝
cp s1/sentinel.conf s2
cp s1/sentinel.conf s3
# 方式二：管道组合命令，一键拷贝
echo s2 s3 | xargs -t -n 1 cp s1/sentinel.conf

Modify the configuration files in the s2 and s3 folders, and change the ports to 27002 and 27003 respectively (execute the following commands in the /usr/local/src directory):

sed -i -e 's/27001/27002/g' -e 's/s1/s2/g' s2/sentinel.conf
sed -i -e 's/27001/27003/g' -e 's/s1/s3/g' s3/sentinel.conf

2.3 start

In order to view the logs conveniently, we can open 3 ssh windows, start 3 redis instances respectively, and start the command (execute the following command in the /usr/local/src directory):  

# 第1个
redis-sentinel s1/sentinel.conf
# 第2个
redis-sentinel s2/sentinel.conf
# 第3个
redis-sentinel s3/sentinel.conf

2.4 Testing

Try to shut down the master node 7001, check the sentinel log:  

Check the log of 7003:

Check the log of 7002:

2.5 redisTemplate (optional)

In the Redis master-slave cluster supervised by the Sentinel cluster, its nodes will change due to automatic failover, and the Redis client must sense this change and update the connection information in time. The bottom layer of Spring's RedisTemplate uses lettuce to realize node perception and automatic switching.

Next, we implement the RedisTemplate integration sentinel mechanism through a test.

1) Introduce dependencies in the pom file of the project:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>

2) Then specify the sentinel related information of redis in the configuration file application.yml:

spring:
  redis:
    sentinel:
      master: mymaster
      nodes:
        - 192.168.70.130:27001
        - 192.168.70.130:27002
        - 192.168.70.130:27003

3) In the startup class of the project, add a new bean and configure read-write separation:

@Bean
public LettuceClientConfigurationBuilderCustomizer clientConfigurationBuilderCustomizer(){
    return clientConfigurationBuilder -> clientConfigurationBuilder.readFrom(ReadFrom.REPLICA_PREFERRED);
}

This bean is configured with read and write strategies, including four types:

• MASTER: read from the master node

• MASTER_PREFERRED: Read from the master node first, read the replica when the master is unavailable

• REPLICA: read from the slave (replica) node

• REPLICA _PREFERRED: Read from the slave (replica) node first, all slaves are unavailable to read the master

Ⅲ. Redis sharding cluster construction 

3.1 Cluster structure

Here we will open 6 redis instances in the same virtual machine to simulate a fragmented cluster. The information is as follows:

IP	PORT	Role
192.168.70.130	7001	master
192.168.70.130	7002	master
192.168.70.130	7003	master
192.168.70.130	8001	slave
192.168.70.130	8002	slave
192.168.70.130	8003	slave

3.2 Prepare and configure the instance

Create 7001, 7002, 7003, 8001, 8002, 8003 directories (if the same node already exists, remember to delete the previous 7001, 7002, 7003 directories):

# 进入/usr/local/src目录
cd /usr/local/src
# 删除旧的，避免配置干扰
rm -rf 7001 7002 7003
# 创建目录
mkdir 7001 7002 7003 8001 8002 8003

Prepare a new redis.conf file under /usr/local/src with the following content:

port 6379
# 开启集群功能
cluster-enabled yes
# 集群的配置文件名称，不需要我们创建，由redis自己维护
cluster-config-file /usr/local/src/6379/nodes.conf
# 节点心跳失败的超时时间
cluster-node-timeout 5000
# 持久化文件存放目录
dir /usr/local/src/6379
# 绑定地址
bind 0.0.0.0
# 让redis后台运行
daemonize yes
# 注册的实例ip
replica-announce-ip 192.168.70.130
# 保护模式
protected-mode no
# 数据库数量
databases 1
# 日志
logfile /usr/local/src/6379/run.log

Copy this file to the 7001, 7002, 7003, 8001, 8002, 8003 directories (execute the following commands in the /usr/local/src directory):

# 执行拷贝
echo 7001 7002 7003 8001 8002 8003 | xargs -t -n 1 cp redis.conf

Modify redis.conf in each directory, and modify 6379 to be consistent with the directory (execute the following command in the /usr/local/src directory):

# 修改配置文件
printf '%s\n' 7001 7002 7003 8001 8002 8003 | xargs -I{} -t sed -i 's/6379/{}/g' {}/redis.conf

3.3 start

 Because the background startup mode has been configured, the service can be started directly (execute the following command in the /usr/local/src directory):

# 一键启动所有服务
printf '%s\n' 7001 7002 7003 8001 8002 8003 | xargs -I{} -t redis-server {}/redis.conf

View status via ps:

ps -ef | grep redis

Discovery services have been started normally:

If you want to close all processes, you can execute the command:

ps -ef | grep redis | awk '{print $2}' | xargs kill
# 或者
printf '%s\n' 7001 7002 7003 8001 8002 8003 | xargs -I{} -t redis-cli -p {} shutdown # 推荐

3.4 Create a cluster

Although the service is started, each service is currently independent without any association.

We need to execute commands to create a cluster. It was troublesome to create a cluster before Redis5.0. After 5.0, the cluster management commands are integrated into redis-cli.

1) Before Redis5.0

Before Redis5.0, the cluster commands were all implemented with src/redis-trib.rb under the redis installation package. Because redis-trib.rb is written in ruby ​​language, it needs to install ruby ​​environment.  

 # 安装依赖
 yum -y install zlib ruby rubygems
 gem install redis

Then use the command to manage the cluster:

# 进入redis的src目录
cd /usr/local/src/redis-6.2.4/src
# 创建集群
./redis-trib.rb create --replicas 1 192.168.70.130:7001 192.168.70.130:7002 192.168.70.130:7003 192.168.70.130:8001 192.168.70.130:8002 192.168.70.130:8003

2) After Redis5.0

We are using Redis6.2.4 version, cluster management and integrated into redis-cli, the format is as follows:

redis-cli --cluster create --cluster-replicas 1 192.168.70.130:7001 192.168.70.130:7002 192.168.70.130:7003 192.168.70.130:8001 192.168.70.130:8002 192.168.70.130:8003

Command description:

• redis-cli --cluster Or  ./redis-trib.rb: represents a cluster operation command

• create: represents the creation of a cluster

• --replicas 1 Or  --cluster-replicas 1: specify that the number of copies of each master in the cluster is 1, and 节点总数 ÷ (replicas + 1)  the number obtained at this time is the number of masters. Therefore, the first n nodes in the node list are masters, and the other nodes are slave nodes, which are randomly assigned to different masters.

What it looks like after running:

Enter yes here, and the cluster will start to be created:

You can view the cluster status with the command:

redis-cli -p 7001 cluster nodes

3.5 Testing

Try to connect to node 7001 and store a piece of data:

# 连接
redis-cli -c -p 7001
# 存储数据
set num 123
# 读取数据
get num
# 再次存储
set a 1

Note: The parameter -c should be added when connecting, otherwise the operation will fail