Docker builds Redis Cluster cluster environment

PS: In order to make the environment more realistic, this article uses a multi-machine environment

• 192.168.10.10
• 192.168.10.11

The infrastructure environment used by each machine is as follows:

• CentOS 7.8.2003
• Docker version 19.03.12

Build

The overall construction steps are mainly divided into the following steps:

• Download the Redis image (in fact, this step can be omitted, because when the container is created, if the local image does not exist, it will be pulled remotely);
• Write Redis configuration file;
• Create Redis container;
• Create a Redis Cluster cluster.

Write Redis configuration file

Create directories and files

Respectively, 192.168.10.10and 192.168.10.11perform the following operations on both machines.

# 创建目录
mkdir -p /usr/local/docker-redis/redis-cluster
# 切换至指定目录
cd /usr/local/docker-redis/redis-cluster/
# 编写 redis-cluster.tmpl 文件
vi redis-cluster.tmpl

Write configuration file

192.168.10.10Machine redis-cluster.tmplcontents of the file are as follows:

port ${PORT}
requirepass 1234
masterauth 1234
protected-mode no
daemonize no
appendonly yes
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
cluster-announce-ip 192.168.10.10
cluster-announce-port ${PORT}
cluster-announce-bus-port 1${PORT}

192.168.10.11Machine redis-cluster.tmplcontents of the file are as follows:

port ${PORT}
requirepass 1234
masterauth 1234
protected-mode no
daemonize no
appendonly yes
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
cluster-announce-ip 192.168.10.11
cluster-announce-port ${PORT}
cluster-announce-bus-port 1${PORT}

• port: Node port;
• requirepass: Add access authentication;
• masterauth: If the master node has enabled access authentication, the slave node requires authentication to access the master node;
• protected-mode：Protection mode, the default value is yes, which is on. After opening the protection mode, configure bind ipor set the access password; off protected mode can directly access the external network;
• daemonize: Whether to start as a daemon thread (start in the background), the default is no;
• appendonly: Whether to enable AOF persistence mode, the default is no;
• cluster-enabled: Whether to enable cluster mode, default no;
• cluster-config-file: Cluster node information file;
• cluster-node-timeout: Cluster node connection timeout time;
• cluster-announce-ip: Cluster node IP, fill in the IP of the host;
• cluster-announce-port: Cluster node mapping port;
• cluster-announce-bus-port: Cluster node bus port.

Each Redis cluster node needs to open two TCP connections . A normal Redis TCP port used to provide services to clients, such as 6379. There is also a port based on port 6379 plus 10000, such as 16379.

The second port is used for the cluster bus, which is a node-to-node communication channel using a binary protocol. Nodes use the cluster bus for fault detection, configuration updates, failover authorization, and so on. The client should never try to communicate with the cluster bus port, just communicate with the normal Redis command port, but please make sure that both ports in the firewall are open, otherwise the Redis cluster node will not be able to communicate.

In 192.168.10.10the machine redis-clusterexecute the following command in the directory:

for port in `seq 6371 6373`; do \
  mkdir -p ${port}/conf \
  && PORT=${port} envsubst < redis-cluster.tmpl > ${port}/conf/redis.conf \
  && mkdir -p ${port}/data;\
done

In 192.168.10.11the machine redis-clusterexecute the following command in the directory:

for port in `seq 6374 6376`; do \
  mkdir -p ${port}/conf \
  && PORT=${port} envsubst < redis-cluster.tmpl > ${port}/conf/redis.conf \
  && mkdir -p ${port}/data;\
done

The above two shell for statements mean to create directories and files related to 6371 ~ 6376 in a loop.

In the 192.168.10.10execution machine command to view the results as follows, if there is no treecommand to install yum install -y tree.

In the 192.168.10.11execution machine command to view the results as follows.

The following content is the configuration file details of each node.

============================== 192.168.10.10 ==============================
[root@localhost redis-cluster]# cat /usr/local/docker-redis/redis-cluster/637{1..3}/conf/redis.conf
port 6371
requirepass 1234
masterauth 1234
protected-mode no
daemonize no
appendonly yes
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
cluster-announce-ip 192.168.10.10
cluster-announce-port 6371
cluster-announce-bus-port 16371

port 6372
requirepass 1234
masterauth 1234
protected-mode no
daemonize no
appendonly yes
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
cluster-announce-ip 192.168.10.10
cluster-announce-port 6372
cluster-announce-bus-port 16372

port 6373
requirepass 1234
masterauth 1234
protected-mode no
daemonize no
appendonly yes
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
cluster-announce-ip 192.168.10.10
cluster-announce-port 6373
cluster-announce-bus-port 16373
============================== 192.168.10.10 ==============================

============================== 192.168.10.11 ==============================
[root@localhost redis-cluster]# cat /usr/local/docker-redis/redis-cluster/637{4..6}/conf/redis.conf
port 6374
requirepass 1234
masterauth 1234
protected-mode no
daemonize no
appendonly yes
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
cluster-announce-ip 192.168.10.11
cluster-announce-port 6374
cluster-announce-bus-port 16374

port 6375
requirepass 1234
masterauth 1234
protected-mode no
daemonize no
appendonly yes
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
cluster-announce-ip 192.168.10.11
cluster-announce-port 6375
cluster-announce-bus-port 16375

port 6376
requirepass 1234
masterauth 1234
protected-mode no
daemonize no
appendonly yes
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
cluster-announce-ip 192.168.10.11
cluster-announce-port 6376
cluster-announce-bus-port 16376
============================== 192.168.10.11 ==============================

Create Redis container

Create a container

The host computer 6371 ~ 6376port 6 Redis mapping between containers, catalog directory within the host vessel and mapping (directory mount). Remember to specify the network mode, using a hostnetwork mode.

In 192.168.10.10executing the following command machine:

for port in $(seq 6371 6373); do \
  docker run -di --restart always --name redis-${port} --net host \
  -v /usr/local/docker-redis/redis-cluster/${port}/conf/redis.conf:/usr/local/etc/redis/redis.conf \
  -v /usr/local/docker-redis/redis-cluster/${port}/data:/data \
  redis redis-server /usr/local/etc/redis/redis.conf; \
done

In 192.168.10.11executing the following command machine:

for port in $(seq 6374 6376); do \
  docker run -di --restart always --name redis-${port} --net host \
  -v /usr/local/docker-redis/redis-cluster/${port}/conf/redis.conf:/usr/local/etc/redis/redis.conf \
  -v /usr/local/docker-redis/redis-cluster/${port}/data:/data \
  redis redis-server /usr/local/etc/redis/redis.conf; \
done

In the 192.168.10.10machine to perform docker ps -n 3to see whether the container has been created.

In the 192.168.10.11machine to perform docker ps -n 3to see whether the container has been created.

Create Redis Cluster

Not enter a container node and enter /usr/local/bin/the directory:

# 进入容器
docker exec -it redis-6371 bash
# 切换至指定目录
cd /usr/local/bin/

Next, we can use the following commands to create a Redis Cluster cluster.

redis-cli -a 1234 --cluster create 192.168.10.10:6371 192.168.10.10:6372 192.168.10.10:6373 192.168.10.11:6374 192.168.10.11:6375 192.168.10.11:6376 --cluster-replicas 1

A selection prompt message appears, enter yes , the result is as follows:

The cluster is successfully created as follows:

The following content is the detailed information returned when the cluster is created, that is, all the content in the previous two pictures.

root@localhost:/usr/local/bin# redis-cli -a 1234 --cluster create 192.168.10.10:6371 192.168.10.10:6372 192.168.10.10:6373 192.168.10.11:6374 192.168.10.11:6375 192.168.10.11:6376 --cluster-replicas 1
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.10.11:6376 to 192.168.10.10:6371
Adding replica 192.168.10.10:6373 to 192.168.10.11:6374
Adding replica 192.168.10.11:6375 to 192.168.10.10:6372
M: 299cf79ddafc83dced27f628f1f82dac483fbc4e 192.168.10.10:6371
   slots:[0-5460] (5461 slots) master
M: ac805b90b6e20e26dc4268454bb2855beea6cc19 192.168.10.10:6372
   slots:[10923-16383] (5461 slots) master
S: db35494fcc5db0c88d27da7885c817e6cdcc9373 192.168.10.10:6373
   replicates 7013270480d37eeab79b9cd0272e934d4548136a
M: 7013270480d37eeab79b9cd0272e934d4548136a 192.168.10.11:6374
   slots:[5461-10922] (5462 slots) master
S: 8435e1b0d51f2690c5f94f9a5682a4ac34e94326 192.168.10.11:6375
   replicates ac805b90b6e20e26dc4268454bb2855beea6cc19
S: 7b13c16fa6fe8e13cdc0b4846b87edffed55c62e 192.168.10.11:6376
   replicates 299cf79ddafc83dced27f628f1f82dac483fbc4e
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 192.168.10.10:6371)
M: 299cf79ddafc83dced27f628f1f82dac483fbc4e 192.168.10.10:6371
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 8435e1b0d51f2690c5f94f9a5682a4ac34e94326 192.168.10.11:6375
   slots: (0 slots) slave
   replicates ac805b90b6e20e26dc4268454bb2855beea6cc19
S: db35494fcc5db0c88d27da7885c817e6cdcc9373 192.168.10.10:6373
   slots: (0 slots) slave
   replicates 7013270480d37eeab79b9cd0272e934d4548136a
S: 7b13c16fa6fe8e13cdc0b4846b87edffed55c62e 192.168.10.11:6376
   slots: (0 slots) slave
   replicates 299cf79ddafc83dced27f628f1f82dac483fbc4e
M: 7013270480d37eeab79b9cd0272e934d4548136a 192.168.10.11:6374
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
M: ac805b90b6e20e26dc4268454bb2855beea6cc19 192.168.10.10:6372
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

So far, a highly available Redis Cluster cluster is built, as shown in the figure below, the cluster contains 6 Redis nodes, 3 masters and 3 slaves. The three master nodes will allocate slots to process the client's command request, and the slave node can be used to replace the master node after the master node fails.

View cluster status

We first enter the container, and then check the status of the cluster through some commonly used commands in the cluster.

# 进入容器
docker exec -it redis-6371 bash
# 切换至指定目录
cd /usr/local/bin/

Check cluster status

redis-cli -a 1234 --cluster check 192.168.10.11:6375

View cluster information and node information

# 连接至集群某个节点
redis-cli -c -a 1234 -h 192.168.10.11 -p 6376
# 查看集群信息
cluster info
# 查看集群结点信息
cluster nodes

SET/GET

To perform writing and reading in the 6371 node, the commands are as follows:

# 进入容器并连接至集群某个节点
docker exec -it redis-6371 /usr/local/bin/redis-cli -c -a 1234 -h 192.168.10.10 -p 6371
# 写入数据
set name mrhelloworld
set aaa 111
set bbb 222
# 读取数据
get name
get aaa
get bbb

Don't worry, let me explain the operation process in the above picture:

• First enter the container and connect to a node in the cluster;
• Then execute the first set command set name mrhelloworld, the namekey is calculated according to the hash function [5798]. The distribution of the current cluster environment grooves are: [0-5460] 6371节点, , [5461-10922] 6374节点, [10923-16383] 6372节点so the key is assigned to the memory 6374 node;
• Let's look at the second set command set aaa, where we may have some doubts, why not see the aaavalue of the bond obtained by the hash function after the operation? Because I was redirected to the 6374 node and inserted data, if there is still data inserted at this time, the value obtained after the key according to the hash function operation is still in the range of the node, then the data can be inserted directly;
• Followed by the third set command set bbb, the bbbkey is the value obtained after the hash function operation [5287], so the storage of the key is allocated to the 6371 node;
• Then there is the read operation. In the fourth command get name, the namekey [5798]is redirected to the 6374 node for reading according to the value obtained after the hash function operation ;
• The fifth command get aaa, aaathe value obtained after the key is calculated according to the hash function is also at the 6374 node, which is directly read;
• The sixth command get bbb, the bbbkey [5287]is redirected to the 6371 node for reading according to the value obtained after the hash function operation .

Through the above operation we learned that namethe storage key is assigned to the 6374 nodes, if a node connected directly to 6374 and get the value of what will happen? Yes, there is no need to redirect the node, because the data is on the node, so it is directly read and returned.

Client connection

Finally, come to a wave of client connection operations, any node, and see if you can access the Redis Cluster cluster externally.