Build Redis6.0-Cluster and Haproxy based on Docker

Build Redis Cluster

This article uses multiple machines to build, one virtual machine with two nodes

IP	environment	until	redis
192.168.119.156	CentOS Linux release 7.6.1810	version 19.03.9	redis：6.0
192.168.119.157	CentOS Linux release 7.6.1810	version 19.03.9	redis：6.0
192.168.119.158	CentOS Linux release 7.6.1810	version 19.03.9	redis：6.0

The overall construction steps are mainly divided into the following steps:

• Download the Redis image (in fact, this step can be omitted, because when creating a container, if the local image does not exist, it will be pulled remotely)
• Write a Redis configuration file
• Create a Redis container
• Create a Redis Cluster cluster
• View cluster status
• View cluster and node information
• Test read and write

1. Pull the Redis image

# 需要什么版本就拉取什么版本,如果是内网，请提前准备好镜像包，找一台可连接外网的虚机即可
# docker pull redis:6.0
6.0: Pulling from library/redis
eff15d958d66: Pull complete 
1aca8391092b: Pull complete 
06e460b3ba1b: Pull complete 
76b2112cc8b7: Pull complete 
0f0e368cbbb6: Pull complete 
686ee6025c94: Pull complete 
Digest: sha256:27de068c6b366096ae12b6f43839d4be22b0681732c09ef47172cf0b21b4c50b
Status: Downloaded newer image for redis:6.0
docker.io/library/redis:6.0
# docker images
REPOSITORY            TAG                 IMAGE ID            CREATED             SIZE
redis                 6.0                 ffe7215bad7b        8 days ago          112MB

2. Write Redis configuration file

Create directories and files

Perform the following operations on the three virtual machines respectively.

# 192.168.119.156
# 创建目录,目录可自己定义
mkdir -p /usr/local/docker-redis/redis-cluster
# 切换至指定目录
cd /usr/local/docker-redis/redis-cluster/
# 编写 redis-cluster.tmpl 文件
vi redis-cluster.tmpl
# 编写配置文件
port ${PORT}
requirepass 123456	# 密码可配置可不配置
masterauth 123456
protected-mode no
daemonize no
appendonly yes
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
cluster-announce-ip 192.168.119.156
cluster-announce-port ${PORT}
cluster-announce-bus-port 1${PORT}

# 192.168.119.157
# 创建目录
mkdir -p /usr/local/docker-redis/redis-cluster
# 切换至指定目录
cd /usr/local/docker-redis/redis-cluster/
# 编写 redis-cluster.tmpl 文件
vi redis-cluster.tmpl
# 编写配置文件
port ${PORT}
requirepass 123456
masterauth 123456
protected-mode no
daemonize no
appendonly yes
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
cluster-announce-ip 192.168.119.157
cluster-announce-port ${PORT}
cluster-announce-bus-port 1${PORT}

# 192.168.119.158
# 创建目录
mkdir -p /usr/local/docker-redis/redis-cluster
# 切换至指定目录
cd /usr/local/docker-redis/redis-cluster/
# 编写 redis-cluster.tmpl 文件
vi redis-cluster.tmpl
# 编写配置文件
port ${PORT}
requirepass 123456
masterauth 123456
protected-mode no
daemonize no
appendonly yes
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
cluster-announce-ip 192.168.119.158
cluster-announce-port ${PORT}
cluster-announce-bus-port 1${PORT}

port: node port;
requirepass: add access authentication;
masterauth: if the master node has enabled access authentication, authentication is required for the slave node to access the master node;
protected-mode: protected mode, the default value is yes, that is, it is enabled. After opening the protection mode, you need to configure bind ip or set the access password; close the protection mode, the external network can directly access;
daemonize: whether to start as a daemon thread (start in the background), the default is no;
appendonly: whether to enable the AOF persistence mode, The default is no;
cluster-enabled: whether to enable the cluster mode, the default is no;
cluster-config-file: cluster node information file;
cluster-node-timeout: cluster node connection timeout;
cluster-announce-ip: cluster node IP, fill in the destination The IP of the host;
cluster-announce-port: cluster node mapping port;
cluster-announce-bus-port: cluster node bus port.

Each Redis Cluster node needs to open two TCP connections. A normal Redis TCP port for serving clients, eg 6379. There is also a port based on port 6379 plus 10000, such as 16379.

The second port is used for the cluster bus, which is a node-to-node communication channel using the binary protocol. Nodes use the cluster bus for failure detection, configuration updates, failover authorization, and more. Clients should never try to communicate with the cluster bus port, just communicate with the normal Redis command port, but make sure that both ports are open in the firewall, otherwise the Redis cluster nodes will not be able to communicate.

192.168.119.156Execute the following command in the directory of the machine redis-cluster:

for port in `seq 6371 6372`; do \
  mkdir -p ${port}/conf \
  && PORT=${port} envsubst < redis-cluster.tmpl > ${port}/conf/redis.conf \
  && mkdir -p ${port}/data;\
done

192.168.119.157Execute the following command in the directory of the machine redis-cluster:

for port in `seq 6373 6374`; do \
  mkdir -p ${port}/conf \
  && PORT=${port} envsubst < redis-cluster.tmpl > ${port}/conf/redis.conf \
  && mkdir -p ${port}/data;\
done

192.168.119.158Execute the following command in the directory of the machine redis-cluster:

for port in `seq 6375 6376`; do \
  mkdir -p ${port}/conf \
  && PORT=${port} envsubst < redis-cluster.tmpl > ${port}/conf/redis.conf \
  && mkdir -p ${port}/data;\
done

The shell for statement above means to create 6371 ~ 6376 related directories and files in a loop.

After execution, you can see the directory structure

# 192.168.119.156
# 其余两台没有tree命令就不看了，结果一致，除了目录名称及配置内容
tree -L 3
.
├── 6371
│   ├── conf
│   │   └── redis.conf
│   └── data
├── 6372
│   ├── conf
│   │   └── redis.conf
│   └── data
└── redis-cluster.tmpl

6 directories, 3 files

# 配置文件输出结果
# cat 6371/conf/redis.conf 
port 6371
requirepass 123456
masterauth 123456
protected-mode no
daemonize no
appendonly yes
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
cluster-announce-ip 192.168.119.156
cluster-announce-port 6371
cluster-announce-bus-port 16371

3. Create a Redis container

6371 ~ 6376Map the port between the host and the 6 Redis containers, and map the directory of the host with the directory in the container (directory mount). Remember to specify the network mode, use hostnetwork_mode .

Execute the following command on 192.168.119.156the machine :

for port in $(seq 6371 6372); do \
  docker run -di --restart always --name redis-${port} --net host \
  -v /usr/local/docker-redis/redis-cluster/${port}/conf/redis.conf:/usr/local/etc/redis/redis.conf \
  -v /usr/local/docker-redis/redis-cluster/${port}/data:/data \
  redis:6.0 redis-server /usr/local/etc/redis/redis.conf; \
done

Execute the following command on 192.168.119.157the machine :

for port in $(seq 6373 6374); do \
  docker run -di --restart always --name redis-${port} --net host \
  -v /usr/local/docker-redis/redis-cluster/${port}/conf/redis.conf:/usr/local/etc/redis/redis.conf \
  -v /usr/local/docker-redis/redis-cluster/${port}/data:/data \
  redis:6.0 redis-server /usr/local/etc/redis/redis.conf; \
done

Execute the following command on 192.168.119.158the machine :

for port in $(seq 6375 6376); do \
  docker run -di --restart always --name redis-${port} --net host \
  -v /usr/local/docker-redis/redis-cluster/${port}/conf/redis.conf:/usr/local/etc/redis/redis.conf \
  -v /usr/local/docker-redis/redis-cluster/${port}/data:/data \
  redis:6.0 redis-server /usr/local/etc/redis/redis.conf; \
done

Remarks: The command is translated into a loop 6371 - 6376 to run the redis container
docker run to run

-di daemon

--restart always keep the container started

–name redis-${port} container name

–net host use host NIC

− v mount directory

redis:6.0 specifies the mirror version

redis redis-server /usr/local/etc/redis/redis.conf Start the redis server according to the mount configuration

Check if created successfully

docker ps

4. Create a Redis Cluster cluster

into a container

docker exec -it redis-6371 bash
cd cd /usr/local/bin/

# 创建Redis Cluster 集群
./redis-cli -a 123456 --cluster create 192.168.119.156:6371 192.168.119.156:6372 192.168.119.157:6373 192.168.119.157:6374 192.168.119.158:6375 192.168.119.158:6376 --cluster-replicas 1

So far, a highly available Redis Cluster has been built, as shown in the figure below, the cluster contains 6 Redis nodes, 3 masters and 3 slaves. The three master nodes will allocate slots to process client command requests, and the slave nodes can replace the master node after the master node fails.

5. View the cluster status

Let's enter the container first, and then check the status of the cluster through some common commands of the cluster.

# 进入容器
docker exec -it redis-6371 bash
# 切换至指定目录
cd /usr/local/bin/

# 检查集群状态
./redis-cli -a 123456 --cluster check 192.168.119.156:6371

6. View cluster information and node information

# 在192.168.119.156机器连接
# 连接至集群某个节点
./redis-cli -c -a 1234 -h 192.168.119.158 -p 6376
# 查看集群信息
cluster info
# 查看集群结点信息
cluster nodes

7. Test reading and writing

在 6376 节点中执行写入和读取，命令如下：

# 写
set name mrhelloworld
set a 123
set b 123456
# 读取数据
get name
get a
get b

Come to 192.168.119.157 virtual machine to check

docker exec -it redis-6374 bash
root@k8s-node01:/data# cd /usr/local/bin/
root@k8s-node01:/usr/local/bin# ./redis-cli -c -a 1234 -h 192.168.119.157 -p 6374
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Warning: AUTH failed
192.168.119.157:6374> get name
-> Redirected to slot [5798] located at 192.168.119.157:6373
Warning: AUTH failed
"mrhelloworld"
192.168.119.157:6373> get a
-> Redirected to slot [15495] located at 192.168.119.158:6375
Warning: AUTH failed
"123"
192.168.119.158:6375> get b
-> Redirected to slot [3300] located at 192.168.119.156:6371
Warning: AUTH failed
"123456"
192.168.119.156:6371>

Through the above operations, we know that the namekey storage is allocated to 6373 nodes, a is allocated to 6375, and b is allocated to 6371 nodes

So we don't need to redirect the node to directly connect to any node and get the value, because the data is on the node, so it can be read and returned directly.

At this point, Redis Cluster is built

8. Haproxy load

pull image

docker pull haproxy
Using default tag: latest
latest: Pulling from library/haproxy
eff15d958d66: Already exists 
2b24affa247e: Pull complete 
8ab01220e071: Pull complete 
5a8e56fc6768: Pull complete 
Digest: sha256:bd15d84d1ff75a1259e7db6e4d8c68bc1a445da0107f39db9225746a00126c57
Status: Downloaded newer image for haproxy:latest
docker.io/library/haproxy:latest
# 创建容器
docker create --name haproxy --net host -v /opt/haproxy:/usr/local/etc/haproxy haproxy:latest
0b85906e3a89d1920e88beebbef1b1cbc78d5978340e728cdc71f0d741ccc517
# 修改haproxy.cfg
vim /opt/haproxy/haproxy.cfg

global
        log 127.0.0.1 local2
        maxconn 4000
        daemon

defaults
        mode http
        log global
        option httplog
        option dontlognull
        option http-server-close
        option forwardfor except 127.0.0.0/8
        option redispatch
        retries 3
        timeout http-request 10s
        timeout queue 1m
        timeout connect 10s
        timeout client 1m
        timeout server 1m
        timeout http-keep-alive 10s
        timeout check 10s
        maxconn 3000
#haproxy的客户页面
listen http_front
  bind 0.0.0.0:25666
  stats uri /haproxy #页面地址
  #页面的用户名和密码，建议主备设为不同，方便确定抢占到VIP的服务器是主机还是备机
  stats auth admin:admin
  stats admin if TRUE #管理界面，成功登陆后可通过webui管理节点

listen proxy-redis
        bind 0.0.0.0:16379		# 负载端口
        mode tcp
        balance roundrobin
        option tcplog
        server redis1 192.168.119.156:6371  check port 6371 maxconn 3000
        server redis2 192.168.119.156:6372  check port 6372 maxconn 3000
        server redis3 192.168.119.157:6373  check port 6373 maxconn 3000
        server redis4 192.168.119.157:6374  check port 6374 maxconn 3000
        server redis5 192.168.119.158:6375  check port 6375 maxconn 3000
        server redis6 192.168.119.158:6376  check port 6376 maxconn 3000
        

# 重启容器 
docker restart haproxy
docker ps 
CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS                              NAMES
0b85906e3a89        haproxy:latest        "docker-entrypoint.s…"   11 minutes ago      Up 8 minutes                                           haproxy
4af3e220ba25        redis:6.0             "docker-entrypoint.s…"   2 hours ago         Up About an hour                                       redis-6372
db3a02890eb3        redis:6.0             "docker-entrypoint.s…"   2 hours ago         Up About an hour                                       redis-6371

# 测试负载
docker run -it redis:6.0 bash
cd /usr/local/bin/
./redis-cli -h 192.168.119.156 -p 16379
192.168.119.156:16379> get a
"123"

So far haproxy is normal
and can be accessed: http://192.168.119.156:25666/haproxy