Redis [Performance 02] Redis-5.0.14 pseudo-cluster and Docker cluster construction and delay and performance testing (neither can improve performance)

1. Pseudo-cluster construction

1.1 Environment

Tencent's cloud server is used 1核心2G内存50G存储, and the system information is as follows:

[root@tcloud ~]# rpm -qi centos-release
Name        : centos-release
Version     : 7
Release     : 9.2009.1.el7.centos
Architecture: x86_64
Install Date: Tue 19 Jan 2021 06:23:16 PM CST
Group       : System Environment/Base
Size        : 44787
License     : GPLv2
Signature   : RSA/SHA256, Thu 03 Dec 2020 12:35:28 AM CST, Key ID 24c6a8a7f4a80eb5
Source RPM  : centos-release-7-9.2009.1.el7.centos.src.rpm
Build Date  : Mon 23 Nov 2020 11:08:41 PM CST
Build Host  : x86-01.bsys.centos.org
Relocations : (not relocatable)
Packager    : CentOS BuildSystem <http://bugs.centos.org>
Vendor      : CentOS
Summary     : CentOS Linux release file
Description :
CentOS Linux release files

1.2 Build

• Each Redis node must have a backup machine. For example, to build a cluster with 3 nodes, there must be 6 Redis instances.
• Data is distributed and stored on different Redis nodes according to slots. The data in the nodes can be shared, and the distribution of data can be dynamically adjusted.
• Strong scalability, can dynamically add or delete nodes, and can expand up to 1000+ nodes.

1.2.1 Cluster configuration

Use a version with relatively better performance 5.0.14to build:

# 1.复制单机版并修改配置
cp -r ./redis-5.0.14 ./redis-5.0.14-0

# 修改【redis-5.0.14-0】的配置信息如下
################################## NETWORK #####################################
protected-mode no
port 6370
################################# GENERAL #####################################
daemonize yes
################################ REDIS CLUSTER  ###############################
cluster-enabled yes
cluster-config-file nodes-6370.conf
cluster-node-timeout 15000

1.2.2 Generate other 5 node configurations

# 2.复制5个节点并修改配置
cp -r ./redis-5.0.14-0 ./redis-5.0.14-1
cp -r ./redis-5.0.14-0 ./redis-5.0.14-2
cp -r ./redis-5.0.14-0 ./redis-5.0.14-3
cp -r ./redis-5.0.14-0 ./redis-5.0.14-4
cp -r ./redis-5.0.14-0 ./redis-5.0.14-5

# 修改每个节点的配置
sed 's/6370/6371/g' ./redis-5.0.14-1/redis.conf > ./redis-5.0.14-1/redis-6371.conf
sed 's/6370/6372/g' ./redis-5.0.14-2/redis.conf > ./redis-5.0.14-2/redis-6372.conf
sed 's/6370/6373/g' ./redis-5.0.14-3/redis.conf > ./redis-5.0.14-3/redis-6373.conf
sed 's/6370/6374/g' ./redis-5.0.14-4/redis.conf > ./redis-5.0.14-4/redis-6374.conf
sed 's/6370/6375/g' ./redis-5.0.14-5/redis.conf > ./redis-5.0.14-5/redis-6375.conf

1.2.3 Start and verify node status

# 3.启动6个节点并验证启动状态
./redis-5.0.14-0/bin/redis-server ./redis-5.0.14-0/redis.conf
./redis-5.0.14-1/bin/redis-server ./redis-5.0.14-1/redis-6371.conf
./redis-5.0.14-2/bin/redis-server ./redis-5.0.14-2/redis-6372.conf
./redis-5.0.14-3/bin/redis-server ./redis-5.0.14-3/redis-6373.conf
./redis-5.0.14-4/bin/redis-server ./redis-5.0.14-4/redis-6374.conf
./redis-5.0.14-5/bin/redis-server ./redis-5.0.14-5/redis-6375.conf

[root@tcloud local]# ps -ef | grep redis
root      1866     1  0 17:12 ?        00:00:00 ./redis-5.0.14-0/bin/redis-server 127.0.0.1:6370 [cluster]
root      1871     1  0 17:12 ?        00:00:00 ./redis-5.0.14-1/bin/redis-server 127.0.0.1:6371 [cluster]
root      1876     1  0 17:12 ?        00:00:00 ./redis-5.0.14-2/bin/redis-server 127.0.0.1:6372 [cluster]
root      1881     1  0 17:12 ?        00:00:00 ./redis-5.0.14-3/bin/redis-server 127.0.0.1:6373 [cluster]
root      1886     1  0 17:12 ?        00:00:00 ./redis-5.0.14-4/bin/redis-server 127.0.0.1:6374 [cluster]
root      1896     1  0 17:12 ?        00:00:00 ./redis-5.0.14-5/bin/redis-server 127.0.0.1:6375 [cluster]
root      1957  1522  0 17:12 pts/0    00:00:00 grep --color=auto redis

1.2.4 Create a cluster

# 4.创建集群
[root@tcloud redis-5.0.14-0]# ./bin/redis-cli --cluster create 127.0.0.1:6370 127.0.0.1:6371 127.0.0.1:6372 127.0.0.1:6373 127.0.0.1:6374 127.0.0.1:6375 --cluster-replicas 1
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 127.0.0.1:6374 to 127.0.0.1:6370
Adding replica 127.0.0.1:6375 to 127.0.0.1:6371
Adding replica 127.0.0.1:6373 to 127.0.0.1:6372
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: 60772cf691608a8ae8e6f519b1a95b52c84b2457 127.0.0.1:6370
   slots:[0-5460] (5461 slots) master
M: ffe9663d3fd1fc55c74c7bea6c70069fb742a69f 127.0.0.1:6371
   slots:[5461-10922] (5462 slots) master
M: 6bb4e92cec0464c9969397eccaf1de1ce54b3e56 127.0.0.1:6372
   slots:[10923-16383] (5461 slots) master
S: e793d2b868dbcd0b701bae10ea4d689b13a1c650 127.0.0.1:6373
   replicates ffe9663d3fd1fc55c74c7bea6c70069fb742a69f
S: baae7ed422dc563bbbd22e1ac2d859ba4835dd63 127.0.0.1:6374
   replicates 6bb4e92cec0464c9969397eccaf1de1ce54b3e56
S: 2fddd068ad1ae78d816d2932f012fae025e4e1f8 127.0.0.1:6375
   replicates 60772cf691608a8ae8e6f519b1a95b52c84b2457
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 127.0.0.1:6370)
M: 60772cf691608a8ae8e6f519b1a95b52c84b2457 127.0.0.1:6370
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: 6bb4e92cec0464c9969397eccaf1de1ce54b3e56 127.0.0.1:6372
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: ffe9663d3fd1fc55c74c7bea6c70069fb742a69f 127.0.0.1:6371
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: baae7ed422dc563bbbd22e1ac2d859ba4835dd63 127.0.0.1:6374
   slots: (0 slots) slave
   replicates 6bb4e92cec0464c9969397eccaf1de1ce54b3e56
S: 2fddd068ad1ae78d816d2932f012fae025e4e1f8 127.0.0.1:6375
   slots: (0 slots) slave
   replicates 60772cf691608a8ae8e6f519b1a95b52c84b2457
S: e793d2b868dbcd0b701bae10ea4d689b13a1c650 127.0.0.1:6373
   slots: (0 slots) slave
   replicates ffe9663d3fd1fc55c74c7bea6c70069fb742a69f
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

1.2.5 Cluster information

# 5.集群并查看节点数据
[root@tcloud redis-5.0.14-0]# ./bin/redis-cli -c -p 6370
127.0.0.1:6370> cluster nodes
6bb4e92cec0464c9969397eccaf1de1ce54b3e56 127.0.0.1:6372@16372 master - 0 1683192656954 3 connected 10923-16383
ffe9663d3fd1fc55c74c7bea6c70069fb742a69f 127.0.0.1:6371@16371 master - 0 1683192657956 2 connected 5461-10922
60772cf691608a8ae8e6f519b1a95b52c84b2457 127.0.0.1:6370@16370 myself,master - 0 1683192655000 1 connected 0-5460
baae7ed422dc563bbbd22e1ac2d859ba4835dd63 127.0.0.1:6374@16374 slave 6bb4e92cec0464c9969397eccaf1de1ce54b3e56 0 1683192655951 5 connected
2fddd068ad1ae78d816d2932f012fae025e4e1f8 127.0.0.1:6375@16375 slave 60772cf691608a8ae8e6f519b1a95b52c84b2457 0 1683192655000 6 connected
e793d2b868dbcd0b701bae10ea4d689b13a1c650 127.0.0.1:6373@16373 slave ffe9663d3fd1fc55c74c7bea6c70069fb742a69f 0 1683192653000 4 connected

1.3 Testing

# 延迟测试
./bin/redis-cli -c -p 6370 --intrinsic-latency 10
# 性能测试
./bin/redis-benchmark -h 127.0.0.1 -p 6370 -q

• Latency Test Results

# 1
150971628 total runs (avg latency: 0.0662 microseconds / 66.24 nanoseconds per run).
Worst run took 117245x longer than the average latency.
# 2
149243181 total runs (avg latency: 0.0670 microseconds / 67.00 nanoseconds per run).
Worst run took 155735x longer than the average latency.
# 3
150936098 total runs (avg latency: 0.0663 microseconds / 66.25 nanoseconds per run).
Worst run took 87950x longer than the average latency.

• performance test results

# 1
PING_INLINE: 62814.07
PING_BULK: 64143.68
SET: 61087.36
GET: 62344.14
INCR: 59665.87
LPUSH: 54945.05
RPUSH: 56785.91
LPOP: 55991.04
RPOP: 57208.24
SADD: 62932.66
HSET: 59347.18
SPOP: 63091.48
LPUSH (needed to benchmark LRANGE): 56211.35
LRANGE_100 (first 100 elements): 34614.05
LRANGE_300 (first 300 elements): 16131.63
LRANGE_500 (first 450 elements): 11568.72
LRANGE_600 (first 600 elements): 9214.89
MSET (10 keys): 54644.81
# 2
PING_INLINE: 61881.19
PING_BULK: 64267.35
SET: 61881.19
GET: 61804.70
INCR: 61387.36
LPUSH: 55555.56
RPUSH: 55370.98
LPOP: 56369.79
RPOP: 57736.72
SADD: 61614.29
HSET: 61387.36
SPOP: 63613.23
LPUSH (needed to benchmark LRANGE): 53792.36
LRANGE_100 (first 100 elements): 33681.38
LRANGE_300 (first 300 elements): 15637.22
LRANGE_500 (first 450 elements): 11828.72
LRANGE_600 (first 600 elements): 9221.69
MSET (10 keys): 52826.20
# 3
PING_INLINE: 63091.48
PING_BULK: 63572.79
SET: 61012.81
GET: 60569.35
INCR: 60459.49
LPUSH: 55463.12
RPUSH: 56850.48
LPOP: 55617.35
RPOP: 57570.52
SADD: 60790.27
HSET: 60168.47
SPOP: 62695.92
LPUSH (needed to benchmark LRANGE): 55679.29
LRANGE_100 (first 100 elements): 34435.26
LRANGE_300 (first 300 elements): 16064.26
LRANGE_500 (first 450 elements): 11496.90
LRANGE_600 (first 600 elements): 9180.21
MSET (10 keys): 53648.07

2. Docker cluster

2.1 Environment

[root@tcloud ~]# docker -v
Docker version 20.10.13, build a224086

The official Docker image address of Redis: https://hub.docker.com/_/redis

2.2 Build

2.2.1 Create a private network

# 创建桥接网卡
docker network create redis --subnet 172.81.0.0/16

# 查看所有 网卡
docker network ls
# 查看网卡详情
docker network inspect NETWORKID

[
    {
    
    
        "Name": "redis",
        "Id": "79a62c1104195f5f077665fcdd8d4e1c92697af7db227c1fd1a92219f22c2501",
        "Created": "2023-05-05T10:38:24.604493511+08:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
    
    
            "Driver": "default",
            "Options": {
    
    },
            "Config": [
                {
    
    
                    "Subnet": "172.81.0.0/16",
                    "Gateway": "172.81.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
    
    
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
    
    },
        "Options": {
    
    },
        "Labels": {
    
    }
    }
]

2.2.2 Generate configuration file

# 通过脚本创建6个Redis配置
# 这里一定要注意不要加 daemonize yes 否则无法启动
for port in $(seq 1 6); \
do \
mkdir -p /usr/local/redis-docker/node-${port}/conf
touch /usr/local/redis-docker/node-${port}/conf/redis.conf
cat << EOF >/usr/local/redis-docker/node-${port}/conf/redis.conf
bind 0.0.0.0
protected-mode no
port 637${port}
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
cluster-announce-ip 172.81.0.1${port}
cluster-announce-port 637${port}
cluster-announce-bus-port 1637${port}
appendonly no
EOF
done

2.2.3 Container startup and verification

# 启动容器
for port in $(seq 1 6); \
do \
docker run -p 637${port}:637${port} -p 1637${port}:1637${port} --name redis-${port} \
-v /usr/local/redis-docker/node-${port}/data:/data \
-v /usr/local/redis-docker/node-${port}/conf/redis.conf:/etc/redis/redis.conf \
-d --net redis --ip 172.81.0.1${port} redis:5.0.14 redis-server /etc/redis/redis.conf
done

# 查看启动状态
[root@tcloud local]# docker ps | grep redis
a4e70ff2d840   redis:5.0.14     "docker-entrypoint.s…"   48 seconds ago   Up 46 seconds       0.0.0.0:6376->6376/tcp, :::6376->6376/tcp, 0.0.0.0:16376->16376/tcp, :::16376->16376/tcp, 6379/tcp   redis-6
854f8e023ce1   redis:5.0.14     "docker-entrypoint.s…"   49 seconds ago   Up 47 seconds       0.0.0.0:6375->6375/tcp, :::6375->6375/tcp, 0.0.0.0:16375->16375/tcp, :::16375->16375/tcp, 6379/tcp   redis-5
5165d1e61a7f   redis:5.0.14     "docker-entrypoint.s…"   50 seconds ago   Up 48 seconds       0.0.0.0:6374->6374/tcp, :::6374->6374/tcp, 0.0.0.0:16374->16374/tcp, :::16374->16374/tcp, 6379/tcp   redis-4
f08dbda27366   redis:5.0.14     "docker-entrypoint.s…"   51 seconds ago   Up 50 seconds       0.0.0.0:6373->6373/tcp, :::6373->6373/tcp, 0.0.0.0:16373->16373/tcp, :::16373->16373/tcp, 6379/tcp   redis-3
e5e0feb9546a   redis:5.0.14     "docker-entrypoint.s…"   52 seconds ago   Up 51 seconds       0.0.0.0:6372->6372/tcp, :::6372->6372/tcp, 0.0.0.0:16372->16372/tcp, :::16372->16372/tcp, 6379/tcp   redis-2
d8fae67a40b2   redis:5.0.14     "docker-entrypoint.s…"   54 seconds ago   Up 52 seconds       0.0.0.0:6371->6371/tcp, :::6371->6371/tcp, 0.0.0.0:16371->16371/tcp, :::16371->16371/tcp, 6379/tcp   redis-1

2.2.4 Create a cluster

# 创建集群
docker exec -it redis-1 /bin/sh 

# 容器内操作
cd /usr/local/bin
redis-cli --cluster create 172.81.0.11:6371 172.81.0.12:6372 172.81.0.13:6373 172.81.0.14:6374 172.81.0.15:6375 172.81.0.16:6376 --cluster-replicas 1
# 打印出来的信息不再贴出

# 连接集群
cd /usr/local/redis-5.0.14
./bin/redis-cli -p 6371 -c
cluster nodes

# 退出容器
exit

2.2.5 Delete container and configuration file

# 停止
for port in $(seq 1 6); \
do \
docker stop redis-${port} 
done

# 删除停止的容器
docker rm $(docker ps -qa)

# 删除配置及数据文件
rm -rf ./redis-docker

2.3. Testing

# 延迟测试
./bin/redis-cli -c -p 6371 --intrinsic-latency 10
# 性能测试
./bin/redis-benchmark -h 172.81.0.1 -p 6371  -q

• Latency Test Results

# 1
148915461 total runs (avg latency: 0.0672 microseconds / 67.15 nanoseconds per run).
Worst run took 172533x longer than the average latency.
# 2
148930804 total runs (avg latency: 0.0671 microseconds / 67.15 nanoseconds per run).
Worst run took 147203x longer than the average latency.
# 3
149999312 total runs (avg latency: 0.0667 microseconds / 66.67 nanoseconds per run).
Worst run took 119474x longer than the average latency.

• performance test results

# 1
PING_INLINE: 49212.60
PING_BULK: 50327.12
SET: 48590.86
GET: 47846.89
INCR: 48923.68
LPUSH: 44863.16
RPUSH: 45106.00
LPOP: 45599.63
RPOP: 44682.75
SADD: 50377.83
HSET: 48239.27
SPOP: 50479.56
LPUSH (needed to benchmark LRANGE): 44345.89
LRANGE_100 (first 100 elements): 29691.21
LRANGE_300 (first 300 elements): 14858.84
LRANGE_500 (first 450 elements): 11119.76
LRANGE_600 (first 600 elements): 8946.94
MSET (10 keys): 43859.65
# 2
PING_INLINE: 49975.02
PING_BULK: 51493.30
SET: 47664.44
GET: 49358.34
INCR: 49358.34
LPUSH: 44822.95
RPUSH: 45146.73
LPOP: 44404.97
RPOP: 45703.84
SADD: 50125.31
HSET: 45829.52
SPOP: 49701.79
LPUSH (needed to benchmark LRANGE): 42643.92
LRANGE_100 (first 100 elements): 30349.01
LRANGE_300 (first 300 elements): 14641.29
LRANGE_500 (first 450 elements): 11111.11
LRANGE_600 (first 600 elements): 8964.59
MSET (10 keys): 44642.86
# 3
PING_INLINE: 50075.11
PING_BULK: 49900.20
SET: 48971.59
GET: 49164.21
INCR: 49925.11
LPUSH: 44130.62
RPUSH: 40950.04
LPOP: 40338.84
RPOP: 44111.16
SADD: 48449.61
HSET: 47687.18
SPOP: 49455.98
LPUSH (needed to benchmark LRANGE): 45004.50
LRANGE_100 (first 100 elements): 30637.26
LRANGE_300 (first 300 elements): 14788.52
LRANGE_500 (first 450 elements): 11183.18
LRANGE_600 (first 600 elements): 9005.76
MSET (10 keys): 45167.12

3. Summary

We compare it with the previous stand-alone version, and the delay comparison results are as follows:

Performance comparison results:

• Pseudo-clustering does not improve performance any more than Docker swarming on a single server.