의사 클러스터 및 Docker 클러스터 빌드 테스트 프로세스
1. 의사 클러스터 구축
1.1 환경
Tencent의 클라우드 서버가 사용되며 1核心2G内存50G存储
시스템 정보는 다음과 같습니다.
[root@tcloud ~]# rpm -qi centos-release
Name : centos-release
Version : 7
Release : 9.2009.1.el7.centos
Architecture: x86_64
Install Date: Tue 19 Jan 2021 06:23:16 PM CST
Group : System Environment/Base
Size : 44787
License : GPLv2
Signature : RSA/SHA256, Thu 03 Dec 2020 12:35:28 AM CST, Key ID 24c6a8a7f4a80eb5
Source RPM : centos-release-7-9.2009.1.el7.centos.src.rpm
Build Date : Mon 23 Nov 2020 11:08:41 PM CST
Build Host : x86-01.bsys.centos.org
Relocations : (not relocatable)
Packager : CentOS BuildSystem <http://bugs.centos.org>
Vendor : CentOS
Summary : CentOS Linux release file
Description :
CentOS Linux release files
1.2 빌드
- 각 Redis 노드에는 백업 머신이 있어야 합니다.예를 들어 3개의 노드로 클러스터를 구축하려면 6개의 Redis 인스턴스가 있어야 합니다.
- 데이터는 슬롯에 따라 서로 다른 Redis 노드에 분산 및 저장되며 노드의 데이터는 공유될 수 있으며 데이터 분포는 동적으로 조정될 수 있습니다.
- 강력한 확장성, 노드를 동적으로 추가 또는 삭제할 수 있으며 최대 1000개 이상의 노드로 확장할 수 있습니다.
1.2.1 클러스터 구성
상대적으로 더 나은 성능을 가진 버전을 사용하여 5.0.14
다음을 빌드하십시오.
# 1.复制单机版并修改配置
cp -r ./redis-5.0.14 ./redis-5.0.14-0
# 修改【redis-5.0.14-0】的配置信息如下
################################## NETWORK #####################################
protected-mode no
port 6370
################################# GENERAL #####################################
daemonize yes
################################ REDIS CLUSTER ###############################
cluster-enabled yes
cluster-config-file nodes-6370.conf
cluster-node-timeout 15000
1.2.2 기타 5개 노드 구성 생성
# 2.复制5个节点并修改配置
cp -r ./redis-5.0.14-0 ./redis-5.0.14-1
cp -r ./redis-5.0.14-0 ./redis-5.0.14-2
cp -r ./redis-5.0.14-0 ./redis-5.0.14-3
cp -r ./redis-5.0.14-0 ./redis-5.0.14-4
cp -r ./redis-5.0.14-0 ./redis-5.0.14-5
# 修改每个节点的配置
sed 's/6370/6371/g' ./redis-5.0.14-1/redis.conf > ./redis-5.0.14-1/redis-6371.conf
sed 's/6370/6372/g' ./redis-5.0.14-2/redis.conf > ./redis-5.0.14-2/redis-6372.conf
sed 's/6370/6373/g' ./redis-5.0.14-3/redis.conf > ./redis-5.0.14-3/redis-6373.conf
sed 's/6370/6374/g' ./redis-5.0.14-4/redis.conf > ./redis-5.0.14-4/redis-6374.conf
sed 's/6370/6375/g' ./redis-5.0.14-5/redis.conf > ./redis-5.0.14-5/redis-6375.conf
1.2.3 노드 상태 시작 및 확인
# 3.启动6个节点并验证启动状态
./redis-5.0.14-0/bin/redis-server ./redis-5.0.14-0/redis.conf
./redis-5.0.14-1/bin/redis-server ./redis-5.0.14-1/redis-6371.conf
./redis-5.0.14-2/bin/redis-server ./redis-5.0.14-2/redis-6372.conf
./redis-5.0.14-3/bin/redis-server ./redis-5.0.14-3/redis-6373.conf
./redis-5.0.14-4/bin/redis-server ./redis-5.0.14-4/redis-6374.conf
./redis-5.0.14-5/bin/redis-server ./redis-5.0.14-5/redis-6375.conf
[root@tcloud local]# ps -ef | grep redis
root 1866 1 0 17:12 ? 00:00:00 ./redis-5.0.14-0/bin/redis-server 127.0.0.1:6370 [cluster]
root 1871 1 0 17:12 ? 00:00:00 ./redis-5.0.14-1/bin/redis-server 127.0.0.1:6371 [cluster]
root 1876 1 0 17:12 ? 00:00:00 ./redis-5.0.14-2/bin/redis-server 127.0.0.1:6372 [cluster]
root 1881 1 0 17:12 ? 00:00:00 ./redis-5.0.14-3/bin/redis-server 127.0.0.1:6373 [cluster]
root 1886 1 0 17:12 ? 00:00:00 ./redis-5.0.14-4/bin/redis-server 127.0.0.1:6374 [cluster]
root 1896 1 0 17:12 ? 00:00:00 ./redis-5.0.14-5/bin/redis-server 127.0.0.1:6375 [cluster]
root 1957 1522 0 17:12 pts/0 00:00:00 grep --color=auto redis
1.2.4 클러스터 생성
# 4.创建集群
[root@tcloud redis-5.0.14-0]# ./bin/redis-cli --cluster create 127.0.0.1:6370 127.0.0.1:6371 127.0.0.1:6372 127.0.0.1:6373 127.0.0.1:6374 127.0.0.1:6375 --cluster-replicas 1
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 127.0.0.1:6374 to 127.0.0.1:6370
Adding replica 127.0.0.1:6375 to 127.0.0.1:6371
Adding replica 127.0.0.1:6373 to 127.0.0.1:6372
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: 60772cf691608a8ae8e6f519b1a95b52c84b2457 127.0.0.1:6370
slots:[0-5460] (5461 slots) master
M: ffe9663d3fd1fc55c74c7bea6c70069fb742a69f 127.0.0.1:6371
slots:[5461-10922] (5462 slots) master
M: 6bb4e92cec0464c9969397eccaf1de1ce54b3e56 127.0.0.1:6372
slots:[10923-16383] (5461 slots) master
S: e793d2b868dbcd0b701bae10ea4d689b13a1c650 127.0.0.1:6373
replicates ffe9663d3fd1fc55c74c7bea6c70069fb742a69f
S: baae7ed422dc563bbbd22e1ac2d859ba4835dd63 127.0.0.1:6374
replicates 6bb4e92cec0464c9969397eccaf1de1ce54b3e56
S: 2fddd068ad1ae78d816d2932f012fae025e4e1f8 127.0.0.1:6375
replicates 60772cf691608a8ae8e6f519b1a95b52c84b2457
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 127.0.0.1:6370)
M: 60772cf691608a8ae8e6f519b1a95b52c84b2457 127.0.0.1:6370
slots:[0-5460] (5461 slots) master
1 additional replica(s)
M: 6bb4e92cec0464c9969397eccaf1de1ce54b3e56 127.0.0.1:6372
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
M: ffe9663d3fd1fc55c74c7bea6c70069fb742a69f 127.0.0.1:6371
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: baae7ed422dc563bbbd22e1ac2d859ba4835dd63 127.0.0.1:6374
slots: (0 slots) slave
replicates 6bb4e92cec0464c9969397eccaf1de1ce54b3e56
S: 2fddd068ad1ae78d816d2932f012fae025e4e1f8 127.0.0.1:6375
slots: (0 slots) slave
replicates 60772cf691608a8ae8e6f519b1a95b52c84b2457
S: e793d2b868dbcd0b701bae10ea4d689b13a1c650 127.0.0.1:6373
slots: (0 slots) slave
replicates ffe9663d3fd1fc55c74c7bea6c70069fb742a69f
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
1.2.5 클러스터 정보
# 5.集群并查看节点数据
[root@tcloud redis-5.0.14-0]# ./bin/redis-cli -c -p 6370
127.0.0.1:6370> cluster nodes
6bb4e92cec0464c9969397eccaf1de1ce54b3e56 127.0.0.1:6372@16372 master - 0 1683192656954 3 connected 10923-16383
ffe9663d3fd1fc55c74c7bea6c70069fb742a69f 127.0.0.1:6371@16371 master - 0 1683192657956 2 connected 5461-10922
60772cf691608a8ae8e6f519b1a95b52c84b2457 127.0.0.1:6370@16370 myself,master - 0 1683192655000 1 connected 0-5460
baae7ed422dc563bbbd22e1ac2d859ba4835dd63 127.0.0.1:6374@16374 slave 6bb4e92cec0464c9969397eccaf1de1ce54b3e56 0 1683192655951 5 connected
2fddd068ad1ae78d816d2932f012fae025e4e1f8 127.0.0.1:6375@16375 slave 60772cf691608a8ae8e6f519b1a95b52c84b2457 0 1683192655000 6 connected
e793d2b868dbcd0b701bae10ea4d689b13a1c650 127.0.0.1:6373@16373 slave ffe9663d3fd1fc55c74c7bea6c70069fb742a69f 0 1683192653000 4 connected
1.3 테스트
# 延迟测试
./bin/redis-cli -c -p 6370 --intrinsic-latency 10
# 性能测试
./bin/redis-benchmark -h 127.0.0.1 -p 6370 -q
- 대기 시간 테스트 결과
# 1
150971628 total runs (avg latency: 0.0662 microseconds / 66.24 nanoseconds per run).
Worst run took 117245x longer than the average latency.
# 2
149243181 total runs (avg latency: 0.0670 microseconds / 67.00 nanoseconds per run).
Worst run took 155735x longer than the average latency.
# 3
150936098 total runs (avg latency: 0.0663 microseconds / 66.25 nanoseconds per run).
Worst run took 87950x longer than the average latency.
- 성능 테스트 결과
# 1
PING_INLINE: 62814.07
PING_BULK: 64143.68
SET: 61087.36
GET: 62344.14
INCR: 59665.87
LPUSH: 54945.05
RPUSH: 56785.91
LPOP: 55991.04
RPOP: 57208.24
SADD: 62932.66
HSET: 59347.18
SPOP: 63091.48
LPUSH (needed to benchmark LRANGE): 56211.35
LRANGE_100 (first 100 elements): 34614.05
LRANGE_300 (first 300 elements): 16131.63
LRANGE_500 (first 450 elements): 11568.72
LRANGE_600 (first 600 elements): 9214.89
MSET (10 keys): 54644.81
# 2
PING_INLINE: 61881.19
PING_BULK: 64267.35
SET: 61881.19
GET: 61804.70
INCR: 61387.36
LPUSH: 55555.56
RPUSH: 55370.98
LPOP: 56369.79
RPOP: 57736.72
SADD: 61614.29
HSET: 61387.36
SPOP: 63613.23
LPUSH (needed to benchmark LRANGE): 53792.36
LRANGE_100 (first 100 elements): 33681.38
LRANGE_300 (first 300 elements): 15637.22
LRANGE_500 (first 450 elements): 11828.72
LRANGE_600 (first 600 elements): 9221.69
MSET (10 keys): 52826.20
# 3
PING_INLINE: 63091.48
PING_BULK: 63572.79
SET: 61012.81
GET: 60569.35
INCR: 60459.49
LPUSH: 55463.12
RPUSH: 56850.48
LPOP: 55617.35
RPOP: 57570.52
SADD: 60790.27
HSET: 60168.47
SPOP: 62695.92
LPUSH (needed to benchmark LRANGE): 55679.29
LRANGE_100 (first 100 elements): 34435.26
LRANGE_300 (first 300 elements): 16064.26
LRANGE_500 (first 450 elements): 11496.90
LRANGE_600 (first 600 elements): 9180.21
MSET (10 keys): 53648.07
2. 도커 클러스터
2.1 환경
[root@tcloud ~]# docker -v
Docker version 20.10.13, build a224086
Redis의 공식 Docker 이미지 주소: https://hub.docker.com/_/redis
2.2 구축
2.2.1 사설 네트워크 생성
# 创建桥接网卡
docker network create redis --subnet 172.81.0.0/16
# 查看所有 网卡
docker network ls
# 查看网卡详情
docker network inspect NETWORKID
[
{
"Name": "redis",
"Id": "79a62c1104195f5f077665fcdd8d4e1c92697af7db227c1fd1a92219f22c2501",
"Created": "2023-05-05T10:38:24.604493511+08:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": {
},
"Config": [
{
"Subnet": "172.81.0.0/16",
"Gateway": "172.81.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
},
"Options": {
},
"Labels": {
}
}
]
2.2.2 구성 파일 생성
# 通过脚本创建6个Redis配置
# 这里一定要注意不要加 daemonize yes 否则无法启动
for port in $(seq 1 6); \
do \
mkdir -p /usr/local/redis-docker/node-${port}/conf
touch /usr/local/redis-docker/node-${port}/conf/redis.conf
cat << EOF >/usr/local/redis-docker/node-${port}/conf/redis.conf
bind 0.0.0.0
protected-mode no
port 637${port}
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
cluster-announce-ip 172.81.0.1${port}
cluster-announce-port 637${port}
cluster-announce-bus-port 1637${port}
appendonly no
EOF
done
2.2.3 컨테이너 시작 및 검증
# 启动容器
for port in $(seq 1 6); \
do \
docker run -p 637${port}:637${port} -p 1637${port}:1637${port} --name redis-${port} \
-v /usr/local/redis-docker/node-${port}/data:/data \
-v /usr/local/redis-docker/node-${port}/conf/redis.conf:/etc/redis/redis.conf \
-d --net redis --ip 172.81.0.1${port} redis:5.0.14 redis-server /etc/redis/redis.conf
done
# 查看启动状态
[root@tcloud local]# docker ps | grep redis
a4e70ff2d840 redis:5.0.14 "docker-entrypoint.s…" 48 seconds ago Up 46 seconds 0.0.0.0:6376->6376/tcp, :::6376->6376/tcp, 0.0.0.0:16376->16376/tcp, :::16376->16376/tcp, 6379/tcp redis-6
854f8e023ce1 redis:5.0.14 "docker-entrypoint.s…" 49 seconds ago Up 47 seconds 0.0.0.0:6375->6375/tcp, :::6375->6375/tcp, 0.0.0.0:16375->16375/tcp, :::16375->16375/tcp, 6379/tcp redis-5
5165d1e61a7f redis:5.0.14 "docker-entrypoint.s…" 50 seconds ago Up 48 seconds 0.0.0.0:6374->6374/tcp, :::6374->6374/tcp, 0.0.0.0:16374->16374/tcp, :::16374->16374/tcp, 6379/tcp redis-4
f08dbda27366 redis:5.0.14 "docker-entrypoint.s…" 51 seconds ago Up 50 seconds 0.0.0.0:6373->6373/tcp, :::6373->6373/tcp, 0.0.0.0:16373->16373/tcp, :::16373->16373/tcp, 6379/tcp redis-3
e5e0feb9546a redis:5.0.14 "docker-entrypoint.s…" 52 seconds ago Up 51 seconds 0.0.0.0:6372->6372/tcp, :::6372->6372/tcp, 0.0.0.0:16372->16372/tcp, :::16372->16372/tcp, 6379/tcp redis-2
d8fae67a40b2 redis:5.0.14 "docker-entrypoint.s…" 54 seconds ago Up 52 seconds 0.0.0.0:6371->6371/tcp, :::6371->6371/tcp, 0.0.0.0:16371->16371/tcp, :::16371->16371/tcp, 6379/tcp redis-1
2.2.4 클러스터 생성
# 创建集群
docker exec -it redis-1 /bin/sh
# 容器内操作
cd /usr/local/bin
redis-cli --cluster create 172.81.0.11:6371 172.81.0.12:6372 172.81.0.13:6373 172.81.0.14:6374 172.81.0.15:6375 172.81.0.16:6376 --cluster-replicas 1
# 打印出来的信息不再贴出
# 连接集群
cd /usr/local/redis-5.0.14
./bin/redis-cli -p 6371 -c
cluster nodes
# 退出容器
exit
2.2.5 컨테이너 및 구성 파일 삭제
# 停止
for port in $(seq 1 6); \
do \
docker stop redis-${port}
done
# 删除停止的容器
docker rm $(docker ps -qa)
# 删除配置及数据文件
rm -rf ./redis-docker
2.3 테스트
# 延迟测试
./bin/redis-cli -c -p 6371 --intrinsic-latency 10
# 性能测试
./bin/redis-benchmark -h 172.81.0.1 -p 6371 -q
- 대기 시간 테스트 결과
# 1
148915461 total runs (avg latency: 0.0672 microseconds / 67.15 nanoseconds per run).
Worst run took 172533x longer than the average latency.
# 2
148930804 total runs (avg latency: 0.0671 microseconds / 67.15 nanoseconds per run).
Worst run took 147203x longer than the average latency.
# 3
149999312 total runs (avg latency: 0.0667 microseconds / 66.67 nanoseconds per run).
Worst run took 119474x longer than the average latency.
- 성능 테스트 결과
# 1
PING_INLINE: 49212.60
PING_BULK: 50327.12
SET: 48590.86
GET: 47846.89
INCR: 48923.68
LPUSH: 44863.16
RPUSH: 45106.00
LPOP: 45599.63
RPOP: 44682.75
SADD: 50377.83
HSET: 48239.27
SPOP: 50479.56
LPUSH (needed to benchmark LRANGE): 44345.89
LRANGE_100 (first 100 elements): 29691.21
LRANGE_300 (first 300 elements): 14858.84
LRANGE_500 (first 450 elements): 11119.76
LRANGE_600 (first 600 elements): 8946.94
MSET (10 keys): 43859.65
# 2
PING_INLINE: 49975.02
PING_BULK: 51493.30
SET: 47664.44
GET: 49358.34
INCR: 49358.34
LPUSH: 44822.95
RPUSH: 45146.73
LPOP: 44404.97
RPOP: 45703.84
SADD: 50125.31
HSET: 45829.52
SPOP: 49701.79
LPUSH (needed to benchmark LRANGE): 42643.92
LRANGE_100 (first 100 elements): 30349.01
LRANGE_300 (first 300 elements): 14641.29
LRANGE_500 (first 450 elements): 11111.11
LRANGE_600 (first 600 elements): 8964.59
MSET (10 keys): 44642.86
# 3
PING_INLINE: 50075.11
PING_BULK: 49900.20
SET: 48971.59
GET: 49164.21
INCR: 49925.11
LPUSH: 44130.62
RPUSH: 40950.04
LPOP: 40338.84
RPOP: 44111.16
SADD: 48449.61
HSET: 47687.18
SPOP: 49455.98
LPUSH (needed to benchmark LRANGE): 45004.50
LRANGE_100 (first 100 elements): 30637.26
LRANGE_300 (first 300 elements): 14788.52
LRANGE_500 (first 450 elements): 11183.18
LRANGE_600 (first 600 elements): 9005.76
MSET (10 keys): 45167.12
3. 요약
이전 독립 실행형 버전과 비교하고 지연 비교 결과는 다음과 같습니다.
성능 비교 결과:
- 의사 클러스터링은 단일 서버에서 도커 무리를 짓는 것보다 더 이상 성능을 향상시키지 않습니다.