1. Deploy redis stand-alone machine on k8s
1.1. Introduction to redis
Redis is an open source non-relational database (nosql database) based on the BSD protocol. The author is Italian developer Salvatore Sanfilippo and was released in 2009. It is written in C language; redis is based on memory storage and is currently a popular key value. Database (key-value database), which provides a service for remote sharing of memory through the network. Similar functions are provided by memcache, but compared to memcache, redis also provides functions such as easy expansion, high performance, and data persistence. . The main application scenarios include session sharing, which is often used for tomcat in a web cluster or session sharing for multiple web servers in PHP; message queue, ELK's log cache, and subscription and publishing systems for some businesses; counters, which are often used for access rankings and product browsing. Numerical statistical scenarios related to numbers and times; caching, often used for data query, e-commerce website product information, news content, etc.; compared to memcache, redis supports data persistence, and can save memory data in the disk and restart the redis service Or the server can later restore data from the backup file to memory for continued use.
1.2, PV/PVC and Redis stand-alone
Since redis data (mainly redis snapshots) are stored in the storage system, even if the redis pod dies, the corresponding data will not be lost; because when a redis stand-alone machine is deployed on k8s and the redis pod dies, k8s will rebuild the corresponding pod. During reconstruction, the corresponding PVC will be mounted to the pod and the snapshot will be loaded, so that the redis data will not be lost due to the pod hanging;
1.3. Build redis image
root@k8s-master01:~/k8s-data/dockerfile/web/magedu/redis# ll
total 1784
drwxr-xr-x 2 root root 4096 Jun 5 15:22 ./
drwxr-xr-x 11 root root 4096 Aug 9 2022 ../
-rw-r--r-- 1 root root 717 Jun 5 15:20 Dockerfile
-rwxr-xr-x 1 root root 235 Jun 5 15:21 build-command.sh*
-rw-r--r-- 1 root root 1740967 Jun 22 2021 redis-4.0.14.tar.gz
-rw-r--r-- 1 root root 58783 Jun 22 2021 redis.conf
-rwxr-xr-x 1 root root 84 Jun 5 15:21 run_redis.sh*
root@k8s-master01:~/k8s-data/dockerfile/web/magedu/redis# cat Dockerfile
#Redis Image
# 导入自定义centos基础镜像
FROM harbor.ik8s.cc/baseimages/magedu-centos-base:7.9.2009
# 添加redis源码包至/usr/local/src
ADD redis-4.0.14.tar.gz /usr/local/src
# 编译安装redis
RUN ln -sv /usr/local/src/redis-4.0.14 /usr/local/redis && cd /usr/local/redis && make && cp src/redis-cli /usr/sbin/ && cp src/redis-server /usr/sbin/ && mkdir -pv /data/redis-data
# 添加redis配置文件
ADD redis.conf /usr/local/redis/redis.conf
# 暴露redis服务端口
EXPOSE 6379
#ADD run_redis.sh /usr/local/redis/run_redis.sh
#CMD ["/usr/local/redis/run_redis.sh"]
# 添加启动脚本
ADD run_redis.sh /usr/local/redis/entrypoint.sh
# 启动redis
ENTRYPOINT ["/usr/local/redis/entrypoint.sh"]
root@k8s-master01:~/k8s-data/dockerfile/web/magedu/redis# cat build-command.sh
#!/bin/bash
TAG=$1
#docker build -t harbor.ik8s.cc/magedu/redis:${TAG} .
#sleep 3
#docker push harbor.ik8s.cc/magedu/redis:${TAG}
nerdctl build -t harbor.ik8s.cc/magedu/redis:${TAG} .
nerdctl push harbor.ik8s.cc/magedu/redis:${TAG}
root@k8s-master01:~/k8s-data/dockerfile/web/magedu/redis# cat run_redis.sh
#!/bin/bash
# Redis启动命令
/usr/sbin/redis-server /usr/local/redis/redis.conf
# 使用tail -f 在pod内部构建守护进程
tail -f /etc/hosts
root@k8s-master01:~/k8s-data/dockerfile/web/magedu/redis# grep -v '^#\|^$' redis.conf
bind 0.0.0.0
protected-mode yes
port 6379
tcp-backlog 511
timeout 0
tcp-keepalive 300
daemonize yes
supervised no
pidfile /var/run/redis_6379.pid
loglevel notice
logfile ""
databases 16
always-show-logo yes
save 900 1
save 5 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error no
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /data/redis-data
slave-serve-stale-data yes
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
requirepass 123456
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
slave-lazy-flush no
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble no
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes
root@k8s-master01:~/k8s-data/dockerfile/web/magedu/redis# 
1.3.1. Verify whether the rdis image is uploaded to harbor?
1.4. Test the redis image
1.4.1. Verify that the redis image is run as a container to see if it is running normally?
1.4.2. Connect redis remotely to see if the connection can be normal?
Being able to run the redis image as a container and being able to connect to redis through a remote host for data reading and writing shows that there is no problem with the reids image we built;
1.5. Create PV and PVC
1.5.1. Prepare the redis data storage directory on the nfs server
root@harbor:~# mkdir -pv /data/k8sdata/magedu/redis-datadir-1
mkdir: created directory '/data/k8sdata/magedu/redis-datadir-1'
root@harbor:~# cat /etc/exports
# /etc/exports: the access control list for filesystems which may be exported
# to NFS clients. See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4 gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes gss/krb5i(rw,sync,no_subtree_check)
#
/data/k8sdata/kuboard *(rw,no_root_squash)
/data/volumes *(rw,no_root_squash)
/pod-vol *(rw,no_root_squash)
/data/k8sdata/myserver *(rw,no_root_squash)
/data/k8sdata/mysite *(rw,no_root_squash)
/data/k8sdata/magedu/images *(rw,no_root_squash)
/data/k8sdata/magedu/static *(rw,no_root_squash)
/data/k8sdata/magedu/zookeeper-datadir-1 *(rw,no_root_squash)
/data/k8sdata/magedu/zookeeper-datadir-2 *(rw,no_root_squash)
/data/k8sdata/magedu/zookeeper-datadir-3 *(rw,no_root_squash)
/data/k8sdata/magedu/redis-datadir-1 *(rw,no_root_squash)
root@harbor:~# exportfs -av
exportfs: /etc/exports [1]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/kuboard".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [2]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/volumes".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [3]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/pod-vol".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [4]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/myserver".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [5]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/mysite".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [7]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/images".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [8]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/static".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [11]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/zookeeper-datadir-1".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [12]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/zookeeper-datadir-2".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [13]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/zookeeper-datadir-3".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [16]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/redis-datadir-1".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exporting *:/data/k8sdata/magedu/redis-datadir-1
exporting *:/data/k8sdata/magedu/zookeeper-datadir-3
exporting *:/data/k8sdata/magedu/zookeeper-datadir-2
exporting *:/data/k8sdata/magedu/zookeeper-datadir-1
exporting *:/data/k8sdata/magedu/static
exporting *:/data/k8sdata/magedu/images
exporting *:/data/k8sdata/mysite
exporting *:/data/k8sdata/myserver
exporting *:/pod-vol
exporting *:/data/volumes
exporting *:/data/k8sdata/kuboard
root@harbor:~# 1.5.2. Create pv
root@k8s-master01:~/k8s-data/yaml/magedu/redis/pv# cat redis-persistentvolume.yaml
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-datadir-pv-1
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
nfs:
path: /data/k8sdata/magedu/redis-datadir-1
server: 192.168.0.42
root@k8s-master01:~/k8s-data/yaml/magedu/redis/pv#
1.5.3. Create pvc
root@k8s-master01:~/k8s-data/yaml/magedu/redis/pv# cat redis-persistentvolumeclaim.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redis-datadir-pvc-1
namespace: magedu
spec:
volumeName: redis-datadir-pv-1
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
root@k8s-master01:~/k8s-data/yaml/magedu/redis/pv#
1.6. Deploy redis service
root@k8s-master01:~/k8s-data/yaml/magedu/redis# cat redis.yaml
kind: Deployment
#apiVersion: extensions/v1beta1
apiVersion: apps/v1
metadata:
labels:
app: devops-redis
name: deploy-devops-redis
namespace: magedu
spec:
replicas: 1
selector:
matchLabels:
app: devops-redis
template:
metadata:
labels:
app: devops-redis
spec:
containers:
- name: redis-container
image: harbor.ik8s.cc/magedu/redis:v4.0.14
imagePullPolicy: Always
volumeMounts:
- mountPath: "/data/redis-data/"
name: redis-datadir
volumes:
- name: redis-datadir
persistentVolumeClaim:
claimName: redis-datadir-pvc-1
---
kind: Service
apiVersion: v1
metadata:
labels:
app: devops-redis
name: srv-devops-redis
namespace: magedu
spec:
type: NodePort
ports:
- name: http
port: 6379
targetPort: 6379
nodePort: 36379
selector:
app: devops-redis
sessionAffinity: ClientIP
sessionAffinityConfig:
clientIP:
timeoutSeconds: 10800
root@k8s-master01:~/k8s-data/yaml/magedu/redis# 
The above error report says that our service port is out of range. This is because we specified the service port range when initializing the k8s cluster;
1.6.1. Modify the nodeport port range
Edit /etc/systemd/system/kube-apiserver.service and modify the value specified by the --service-node-port-range option; the other two master nodes also need to be modified.
1.6.2. Reload kube-apiserver.service and restart kube-apiserver
root@k8s-master01:~# systemctl daemon-reload
root@k8s-master01:~# systemctl restart kube-apiserver.service
root@k8s-master01:~# Deploy redis again
1.7. Verify redis data reading and writing
1.7.1. Connect to port 36376 of any k8s node and test redis to read and write data.
1.8. Verify whether the data corresponding to redis pod reconstruction is lost?
1.8.1. Check whether the redis snapshot file is stored in the storage?
root@harbor:~# ll /data/k8sdata/magedu/redis-datadir-1
total 12
drwxr-xr-x 2 root root 4096 Jun 5 16:29 ./
drwxr-xr-x 8 root root 4096 Jun 5 15:53 ../
-rw-r--r-- 1 root root 116 Jun 5 16:29 dump.rdb
root@harbor:~#
You can see that we just wrote data to redis, and redis took a snapshot when it found the key change within the specified time. Because the redis data directory is nfs mounted through pv/pvc, so we can work normally in the directory corresponding to nfs. See this snapshot file;
1.8.2. Delete the redis pod and wait for k8s to rebuild the redis pod.
1.8.3. Verify the reconstructed redis pod data
You can see that the redis pod after k8s reconstruction still retains the data of the original pod; this shows that the pvc of the previous pod was mounted when k8s was rebuilt;
2. Deploy redis cluster on k8s
2.1、PV/PVC及Redis Cluster-StatefulSet
Redis cluster is slightly more complicated than redis stand-alone machine. We also store redis cluster data in the storage system through pv/pvc. Unlike redis stand-alone machine, redis cluster will perform crc16 calculation on the stored data, and then take the modulus with 16384. Calculate and get a number, which is a slot stored in the redis cluster; that is, the redis cluster evenly distributes 16384 slots to all master nodes in the cluster, and each master node stores part of the entire cluster data; in this way There is a problem. If the master goes down, the data in the corresponding slot will be unavailable. In order to prevent the master from being a single point of failure, we also need to make the master highly available, that is, use a slave node to back up the master. If the master goes down, When the machine is offline, the corresponding slave will take over the master and continue to provide services to the cluster, thereby achieving high availability of the redis cluster master; as shown in the figure above, we use a redis cluster with 3 masters and 3 slaves, redis0, 1, and 2 are masters, then 3 , 4, and 5 correspond to slaves 0, 1, and 2, which are responsible for backing up the data of their respective masters; these six pods store data in the storage system through the pv/pvc of the k8s cluster;
2.2. Create PV
2.2.1. Prepare the redis cluster data directory on nfs
root@harbor:~# mkdir -pv /data/k8sdata/magedu/redis{0,1,2,3,4,5}
mkdir: created directory '/data/k8sdata/magedu/redis0'
mkdir: created directory '/data/k8sdata/magedu/redis1'
mkdir: created directory '/data/k8sdata/magedu/redis2'
mkdir: created directory '/data/k8sdata/magedu/redis3'
mkdir: created directory '/data/k8sdata/magedu/redis4'
mkdir: created directory '/data/k8sdata/magedu/redis5'
root@harbor:~# tail -6 /etc/exports
/data/k8sdata/magedu/redis0 *(rw,no_root_squash)
/data/k8sdata/magedu/redis1 *(rw,no_root_squash)
/data/k8sdata/magedu/redis2 *(rw,no_root_squash)
/data/k8sdata/magedu/redis3 *(rw,no_root_squash)
/data/k8sdata/magedu/redis4 *(rw,no_root_squash)
/data/k8sdata/magedu/redis5 *(rw,no_root_squash)
root@harbor:~# exportfs -av
exportfs: /etc/exports [1]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/kuboard".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [2]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/volumes".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [3]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/pod-vol".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [4]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/myserver".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [5]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/mysite".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [7]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/images".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [8]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/static".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [11]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/zookeeper-datadir-1".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [12]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/zookeeper-datadir-2".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [13]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/zookeeper-datadir-3".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [16]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/redis-datadir-1".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [18]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/redis0".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [19]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/redis1".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [20]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/redis2".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [21]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/redis3".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [22]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/redis4".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [23]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/redis5".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exporting *:/data/k8sdata/magedu/redis5
exporting *:/data/k8sdata/magedu/redis4
exporting *:/data/k8sdata/magedu/redis3
exporting *:/data/k8sdata/magedu/redis2
exporting *:/data/k8sdata/magedu/redis1
exporting *:/data/k8sdata/magedu/redis0
exporting *:/data/k8sdata/magedu/redis-datadir-1
exporting *:/data/k8sdata/magedu/zookeeper-datadir-3
exporting *:/data/k8sdata/magedu/zookeeper-datadir-2
exporting *:/data/k8sdata/magedu/zookeeper-datadir-1
exporting *:/data/k8sdata/magedu/static
exporting *:/data/k8sdata/magedu/images
exporting *:/data/k8sdata/mysite
exporting *:/data/k8sdata/myserver
exporting *:/pod-vol
exporting *:/data/volumes
exporting *:/data/k8sdata/kuboard
root@harbor:~# 2.2.2. Create pv
root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster# cat pv/redis-cluster-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-cluster-pv0
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
nfs:
server: 192.168.0.42
path: /data/k8sdata/magedu/redis0
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-cluster-pv1
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
nfs:
server: 192.168.0.42
path: /data/k8sdata/magedu/redis1
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-cluster-pv2
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
nfs:
server: 192.168.0.42
path: /data/k8sdata/magedu/redis2
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-cluster-pv3
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
nfs:
server: 192.168.0.42
path: /data/k8sdata/magedu/redis3
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-cluster-pv4
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
nfs:
server: 192.168.0.42
path: /data/k8sdata/magedu/redis4
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-cluster-pv5
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
nfs:
server: 192.168.0.42
path: /data/k8sdata/magedu/redis5
root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster#
2.3. Deploy redis cluster
2.3.1. Create configmap based on redis.conf file
root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster# cat redis.conf
appendonly yes
cluster-enabled yes
cluster-config-file /var/lib/redis/nodes.conf
cluster-node-timeout 5000
dir /var/lib/redis
port 6379
root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster# 2.3.2. Create configmap
root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster# kubectl create cm redis-conf --from-file=./redis.conf -n magedu
configmap/redis-conf created
root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster# kubectl get cm -n magedu
NAME DATA AGE
kube-root-ca.crt 1 35h
redis-conf 1 6s
root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster# 2.3.3. Verify configmap
root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster# kubectl describe cm redis-conf -n magedu
Name: redis-conf
Namespace: magedu
Labels: <none>
Annotations: <none>
Data
====
redis.conf:
----
appendonly yes
cluster-enabled yes
cluster-config-file /var/lib/redis/nodes.conf
cluster-node-timeout 5000
dir /var/lib/redis
port 6379
BinaryData
====
Events: <none>
root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster#2.3.4. Deploy redis cluster
root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster# cat redis.yaml
apiVersion: v1
kind: Service
metadata:
name: redis
namespace: magedu
labels:
app: redis
spec:
selector:
app: redis
appCluster: redis-cluster
ports:
- name: redis
port: 6379
clusterIP: None
---
apiVersion: v1
kind: Service
metadata:
name: redis-access
namespace: magedu
labels:
app: redis
spec:
type: NodePort
selector:
app: redis
appCluster: redis-cluster
ports:
- name: redis-access
protocol: TCP
port: 6379
targetPort: 6379
nodePort: 36379
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: redis
namespace: magedu
spec:
serviceName: redis
replicas: 6
selector:
matchLabels:
app: redis
appCluster: redis-cluster
template:
metadata:
labels:
app: redis
appCluster: redis-cluster
spec:
terminationGracePeriodSeconds: 20
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- redis
topologyKey: kubernetes.io/hostname
containers:
- name: redis
image: redis:4.0.14
command:
- "redis-server"
args:
- "/etc/redis/redis.conf"
- "--protected-mode"
- "no"
resources:
requests:
cpu: "500m"
memory: "500Mi"
ports:
- containerPort: 6379
name: redis
protocol: TCP
- containerPort: 16379
name: cluster
protocol: TCP
volumeMounts:
- name: conf
mountPath: /etc/redis
- name: data
mountPath: /var/lib/redis
volumes:
- name: conf
configMap:
name: redis-conf
items:
- key: redis.conf
path: redis.conf
volumeClaimTemplates:
- metadata:
name: data
namespace: magedu
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 5Gi
root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster#
The above configuration list mainly uses the sts controller to create 6 pod copies. Each copy uses the configuration file in the configmap as the redis configuration file, uses the pvc template to specify the pod to automatically associate the pv on k8s, and creates the pvc in the magedu namespace. , that is, as long as there is free pv on k8s, the corresponding pod will create a pvc according to the pvc template information in the name space of magedu; of course, we can use the storage class to automatically create the pvc, or we can create the pvc in advance. Generally, we use the sts controller , we can use pvc template to specify pod to automatically create pvc (provided that k8s has enough pv available);
Apply configuration list to deploy redis cluster
Use the sts controller to create a pod. The pod name is the name-id of the sts controller. The name of the pvc created using the pvc template is the pvc template name-pod name, that is, the pvc template name-sts controller name-id;
2.4. Initialize redis cluster
2.4.1. Create a temporary container on k8s and install the redis cluster initialization tool
root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster# kubectl run -it ubuntu1804 --image=ubuntu:18.04 --restart=Never -n magedu bash
If you don't see a command prompt, try pressing enter.
root@ubuntu1804:/#
root@ubuntu1804:/# apt update
# 安装必要工具
root@ubuntu1804:/# apt install python2.7 python-pip redis-tools dnsutils iputils-ping net-tools
# 更新pip
root@ubuntu1804:/# pip install --upgrade pip
# 使用pip安装redis cluster初始化工具redis-trib
root@ubuntu1804:/# pip install redis-trib==0.5.1
root@ubuntu1804:/#2.4.2. Initialize redis cluster
root@ubuntu1804:/# redis-trib.py create \
`dig +short redis-0.redis.magedu.svc.cluster.local`:6379 \
`dig +short redis-1.redis.magedu.svc.cluster.local`:6379 \
`dig +short redis-2.redis.magedu.svc.cluster.local`:6379
On k8s, we use sts to create pods. The name of the corresponding pod is fixed, so when we initialize the redis cluster, we can directly use the redis pod name to directly resolve to the IP address of the corresponding pod; initialize redis on a traditional virtual machine or physical machine For clusters, we can use the IP address directly. The reason is that the IP address of the physical machine or virtual machine is fixed, but the IP address of the pod on k8s is not fixed;
2.4.3. Specify slave for master
-
Specify slave as redis-3 for redis-0
root@ubuntu1804:/# redis-trib.py replicate \
--master-addr `dig +short redis-0.redis.magedu.svc.cluster.local`:6379 \
--slave-addr `dig +short redis-3.redis.magedu.svc.cluster.local`:6379
-
Specify slave as redis-4 for redis-1
root@ubuntu1804:/# redis-trib.py replicate \
--master-addr `dig +short redis-1.redis.magedu.svc.cluster.local`:6379 \
--slave-addr `dig +short redis-4.redis.magedu.svc.cluster.local`:6379
-
Specify the slave as redis-5 for redis-2
root@ubuntu1804:/# redis-trib.py replicate \
--master-addr `dig +short redis-2.redis.magedu.svc.cluster.local`:6379 \
--slave-addr `dig +short redis-5.redis.magedu.svc.cluster.local`:6379
2.5. Verify redis cluster status
2.5.1. Enter any pod in redis cluster to view cluster information
2.5.2. View cluster nodes
The cluster node information records the master node id and slave id, where the slave will correspond to the master's id, indicating that the slave backup corresponds to the master data;
2.5.3. View current node information
127.0.0.1:6379> info
# Server
redis_version:4.0.14
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:165c932261a105d7
redis_mode:cluster
os:Linux 5.15.0-73-generic x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:8.3.0
process_id:1
run_id:aa8ef00d843b4f622374dbb643cf27cdbd4d5ba3
tcp_port:6379
uptime_in_seconds:4303
uptime_in_days:0
hz:10
lru_clock:8272053
executable:/data/redis-server
config_file:/etc/redis/redis.conf
# Clients
connected_clients:1
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
# Memory
used_memory:2642336
used_memory_human:2.52M
used_memory_rss:5353472
used_memory_rss_human:5.11M
used_memory_peak:2682248
used_memory_peak_human:2.56M
used_memory_peak_perc:98.51%
used_memory_overhead:2559936
used_memory_startup:1444856
used_memory_dataset:82400
used_memory_dataset_perc:6.88%
total_system_memory:16740012032
total_system_memory_human:15.59G
used_memory_lua:37888
used_memory_lua_human:37.00K
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
mem_fragmentation_ratio:2.03
mem_allocator:jemalloc-4.0.3
active_defrag_running:0
lazyfree_pending_objects:0
# Persistence
loading:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1685992849
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:245760
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
aof_current_size:0
aof_base_size:0
aof_pending_rewrite:0
aof_buffer_length:0
aof_rewrite_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:0
# Stats
total_connections_received:7
total_commands_processed:17223
instantaneous_ops_per_sec:1
total_net_input_bytes:1530962
total_net_output_bytes:108793
instantaneous_input_kbps:0.04
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:1
sync_partial_ok:0
sync_partial_err:1
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:853
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
# Replication
role:master
connected_slaves:1
slave0:ip=10.200.155.175,port=6379,state=online,offset=1120,lag=1
master_replid:60381a28fee40b44c409e53eeef49215a9d3b0ff
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:1120
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:1120
# CPU
used_cpu_sys:12.50
used_cpu_user:7.51
used_cpu_sys_children:0.01
used_cpu_user_children:0.00
# Cluster
cluster_enabled:1
# Keyspace
127.0.0.1:6379> 2.5.4. Verify whether the redis cluster reads and writes data normally?
2.5.4.1. Manually connect to redis cluster to read and write data
Manually connect to the redis cluster master node to read and write data. There is a problem that when the key we write is calculated modulo 16384 by crc16, the corresponding slot may not be on the current node. Redis will tell us where to write the key. ;As you can see from the screenshot above, redis cluster can now read and write data normally.
2.5.4.2. Use python script to connect to redis cluster for data reading and writing
root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster# cat redis-client-test.py
#!/usr/bin/env python
#coding:utf-8
#Author:Zhang ShiJie
#python 2.7/3.8
#pip install redis-py-cluster
import sys,time
from rediscluster import RedisCluster
def init_redis():
startup_nodes = [
{'host': '192.168.0.34', 'port': 36379},
{'host': '192.168.0.35', 'port': 36379},
{'host': '192.168.0.36', 'port': 36379},
{'host': '192.168.0.34', 'port': 36379},
{'host': '192.168.0.35', 'port': 36379},
{'host': '192.168.0.36', 'port': 36379},
]
try:
conn = RedisCluster(startup_nodes=startup_nodes,
# 有密码要加上密码哦
decode_responses=True, password='')
print('连接成功！！！！！1', conn)
#conn.set("key-cluster","value-cluster")
for i in range(100):
conn.set("key%s" % i, "value%s" % i)
time.sleep(0.1)
data = conn.get("key%s" % i)
print(data)
#return conn
except Exception as e:
print("connect error ", str(e))
sys.exit(1)
init_redis()
root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster# Run the script and write data to redis cluster
root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster# python redis-client-test.py
Traceback (most recent call last):
File "/root/k8s-data/yaml/magedu/redis-cluster/redis-client-test.py", line 8, in <module>
from rediscluster import RedisCluster
ModuleNotFoundError: No module named 'rediscluster'
root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster#Here it is prompted that the rediscluster module is not found. The solution is to install the redis-py-cluster module through pip;
Install redis-py-cluster module
Run the script to connect to redis cluster for data reading and writing
Connect to the redis pod and verify whether the data is written normally?
From the screenshot above, we can see that each of the three reids cluster master pods stores part of the key, not all; it means that we just used a python script to write the data to the redis cluster normally;
Verify that the slave node is available to read data normally?
From the screenshot above, we can understand that data cannot be read on the slave node;
Read data from the master node corresponding to the slave
The above verification shows that only the master in the redis cluster can read and write data, and the slave only backs up the master data and cannot read or write data on the slave;
2.6. Verify the high availability of redis cluster
2.6.1. Upload the redis:4.0.14 image to the local harbor on the k8s node node.
-
Modify image tag
root@k8s-node01:~# nerdctl tag redis:4.0.14 harbor.ik8s.cc/redis-cluster/redis:4.0.14-
Upload the redis image to the local harbor
root@k8s-node01:~# nerdctl push harbor.ik8s.cc/redis-cluster/redis:4.0.14
INFO[0000] pushing as a reduced-platform image (application/vnd.docker.distribution.manifest.list.v2+json, sha256:1ae9e0f790001af4b9f83a2b3d79c593c6f3e9a881b754a99527536259fb6625)
WARN[0000] skipping verifying HTTPS certs for "harbor.ik8s.cc"
index-sha256:1ae9e0f790001af4b9f83a2b3d79c593c6f3e9a881b754a99527536259fb6625: done |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:5bd4fe08813b057df2ae55003a75c39d80a4aea9f1a0fbc0fbd7024edf555786: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:191c4017dcdd3370f871a4c6e7e1d55c7d9abed2bebf3005fb3e7d12161262b8: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 1.4 s total: 8.5 Ki (6.1 KiB/s)
root@k8s-node01:~# 2.6.2. Modify the redis cluster deployment list image and image pull strategy
Modifying the image to a local harbor image and pull strategy is convenient for us to test the high availability of the redis cluster;
2.6.3. Reapply redis cluster deployment list
root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster# kubectl apply -f redis.yaml
service/redis unchanged
service/redis-access unchanged
statefulset.apps/redis configured
root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster# This is equivalent to updating the redis cluster. The cluster relationship between them still exists because the cluster relationship configuration is saved on the remote storage;
-
Verify that all pods are running normally?
-
Verify cluster status and cluster relationships
Different from before, here rdis-0 has become slave and redis-3 has become master; from the screenshot above, we also found that after deploying redis cluster pod on k8s and rebuilding it (the IP address changes), the corresponding cluster relationship will not change. Changes occur; the corresponding relationship between master and salve is always switched between the corresponding master and salve pods. This is actually high availability;
2.6.4. Stop the local harbor, delete the redis master pod, and see if the corresponding slave will be promoted to master?
-
Stop harbor service
root@harbor:~# systemctl stop harbor
-
Delete redis-3 and see if redis-0 will be promoted to master?
You can see that after we deleted redis-3 (equivalent to master downtime), the corresponding slave was promoted to master;
2.6.5. Restore the harbor service and see if it is still the slave of redis-0 after the corresponding redis-3 resume meeting?
-
Restore harbor service
-
Verify whether redis-3pod is restored?
After deleting redis-3 again, the corresponding pod is rebuilt normally and is in the running state;
-
Verify the master-slave relationship of redis-3 ,
You can see that after redis-3 is restored, it automatically joins the cluster and becomes the slave of redis-0;
The article is reproduced from: Linux-1874
Original link: https://www.cnblogs.com/qiuhom-1874/p/17459116.html