I. Introduction
architecture principle: Each Master can have multiple Slave. When the Master offline, Redis cluster will elect a new Master from multiple Slave as an alternative, and Old Master back on the line to become the new Master of Slave.
Second, prepare for the operation
of this deployment is mainly based on the project:
https://github.com/zuxqoj/kubernetes-redis-cluster
It contains two deployment Redis cluster ways:
StatefulSet
Service&Deployment
Two methods have advantages and disadvantages, for services like Redis, Mongodb, Zookeeper such as stateful, using StatefulSet is the preferred way. This article describes how to use StatefulSet deployment Redis cluster.
Three, StatefulSet Profile
RC, Deployment, DaemonSet are for stateless services, they manage Pod of IP, name, start and stop order, etc. are random, but what StatefulSet that? As the name suggests, there is a collection of state, you have to manage all state services, such as MySQL, MongoDB clusters and so on.
On StatefulSet is essentially a variant of Deployment has become a GA version v1.9 version, it has to solve the problem of state services, it manages Pod Pod has a fixed name, start and stop order, in StatefulSet in Pod name called network identification (hostname), must also use the shared storage.
In Deployment, the corresponding service is a service, in the corresponding StatefulSet the headless service, headless service, i.e., headless service, and the service is that it does not distinguish Cluster IP, which will return its name parsing Headless Endpoint list of all corresponding Pod of Service.
In addition, StatefulSet on the basis Headless Service has created a DNS domain name for each Pod copy StatefulSet controlled format of this domain is:
$(podname).(headless server name)
FQDN： $(podname).(headless server name).namespace.svc.cluster.local
That is, for a state service, the best use of our fixed network identity (such as domain information) to mark the node, of course, this also requires application support program (such as Zookeeper is written to support the host domain name in the configuration file).
StatefulSet based Headless Service (ie no Cluster IP of Service) is to achieve a stable Pod logo (including Pod's hostname and DNS Records), also remain unchanged after Pod reschedule. Meanwhile, in conjunction with PV / PVC, StatefulSet possible to achieve stable persistent storage, even after re-scheduling Pod, access to the original or persistent data.
The following is the use Redis StatefulSet deployment architecture, either Master or Slave, both as a copy StatefulSet, and data through the PV for persistence, exposed as a Foreign Service, accept client requests.
Fourth, the deployment process
README herein by reference project, a brief description of the steps to create based on Redis StatefulSet of:
1. Create NFS storage
2. Create the PV
3. Create of PVC
4. Create ConfigMap
5. The Create Service headless
6. Create StatefulSet Redis
7. The initialization cluster Redis
Here, I will refer to the above steps, hands-on and detail the deployment process Redis cluster. This paper will involve many concepts K8S of hope that we can learn to know in advance
1. Create an NFS storage
to create NFS storage primarily to give Redis provide a stable back-end storage, when the Pod Redis restart or migration, it was still able to obtain the original data. Here, we need to create NFS, and then mount a remote NFS path Redis by using PV.
NFS installation
yum -y install nfs-utils（主包提供文件系统）
yum -y install rpcbind（提供rpc协议）
Then, add / etc / exports file, set the need for shared path:
[root@ftp pv3]# cat /etc/exports
/usr/local/k8s/redis/pv1 192.168.0.0/24(rw,sync,no_root_squash)
/usr/local/k8s/redis/pv2 192.168.0.0/24(rw,sync,no_root_squash)
/usr/local/k8s/redis/pv3 192.168.0.0/24(rw,sync,no_root_squash)
/usr/local/k8s/redis/pv4 192.168.0.0/24(rw,sync,no_root_squash)
/usr/local/k8s/redis/pv5 192.168.0.0/24(rw,sync,no_root_squash)
/usr/local/k8s/redis/pv6 192.168.0.0/24(rw,sync,no_root_squash)
Create the appropriate directory
[root@ftp quizii]# mkdir -p /usr/local/k8s/redis/pv{1..6}
Then, start the NFS and rpcbind service:
systemctl restart rpcbind
systemctl restart nfs
systemctl enable nfs
[root@ftp pv3]# exportfs -v
/usr/local/k8s/redis/pv1
192.168.0.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
/usr/local/k8s/redis/pv2
192.168.0.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
/usr/local/k8s/redis/pv3
192.168.0.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
/usr/local/k8s/redis/pv4
192.168.0.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
/usr/local/k8s/redis/pv5
192.168.0.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
/usr/local/k8s/redis/pv6
192.168.0.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
Client
yum -y install nfs-utils
View shared storage end
[root@node2 ~]# showmount -e 192.168.0.222
Export list for 192.168.0.222:
/usr/local/k8s/redis/pv6 192.168.0.0/24
/usr/local/k8s/redis/pv5 192.168.0.0/24
/usr/local/k8s/redis/pv4 192.168.0.0/24
/usr/local/k8s/redis/pv3 192.168.0.0/24
/usr/local/k8s/redis/pv2 192.168.0.0/24
/usr/local/k8s/redis/pv1 192.168.0.0/24
Create PV
each Redis Pod requires a separate PV to store their data, so you can create a pv.yaml file that contains six PV:
[root@master redis]# cat pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs-pv1
spec:
capacity:
storage: 200M
accessModes:
- ReadWriteMany
nfs:
server: 192.168.0.222
path: "/usr/local/k8s/redis/pv1"
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs-vp2
spec:
capacity:
storage: 200M
accessModes:
- ReadWriteMany
nfs:
server: 192.168.0.222
path: "/usr/local/k8s/redis/pv2"
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs-pv3
spec:
capacity:
storage: 200M
accessModes:
- ReadWriteMany
nfs:
server: 192.168.0.222
path: "/usr/local/k8s/redis/pv3"
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs-pv4
spec:
capacity:
storage: 200M
accessModes:
- ReadWriteMany
nfs:
server: 192.168.0.222
path: "/usr/local/k8s/redis/pv4"
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs-pv5
spec:
capacity:
storage: 200M
accessModes:
- ReadWriteMany
nfs:
server: 192.168.0.222
path: "/usr/local/k8s/redis/pv5"
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs-pv6
spec:
capacity:
storage: 200M
accessModes:
- ReadWriteMany
nfs:
server: 192.168.0.222
path: "/usr/local/k8s/redis/pv6"
As can be seen PV are all basically the same except the name and path of the mount. Created to perform:
[root@master redis]#kubectl create -f pv.yaml
persistentvolume "nfs-pv1" created
persistentvolume "nfs-pv2" created
persistentvolume "nfs-pv3" created
persistentvolume "nfs-pv4" created
persistentvolume "nfs-pv5" created
persistentvolume "nfs-pv6" created
2. Create Configmap
here, we can directly convert Redis profile for Configmap, this is a more convenient way to configure read. Profile redis.conf follows
[root@master redis]# cat redis.conf
appendonly yes
cluster-enabled yes
cluster-config-file /var/lib/redis/nodes.conf
cluster-node-timeout 5000
dir /var/lib/redis
port 6379
Creating Configmap named redis-conf of:
kubectl create configmap redis-conf --from-file=redis.conf
View created configmap:
[root@master redis]# kubectl describe cm redis-conf
Name: redis-conf
Namespace: default
Labels: <none>
Annotations: <none>
Data
====
redis.conf:
----
appendonly yes
cluster-enabled yes
cluster-config-file /var/lib/redis/nodes.conf
cluster-node-timeout 5000
dir /var/lib/redis
port 6379
Events: <none>
As above, all the configuration items redis.conf are saved to the Configmap in redis-conf.
3. Create Service Headless
Headless Service is the foundation StatefulSet stable network identity, we need to create in advance. Preparing Files headless-service.yml as follows:
[root@master redis]# cat headless-service.yaml
apiVersion: v1
kind: Service
metadata:
name: redis-service
labels:
app: redis
spec:
ports:
- name: redis-port
port: 6379
clusterIP: None
selector:
app: redis
appCluster: redis-cluster
create:
kubectl create -f headless-service.yml
View:
you can see the service name is redis-service, its CLUSTER-IP to None, that this is a "headless" service.
4. Create Redis cluster node
After creating the Headless service, you can use Redis StatefulSet create a cluster node, which is the core content of this article. We create redis.yml file:
[root@master redis]# cat redis.yaml
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: redis-app
spec:
serviceName: "redis-service"
replicas: 6
template:
metadata:
labels:
app: redis
appCluster: redis-cluster
spec:
terminationGracePeriodSeconds: 20
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- redis
topologyKey: kubernetes.io/hostname
containers:
- name: redis
image: redis
command:
- "redis-server"
args:
- "/etc/redis/redis.conf"
- "--protected-mode"
- "no"
resources:
requests:
cpu: "100m"
memory: "100Mi"
ports:
- name: redis
containerPort: 6379
protocol: "TCP"
- name: cluster
containerPort: 16379
protocol: "TCP"
volumeMounts:
- name: "redis-conf"
mountPath: "/etc/redis"
- name: "redis-data"
mountPath: "/var/lib/redis"
volumes:
- name: "redis-conf"
configMap:
name: "redis-conf"
items:
- key: "redis.conf"
path: "redis.conf"
volumeClaimTemplates:
- metadata:
name: redis-data
spec:
accessModes: [ "ReadWriteMany" ]
resources:
requests:
storage: 200M
As above, a total of six Redis created nodes (Pod), which will be used for three master, the other three were used as a slave master; Redis configuration previously generated by the volume redis-conf ConfigMap this, mounted to the vessel /etc/redis/redis.conf;Redis data storage path using volumeClaimTemplates statement (ie PVC), which binds to the PV we created earlier.
There is a key concept --Affinity, please refer to the official documentation for further details. Which, podAntiAffinity expressed anti-affinity, which determines a pod and which can not be deployed on the same topological domains Pod, POD can be used to disperse a service in a different host or domain topology, improve the stability of the service itself .
And PreferredDuringSchedulingIgnoredDuringExecution said, during the scheduling to try to meet affinity or anti-affinity rules, if the rule is not satisfied, POD may also be scheduled to the corresponding host. In the process of running after, the system will not check whether those rules are met.
Here, matchExpressions provides Redis Pod to try not to schedule contains app is on Node redis also try not to say there is already a redistribution Redis Pod on the Node Redis. However, because we only have three Node, and a copy has six, so according to PreferredDuringSchedulingIgnoredDuringExecution, these peas shall not squeeze, squeeze squeeze more healthy ~
In addition, according to the rules of StatefulSet, 6 Pod our generation's hostname will Redis It is sequentially designated $(statefulset名称)-$(序号)as shown below:
[root@master redis]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
redis-app-0 1/1 Running 0 2h 172.17.24.3 192.168.0.144 <none>
redis-app-1 1/1 Running 0 2h 172.17.63.8 192.168.0.148 <none>
redis-app-2 1/1 Running 0 2h 172.17.24.8 192.168.0.144 <none>
redis-app-3 1/1 Running 0 2h 172.17.63.9 192.168.0.148 <none>
redis-app-4 1/1 Running 0 2h 172.17.24.9 192.168.0.144 <none>
redis-app-5 1/1 Running 0 2h 172.17.63.10 192.168.0.148 <none>
As can be seen in these Pods deployment sequence is {0 ... N-1} are sequentially created. Note that until redis-app-0 reaches Running state after state after starting, redis-app-1 was started.
At the same time, each Pod will get a DNS domain name in the cluster, in the format $(podname).$(service name).$(namespace).svc.cluster.local, that is:
redis-app-0.redis-service.default.svc.cluster.local
redis-app-1.redis-service.default.svc.cluster.local
...以此类推...
K8S within the cluster, which can use the domain Pod communicate with each other. We can use busybox mirrored nslookup test these domains:
[root@master redis]# kubectl exec -ti busybox -- nslookup redis-app-0.redis-service
Server: 10.0.0.2
Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local
Name: redis-app-0.redis-service
Address 1: 172.17.24.3
You can see, redis-app-0's IP is 172.17.24.3. Of course, if Redis Pod migration or restart (We can delete manually to test out a Redis Pod), IP also will not change (probably based on the reason StatefulSet) , Pod domain, SRV records, A record will not change .
Also can be found, pv we created earlier were successfully bound:
[root@master redis]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
nfs-pv1 200M RWX Retain Bound default/redis-data-redis-app-2 3h
nfs-pv3 200M RWX Retain Bound default/redis-data-redis-app-4 3h
nfs-pv4 200M RWX Retain Bound default/redis-data-redis-app-5 3h
nfs-pv5 200M RWX Retain Bound default/redis-data-redis-app-1 3h
nfs-pv6 200M RWX Retain Bound default/redis-data-redis-app-0 3h
nfs-vp2 200M RWX Retain Bound default/redis-data-redis-app-3 3h
5. Initialize Redis cluster
Once you've created six Redis Pod, we also need to use common tools Redis-tribe cluster initialization
Ubuntu create container
due to Redis cluster must be initialized after all nodes to start, but if the initialization logic is written in Statefulset, it is a very complex and inefficient behavior. Here, I have to praise the author of the original project ideas worth learning. In other words, we can create an additional container on K8S, dedicated cluster management control internal K8S certain services.
Here, we specialize start a container Ubuntu, you can install Redis-tribe in the container, and then initialize Redis cluster, execute:
kubectl run -it ubuntu --image=ubuntu --restart=Never /bin/bash
We use Ali cloud Ubuntu source, execute:
root@ubuntu:/# cat > /etc/apt/sources.list << EOF
deb http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
> EOF
After the success of the original project required to perform the following command to install the basic software environment:
apt-get update
apt-get install -y vim wget python2.7 python-pip redis-tools dnsutils
Initialize cluster
First, we need to install redis-trib:
pip install redis-trib==0.5.1
Then, create only Master node of the cluster:
redis-trib.py create \
`dig +short redis-app-0.redis-service.default.svc.cluster.local`:6379 \
`dig +short redis-app-1.redis-service.default.svc.cluster.local`:6379 \
`dig +short redis-app-2.redis-service.default.svc.cluster.local`:6379
Secondly, each Adding Slave Master
redis-trib.py replicate \
--master-addr `dig +short redis-app-0.redis-service.default.svc.cluster.local`:6379 \
--slave-addr `dig +short redis-app-3.redis-service.default.svc.cluster.local`:6379
redis-trib.py replicate \
--master-addr `dig +short redis-app-1.redis-service.default.svc.cluster.local`:6379 \
--slave-addr `dig +short redis-app-4.redis-service.default.svc.cluster.local`:6379
redis-trib.py replicate \
--master-addr `dig +short redis-app-2.redis-service.default.svc.cluster.local`:6379 \
--slave-addr `dig +short redis-app-5.redis-service.default.svc.cluster.local`:6379
At this point, we would really Redis cluster is created, and even test it to any of the Redis Pod:
[root@master redis]# kubectl exec -it redis-app-2 /bin/bash
root@redis-app-2:/data# /usr/local/bin/redis-cli -c
127.0.0.1:6379> cluster nodes
5d3e77f6131c6f272576530b23d1cd7592942eec 172.17.24.3:6379@16379 master - 0 1559628533000 1 connected 0-5461
a4b529c40a920da314c6c93d17dc603625d6412c 172.17.63.10:6379@16379 master - 0 1559628531670 6 connected 10923-16383
368971dc8916611a86577a8726e4f1f3a69c5eb7 172.17.24.9:6379@16379 slave 0025e6140f85cb243c60c214467b7e77bf819ae3 0 1559628533672 4 connected
0025e6140f85cb243c60c214467b7e77bf819ae3 172.17.63.8:6379@16379 master - 0 1559628533000 2 connected 5462-10922
6d5ee94b78b279e7d3c77a55437695662e8c039e 172.17.24.8:6379@16379 myself,slave a4b529c40a920da314c6c93d17dc603625d6412c 0 1559628532000 5 connected
2eb3e06ce914e0e285d6284c4df32573e318bc01 172.17.63.9:6379@16379 slave 5d3e77f6131c6f272576530b23d1cd7592942eec 0 1559628533000 3 connected
127.0.0.1:6379> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:6
cluster_stats_messages_ping_sent:14910
cluster_stats_messages_pong_sent:15139
cluster_stats_messages_sent:30049
cluster_stats_messages_ping_received:15139
cluster_stats_messages_pong_received:14910
cluster_stats_messages_received:30049
127.0.0.1:6379>
In addition, you can also view data in Redis mounted on NFS:
[root@ftp pv3]# ll /usr/local/k8s/redis/pv3
total 12
-rw-r--r-- 1 root root 92 Jun 4 11:36 appendonly.aof
-rw-r--r-- 1 root root 175 Jun 4 11:36 dump.rdb
-rw-r--r-- 1 root root 794 Jun 4 11:49 nodes.conf
6. Create Service for access to
the front we created for implementing StatefulSet the Headless Service, but not the Service Cluster Ip, and therefore can not be used outside access. Therefore, we also need to create a Service, dedicated to providing access and load balancing for the cluster Redis:
[root@master redis]# cat redis-access-service.yaml
apiVersion: v1
kind: Service
metadata:
name: redis-access-service
labels:
app: redis
spec:
ports:
- name: redis-port
protocol: "TCP"
port: 6379
targetPort: 6379
selector:
app: redis
appCluster: redis-cluster
As above, the Service name redis-access-service, port 6379 exposed in K8S cluster, and would have labels nameto app: redisor appCluster: redis-clusterload balancing pod.
After you create a view:
[root@master redis]# kubectl get svc redis-access-service -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
redis-access-service ClusterIP 10.0.0.64 <none> 6379/TCP 2h app=redis,appCluster=redis-cluster
As above, in K8S cluster, all applications can 10.0.0.64 :6379be accessed Redis cluster. Of course, in order to facilitate testing, we can also add a NodePort Service mapped to a physical machine, not described in detail here.
Fifth, the test master switch from
the building intact Redis cluster on K8S, we are most concerned about is its original high availability mechanism is normal. Here, we can arbitrarily pick a Master of the main cluster from Pod to test the switching mechanism, such as redis-app-0:
[root@master redis]# kubectl get pods redis-app-0 -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
redis-app-1 1/1 Running 0 3h 172.17.24.3 192.168.0.144 <none>
Enter redis-app-0Views:
[root@master redis]# kubectl exec -it redis-app-0 /bin/bash
root@redis-app-0:/data# /usr/local/bin/redis-cli -c
127.0.0.1:6379> role
1) "master"
2) (integer) 13370
3) 1) 1) "172.17.63.9"
2) "6379"
3) "13370"
127.0.0.1:6379>
As can be seen, app-0as a master, slave is 172.17.63.9namely redis-app-3.
We then manually delete redis-app-0:
[root@master redis]# kubectl delete pod redis-app-0
pod "redis-app-0" deleted
[root@master redis]# kubectl get pod redis-app-0 -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
redis-app-0 1/1 Running 0 4m 172.17.24.3 192.168.0.144 <none>
We re-entering the redis-app-0internal view:
[root@master redis]# kubectl exec -it redis-app-0 /bin/bash
root@redis-app-0:/data# /usr/local/bin/redis-cli -c
127.0.0.1:6379> role
1) "slave"
2) "172.17.63.9"
3) (integer) 6379
4) "connected"
5) (integer) 13958
Above, redis-app-0into a Slave, it is subordinate to the previous node 172.17.63.9, ie redis-app-3.