Detailed Explanation of StatefulSet of k8s

This article is based on the k8s-v1.15.0 version.

1 Introduction

RC, Deployment, and DaemonSet are all stateless services. The IP, name, start-stop sequence, etc. of the Pods they manage are all random, but what is StatefulSet? As the name implies, a stateful collection manages all stateful services, such as MySQL and MongoDB clusters.
StatefulSet is essentially a variant of Deployment. It has become the GA version in v1.9. In order to solve the problem of stateful services, the Pod it manages has a fixed Pod name, start and stop order, in StatefulSet, The Pod name is called the hostname, and shared storage must also be used.
In Deployment, the corresponding service is service, and in StatefulSet, the corresponding headless service, headless service, is headless service. The difference with service is that it does not have a Cluster IP, and the Headless will be returned when its name is resolved. The Endpoint list of all Pods corresponding to the Service.
In addition, StatefulSet creates a DNS domain name for each Pod copy controlled by StatefulSet based on Headless Service. The format of this domain name is:
$(podname).(headless server name)
FQDN：$(podname).(headless server name).namespace.svc.cluster.local

2. Features

Pod consistency: including sequence (start, stop sequence), network consistency. This consistency is related to Pod, and has nothing to do with which node is scheduled;
Stable order: For a StatefulSet with N copies, each Pod is assigned a numerical sequence number in the range of [0, N), and it is unique;
Stable network: Pod's hostname mode is $\left(STATEFULSETnamecalled\right)-$ (number);
stable storage: create a PV per Pod by VolumeClaimTemplate. Deleting and reducing copies will not delete related volumes.

3. Components

Headless Service: used to define the Pod network identifier (DNS domain);
volumeClaimTemplates: storage volume application template, create PVC, specify the size of the PVC name, and automatically create PVC, and the PVC must be supplied by the storage class;
StatefulSet: define the specific application, named Nginx has three Pod copies, and defines a domain name for each Pod to deploy statefulset.

Why do you need headless service?
When using Deployment, each Pod name is not ordered, it is a random string, so the Pod name is disordered, but it must be ordered in the statefulset, and each pod cannot be replaced at will. After the pod is rebuilt, the pod The name is still the same. The pod IP changes, so it is identified by the name of the pod. The pod name is the unique identifier of the pod and must be stable and effective. At this time, headless service is used, which can give each Pod a unique name.

Why do I need volumeClaimTemplate?
For stateful replica sets, persistent storage is used. For distributed systems, its biggest feature is that the data is different, so each node cannot use the same storage volume, and each node has its own dedicated storage, but if The storage volume defined in the Pod template in the Deployment is a storage volume shared by all replica sets. The data is the same because it is based on the template, and each Pod in the statefulset has its own dedicated storage volume, so the statefulset The storage volume can no longer be created with a Pod template, so statefulSet uses a volumeClaimTemplate, which is called a volume application template. It will generate a different PVC for each Pod and bind it to a PV, so that each pod has dedicated storage. This is why volumeClaimTemplate is used.

4. Detailed Explanation of StatefulSet

kubectl explain sts.spec: Main field explanation
replicas: Number of copies
selector: The pod is managed by itself
serviceName: Must be associated with a headless service provider
template: Define the pod template (which defines the associated storage volume)
volumeClaimTemplates: Generate PVC

5. Deploy a statefulset service

This tutorial assumes that your cluster is configured to dynamically provide PersistentVolume. For dynamic PV, refer to ks8's data management-dynamic configuration of StorageClass ; if not configured like this, you need to manually prepare storage volumes before starting this tutorial.
If there is no StorageClass dynamic provisioning PVC mechanism in the cluster, you can also manually create multiple PVs and PVCs in advance. The manually created PVC name must conform to the StatefulSet naming rules created later: (volumeClaimTemplates.name)-(pod_name)

5.1. Create a pv based on dynamic sc

step1: Create a namespace for statefueset

cat << EOF > nginx-ns.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: nginx-ss
EOF

step2, create dynamic storage based on sc

cat << EOF > nginx-sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nginx-nfs-storage
provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
  archiveOnDelete: "false" # # When set to "false" your PVs will not be archived
                           # by the provisioner upon deletion of the PVC.
EOF

kubectl apply -f nginx-sc.yaml

step3, create a statefulset

cat << EOF > nginx-ss.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
  namespace: nginx-ss
spec:
  selector:
    matchLabels:
      app: nginx #必须匹配 .spec.template.metadata.labels
  serviceName: "nginx"  #声明它属于哪个Headless Service.
  replicas: 3 #副本数
  template:
    metadata:
      labels:
        app: nginx # 必须配置 .spec.selector.matchLabels
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: nginx
        image: www.my.com/web/nginx:v1
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: nginx-pvc
          mountPath: /usr/share/nginx/html

  volumeClaimTemplates:   #可看作pvc的模板
  - metadata:
      name: nginx-pvc
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "nginx-nfs-storage"  #存储类名，改为集群中已存在的
      resources:
        requests:
          storage: 1Gi
EOF

kubectl apply -f nginx-ss.yaml

step4, observe the creation of pod, you will find that it is created in order

kubectl get pod -n nginx-ss

step5, create svc

cat << EOF > nginx-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: nginx-ss
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: nginx
EOF

kubectl apply -f nginx-svc.yaml

step6, check sc, pv, pvc, pod, statefulset, svc

5.2, verification analysis

Each Pod has a stable hostname based on its sequential index.

Use kubectl run to run a container that provides the nslookup command, which comes from the dnsutils package. By executing nslookup on the hostname of the Pod, you can check their DNS address inside the cluster
kubectl run -i --tty --image www.my.com/k8s/busybox:1.27 -n nginx-ss dns-test - restart=Never --rm
nslookup web-0.nginx

can find that the CNAME of the headless service points to the SRV record (recording each Pod in Running and Ready status). The SRV record points to a record entry that contains the Pod IP address.
Restarting the pod will find that the ip in the pod has changed, but the name of the pod has not changed; this is why you should not use the IP address of the Pod in the StatefulSet to connect in other applications. This is very important

for the serial number and host of the pod The name, SRV entry, and record name have not changed, but the IP address associated with the Pod may have changed.
If you need to find and connect to an active member of a StatefulSet, you should query the CNAME of Headless Service. The SRV records associated with the CNAME will only contain Pods in the Running and Ready states in the StatefulSet.

If your application has implemented the connection logic for testing liveness and readiness, you can use Pod SRV records (web-0.nginx.nginx-ss.svc.cluster.local, web-1.nginx.nginx-ss .svc.cluster.local, web-2.nginx.nginx-ss.svc.cluster.local). Because they are stable, and when the status of your Pod becomes Running and Ready, your application can discover their addresses.

5.3, write to stable storage

Write the hostname of the Pod into their index.html file and verify that the NGINX web server uses the hostname to provide services

kubectl exec -it web-0 -n nginx-ss /bin/bash
echo $(hostname) > /usr/share/nginx/html/index.html
kubectl exec -it web-1 -n nginx-ss /bin/bash
echo $(hostname) > /usr/share/nginx/html/index.html
kubectl exec -it web-2 -n nginx-ss /bin/bash
echo $(hostname) > /usr/share/nginx/html/index.html

After the pod is deleted, even if the pod’s ip changes, the access

is still not affected. Although web-0, web-1 and web-2 are rescheduled, they still continue to monitor their respective hostnames because they are associated with their PersistentVolumeClaim The PersistentVolume was remounted to its respective volumeMount. No matter which node web-0, web-1, web-3 are scheduled to, their PersistentVolumes will be mounted to the appropriate mount point

5.4, ​​expand/shrink StatefulSet

Expanding/shrinking a StatefulSet refers to increasing or decreasing the number of copies of it. This is done by updating the replicasfields. You can use kubectl scale or kubectl patch to expand/shrink a StatefulSet.

kubectl scale sts web --replicas=4 -n nginx-ss   #扩容
kubectl scale sts web --replicas=2 -n nginx-ss   #缩容

or

kubectl patch sts web -p '{"spec":{"replicas":4}}' -n nginx-ss  #扩容
kubectl patch sts web -p '{"spec":{"replicas":2}}' -n nginx-ss  #缩容

You will find that the expansion is, if the pv is a dynamic sc, the pvc will also increase, but when shrinking, the pvc will not be deleted by itself

6. Case (Deploy mysql)

step1: Create mysql namespace, sc

cat << EOF > mysql-ns.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: mysql-sts
EOF

cat << EOF > mysql-sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: mysql-nfs-storage
provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
  archiveOnDelete: "false" # # When set to "false" your PVs will not be archived
                           # by the provisioner upon deletion of the PVC.  
EOF

kubectl apply -f mysql-ns.yaml
kubectl apply -f mysql-sc.yaml

step2: deploy mysql statefulset

cat << EOF > mysql-ss.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
  namespace: mysql-sts
spec:
  selector:
    matchLabels:
      app: mysql #必须匹配 .spec.template.metadata.labels
  serviceName: "mysql"  #声明它属于哪个Headless Service.
  replicas: 3 #副本数
  template:
    metadata:
      labels:
        app: mysql # 必须配置 .spec.selector.matchLabels
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: mysql
        image: www.my.com/sys/mysql:5.7
        ports:
        - containerPort: 3306
        name: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: "123456"
        volumeMounts:
        - name: mysql-pvc
          mountPath: /var/lib/mysql

  volumeClaimTemplates:   #可看作pvc的模板
  - metadata:
      name: mysql-pvc
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "mysql-nfs-storage"  #存储类名，改为集群中已存在的
      resources:
        requests:
          storage: 1Gi
EOF

kubectl apply -f mysql-ss.yaml

step3: create svc

cat << EOF > mysql-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: mysql
  namespace: mysql-sts
  labels:
    app: mysql
spec:
  ports:
  - port: 3306
    name: mysql
  clusterIP: None
  selector:
    app: mysql
EOF

kubectl apply -f mysql-svc.yaml

step4: View

step5: access test

And the three mysql pod data are independent of each other

step6: write mysql-read svc

cat << EOF > mysql-svc-port.yaml
apiVersion: v1
kind: Service
metadata:
  name: mysql-read
  namespace: mysql-sts
  labels:
    app: mysql-2
spec:
  ports:
  - port: 3306
    name: mysql
  selector:
    app: mysql-2
EOF   

kubectl apply -f mysql-svc-port.yaml

Allow mysql-read to be accessed externally.
Use NodePort as the access method.
Execute commands. kubectl edit svc/mysql-read -n mysql-sts
Enter the modification interface, modify type为NodePort, and spec/portsadd nodePort: 31988configuration to it.
Execute again, kubectl get svc -n mysql-sts

log in through the mysql client tool on the local computer