In-depth Kubernetes: K8s container persistent storage operations

Recommended reading:

• I summarized 72 interview questions, totaling 3170 pages, and won 30+ Internet company offers (including BATJM)

• I won the first battle in 2020. This Java Interview Magic Plus version allowed me to successfully get offers from major manufacturers such as Ali, JD.com, and Bytedance.

• Under the epidemic situation, I received a phone interview from Meituan (successfully won the offer), with a learning route + a question bank

Start with PV and PVC from an example

The Kubernetes project introduced a set of API objects called Persistent Volume Claim (PVC) and Persistent Volume (PV) to manage storage volumes.

Simply put, PersistentVolume (PV) is a segment of network storage configured by the administrator in the cluster, which is a persistent storage data volume; Persistent Volume Claim (PVC) describes the properties of persistent storage that Pod wants to use, such as , Volume storage size, read and write permissions, etc.

The above text description may be too vague, let's take an example:

We define a PVC and declare the required Volume attributes:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: manual
  resources:
    requests:
      storage: 1Gi

A 1 GiB PVC is defined in the yaml file, Access Modes indicates the required volume storage type, ReadWriteOnce indicates that read and write operations can only be performed on one node node. For other Access Modes, see: https://kubernetes.io/docs /concepts/storage/persistent-volumes/#access-modes .

Then define a PV:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs
spec:
  storageClassName: manual
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 10.244.1.4
    path: "/"

This PV object will define in detail that the storage type is NFS and the size is 1 GiB.

PVC and PV are equivalent to "interface" and "implementation", so we need to bind PVC and PV before they can be used, and when binding PVC and PV, we need to meet:

1. To match the spec field of PV and PVC, such as the storage size of PV, it must meet the requirements of PVC.
2. The storageClassName fields of PV and PVC must be the same for binding. storageClassName represents the name attribute of StorageClass.

After the PVC statement is made and the PV is established, the PVC can be used:

apiVersion: v1
kind: Pod
metadata:
  labels:
    role: web-frontend
spec:
  containers:
  - name: web
    image: nginx
    ports:
      - name: web
        containerPort: 80
    volumeMounts:
        - name: nfs
          mountPath: "/usr/share/nginx/html"
  volumes:
  - name: nfs
    persistentVolumeClaim:
      claimName: nfs

You only need to declare the name of the PVC in the Pod. After the Pod is created, kubelet will mount the PV corresponding to the PVC, which is an NFS-type Volume, on the directory in the Pod container.

PersistentVolumeController will continuously check whether each current PVC is in the Bound state. If it is not, it will traverse all available PVs and try to bind them to this "single" PVC. So if there is no PV that can be bound to PVC, then the Pod will start an error.

At this time, StorageClass is needed. The process of binding PV and PVC mentioned above is called Static Provisioning, which requires manual creation of PV; StorageClass also provides a Dynamic Provisioning mechanism that can create PV based on a template.

StorageClass的Dynamic Provisioning

The StorageClass object defines the following two parts:

1. The attributes of PV. For example, storage type, volume size, etc.
2. The storage plug-in needed to create this kind of PV. For example, Ceph and so on.

In this way, k8s can find a corresponding StorageClass according to the PVC submitted by the user, and then call the storage plug-in declared by the StorageClass to create the required PV.

For example, declare the following StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: block-service
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-ssd

The StorageClass named block-service is defined here. The value of the provisioner field is: kubernetes.io/gce-pd, which is a built-in storage plug-in of k8s. The type field is also defined with provisioner. The official default supports the built-in storage plug-in of Dynamic Provisioning : Https://kubernetes.io/docs/concepts/storage/storage-classes/ .

Then you can declare the storageClassName as block-service in the PVC. After the PVC object is created, k8s will call the corresponding storage plug-in API to create a PV object.

as follows:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: claim1
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: block-service
  resources:
    requests:
      storage: 30Gi

This automatic PV creation mechanism is Dynamic Provisioning. Kubernetes can find a corresponding StorageClass based on the PVC submitted by the user, and then call the storage plugin declared by the StorageClass to create the required PV.

It should be noted that if the StorageClassName is not declared in the PVC, the value of the storageClassName of the PVC is "", which also means that it can only be bound to the PV whose storageClassName is also "".

Life cycle of PV and PVC

The interaction between PV and PVC follows this life cycle:

Provisioning —>Binding —>Using —>Reclaiming

Provisioning

k8s provides two PV generation methods: statically or dynamically

statically: PVs are created by the administrator, and they carry detailed information about the real storage available to cluster users. They exist in the Kubernetes API and can be used for consumption.

dynamically: When the static PV created by the administrator does not match the user's PersistentVolumeClaim, the cluster may try to dynamically configure the volume for the PVC. This configuration is based on StorageClasses. The PVC must request a StorageClasses, and the administrator must have created and configured the class to be dynamically configured.

Binding

After the PersistentVolumeClaim is created by the user, the PersistentVolumeController will continuously check whether each current PVC is already in the Bound state. If it is not, it will traverse all available PVs and try to bind them to this "single" PVC.

Using

After the Pods declare and use the PVC as the volume, the cluster will find the PVC. If the PVC is already bound to the PV, it will mount the volume to the Pod.

Reclaiming

When the user no longer uses the volume, the PVC can be deleted so that the resources can be recycled. Correspondingly, after the PVC is deleted, the PV recycling strategy can be Retained, Recycled, or Deleted. This strategy can be set in the field spec.persistentVolumeReclaimPolicy.

• Retain: This policy allows manual recovery of resources. When the PVC is deleted, the PV can still exist. The administrator can manually delete the PV, and the storage resources bound to the PV will not be deleted. If you want to delete the corresponding storage For resource data, you need to manually delete the data corresponding to the storage resource.
• Delete: This strategy will delete PV and storage resources managed by PV after the PVC is deleted.
• Recycle: It is equivalent to executing the rm -rf /thevolume/* command in the volume so that the volume can be reused.

Delete process

Under normal circumstances, we follow this deletion process:

1. Delete the Pod using this PV;
2. Remove the local disk from the host (for example, umount it);
3. Delete PVC;
4. Delete the PV.

Local Persistent Volume actual combat

Local Persistent Volume is suitable for applications that are similar to distributed data storage such as MongoDB and Cassandra that need to store data on multiple different nodes and are more sensitive to I/O. But compared with normal PV, once these nodes are down and cannot be recovered, the data of Local Persistent Volume may be lost.

In our experimental environment, mount several RAM Disks (memory disks) on the host to simulate local disks. E.g:

We mount several disks on node1

$ mkdir /mnt/disks
$ for vol in vol1 vol2 vol3; do
    mkdir /mnt/disks/$vol
    mount -t tmpfs $vol /mnt/disks/$vol
done

Then create the corresponding PV:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: example-pv
spec:
  capacity:
    storage: 512Mi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /mnt/disks/vol1
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node1

In the definition of this PV: the local field specifies that it is a Local Persistent Volume; and the path field specifies the path of the local disk corresponding to this PV, namely: /mnt/disks/vol1. And use nodeAffinity to specify that this PV must run on node1.

Run the above PV:

$ kubectl create -f local-pv.yaml 
persistentvolume/example-pv created

$ kubectl get pv
NAME         CAPACITY   ACCESS MODES   RECLAIM POLICY  STATUS      CLAIM             STORAGECLASS    REASON    AGE
example-pv   512Mi        RWO            Delete           Available                     local-storage             16s

Then create a StorageClass to describe this PV:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

This StorageClass is called local-storage, and the provisioner is no-provisioner, which means that PV does not need to be created automatically.

volumeBindingMode=WaitForFirstConsumer means that you need to wait until the Pod is running before letting the PVC and PV bind. Because when using Local Persistent Volume, the PV and the corresponding PVC must follow the Pod under the same node, otherwise the scheduling will fail.

Then we run StorageClass:

$ kubectl create -f local-sc.yaml 
storageclass.storage.k8s.io/local-storage created

Create another PVC:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: example-local-claim
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 512Mi
  storageClassName: local-storage

Note here that the storageClassName needs to be the StorageClass we created above.

Then create the PVC:

$ kubectl create -f local-pvc.yaml 
persistentvolumeclaim/example-local-claim created

$ kubectl get pvc
NAME                  STATUS    VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS    AGE
example-local-claim   Pending                                       local-storage   7s

At this time, because the Pod has not been created yet, the status is still Pending.

Create a pod:

kind: Pod
apiVersion: v1
metadata:
  name: example-pv-pod
spec:
  volumes:
    - name: example-pv-storage
      persistentVolumeClaim:
       claimName: example-local-claim
  containers:
    - name: example-pv-container
      image: nginx
      ports:
        - containerPort: 80
          name: "http-server"
      volumeMounts:
        - mountPath: "/usr/share/nginx/html"
          name: example-pv-storage

Then we will look at the PVC binding status after creating the pod:

$ kubectl create -f local-pod.yaml 
pod/example-pv-pod created

$ kubectl get pvc
NAME                  STATUS    VOLUME       CAPACITY   ACCESS MODES   STORAGECLASS    AGE
example-local-claim   Bound     example-pv   512Mi        RWO            local-storage   6h

Then we try to write a file to /usr/share/nginx/html:

$ kubectl exec -it example-pv-pod -- /bin/sh
# cd /usr/share/nginx/html
# touch test.txt

# 在node1上
$ ls /mnt/disks/vol1
test.txt