Detailed data storage of kubernetes

Table of contents

1. The role of storage volumes

2. Data volume overview

3. Data volume emptyDir

4. Data volume hostPath

5. Data volume: NFS

6. Persistent volume overview

6.1PV ​​static supply

6.2PV dynamic supply (StorageClass)

6.3 PV life cycle

6.3.1 ACCESS MODES (access mode)

6.3.2 RECLAIM POLICY (recycling policy)

6.3.3 STATUS (Status)

6.3.4PV and PVC life cycle

---

1. The role of storage volumes

---

There are generally three types of data in the container deployment process:

• Initial data required at startup, such as configuration files

• Temporary data generated during startup, which needs to be shared among multiple containers

• Persistent data generated during startup, such as MySQL's data directory

---

2. Data volume overview

---

• Volume in Kubernetes provides the ability to mount external storage in containers

• The Pod needs to set the volume source (spec.volume) and the mount point (spec.containers.volumeMounts) before using the corresponding Volume

Data volume types are roughly classified as:

• local (hostPath, emptyDir, etc.)

• Network (NFS, Ceph, GlusterFS, etc.)

• Public cloud (AWS EBS, etc.)

• K8S resources (configmap, secret, etc.)

---

3. Data volume emptyDir

---

emptyDir volume : It is a temporary storage volume, which is bound to the Pod life cycle. If the Pod deletes the volume, it will also be deleted.

Application scenario: Data sharing between containers in a Pod

Official Example : Volumes | Kubernetes

Create emptyDir.yaml

apiVersion: v1
kind: Pod
metadata:
  name: emptydir-pod
spec:
  containers:
  - name: write   
    image: centos
    command: ["bash","-c","for i in {1..100};do echo $i >> /data/hello;sleep 1;done"]
    volumeMounts:
      - name: data          
        mountPath: /data
  - name: read    
    image: centos
    command: ["bash","-c","tail -f /data/hello"] 
    volumeMounts:
      - name: data          
        mountPath: /data    ## 挂载点
  volumes:
  - name: data  
    emptyDir: {}   ##  数据卷类型

emptyDir will create an empty-Dir folder locally where the pod is located to store the mounted files

Enter the write container, we can see that the /data/hello file is written normally

---

4. Data volume hostPath

---

hostPath volume : Mount the file or directory on the Node file system (the node where the Pod is located) to the container in the Pod.

Application scenario: Containers in a Pod need to access host files

Official Example : Volumes | Kubernetes

Create hostPath.yaml

apiVersion: v1
kind: Pod
metadata:
  name: hostpath-pod
spec:
  containers:
  - name: busybox 
    image: busybox 
    args:  
    - /bin/sh
    - -c
    - sleep 36000   
    volumeMounts:
    - name: data          
      mountPath: /data
  volumes:
  - name: data 
    hostPath:  
      path: /tmp
      type: Directory

View the node where hostpath-pod is located

---

5. Data volume: NFS

---

NFS data volume: Provides support for NFS mounting, and can automatically mount NFS shares and paths to Pods

NFS: It is a mainstream file sharing server.

# 安装包，所有访问共享文件的 服务器
yum install nfs-utils

## 编辑配置
vi /etc/exports
/ifs/kubernetes *(rw,no_root_squash)

## 创建共享文件夹
mkdir -p /ifs/kubernetes
systemctl start nfs
systemctl enable nfs
注：每个Node上都要安装nfs-utils包

Use k8s-node1 as NFS Server to create a data storage path 

Manually mount the /mnt directory to the NFS Server on the k8s-master1 node

[root@k8s-master1 ~]# mount -t nfs  192.168.2.118:/ifs/kubernetes /mnt
[root@k8s-master1 ~]# umount /mnt 

 Create a file in the /mnt directory of the k8s-master1 node to verify whether the nfs mount is successful

Create nginx-nfs.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: depployment-nfs
spec:
  selector:
    matchLabels:
      app: nginx-nfs
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx-nfs
    spec:
      containers:
      - name: nginx
        image: nginx
        volumeMounts:
        - name: wwwroot
          mountPath: /usr/share/nginx/html
        ports:
        - containerPort: 80
      volumes:
      - name: wwwroot
        nfs:
          server: 192.168.2.118 
          path: /ifs/kubernetes

Create 3 nfs pods 

We enter the container and create index.html, successfully accessed in k8s-node1

---

6. Persistent volume overview

---

There are many storage systems supported by kubernetes, and it is obviously unrealistic to require customers to master all of them. In order to shield the details of the underlying storage implementation and facilitate users' use, kubernetes introduces two resource objects, PV and PVC. 

• The full name of PV is Persistent Volume, persistent storage volume . It is used to describe or define a storage volume, which is usually defined by operation and maintenance engineers.
• The full name of PVC is Persistent Volume claim, which is a request for persistent storage . It is used to describe what kind of PIV storage you want to use or meet what conditions.

PVC usage logic : Define a storage volume in the Pod (the storage volume type is PVC), specify the size directly when defining, the PVC must establish a relationship with the corresponding PV, and the PVC will go to the PV application according to the definition of the configuration, and the PV is Created from storage space. PV and PVC are storage resources abstracted by Kubernetes.

PV is an abstraction of storage resources. PV resources belong to cluster resources and cross namespaces (not isolated by namespaces)

 PV resource manifest file:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv2
spec:
  nfs: # Storage type, corresponding to the underlying real storage
  capacity: # Storage capacity, currently only supports the setting of storage space
    storage: 2Gi
  accessModes: #Access mode
  storageClassName: #Storage category
  persistentVolumeReclaimPolicy: #reclaim policy

Description of the key configuration parameters of PV:
storage type
    The underlying actual storage type, kubernetes supports multiple storage types (nfs, cifs, glusterfs, etc.), and the configuration of each storage type is different
Storage capacity (capacity)
    currently only supports storage space ( storage=1Gi ), but in the future, the configuration access
mode (accessModes)
of indicators such as IOPS and throughput may be added     to describe the access rights of user applications to storage resources. The access rights include the following methods:
        ReadWriteOnce (RWO) : Read and write permissions, but can only be mounted by a single node
        ReadOnlyMany (ROX): Read-only permissions, can be mounted by multiple nodes
        ReadWriteMany (RWX): Read and write permissions, can be mounted by multiple nodes
        It should be noted that, Different underlying storage types may support different access modes and different
recycling policies (persistentVolumeReclaimPolicy)
    when the PV is no longer used, how to deal with it. Three strategies are currently supported:
        Retain (retain) to retain data, requires the administrator to manually clean up the data
        Recycle (recycle) clears the data in the PV, the effect is equivalent to executing rm -rf /thevolume/*
        Delete (delete) the backend connected to the PV The storage completes the volume deletion operation. Of course, this is common in the storage services of cloud service providers. It
        should be noted that different underlying storage types may support different recovery strategies
The storage class
    PV can specify a storage class through the storageClassName parameter. A
        PV with a specific class can only be bound to a PVC that requests this class.
        A PV that does not set a class can only be bound to a PVC that does not request any class.
Status (status )
    In the life cycle of a PV, it may be in four different stages:
        Available (available): Indicates that it is available and has not been bound by any PVC Bound
        (bound): Indicates that the PV has been bound by a PVC Released
        (has been bound) Release): Indicates that the PVC has been deleted, but the resource has not been re-declared by the cluster
        Failed (failure): Indicates that the automatic recycling of the PV failed

6.1PV ​​static supply

Create pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  nfs:
    path: /ifs/kubernetes
    server: 192.168.2.118

Create deployment-pvc.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: depployment-pvc
spec:
  selector:
    matchLabels:
      app: nginx-pvc
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx-pvc
    spec:
      containers:
      - name: nginx
        image: nginx
        volumeMounts:
        - name: wwwroot
          mountPath: /usr/share/nginx/html
        ports:
        - containerPort: 80
      volumes:
      - name: wwwroot
        persistentVolumeClaim:  
          claimName: my-pvc   ## 引用PVC

---
apiVersion: v1    
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  accessModes:      ## 读写权限
  - ReadWriteMany
  resources:
    requests:       ## 请求的资源大小
      storage: 1Gi

Create my-pv, resource 1G

Create the my-pv used by deployment-pvc.yaml, use 1G storage, and the permission is read, write and execute

We modified the content of the nginx container and the access was successful. It means that creating PV is OK.

6.2PV dynamic supply (StorageClass)

The current PV usage method is called static provisioning, which requires K8s operation and maintenance engineers to create a bunch of PVs in advance for developers to use.

       The PV and PVC modes described above require the operation and maintenance personnel to create PVs first, and then developers define PVCs for one-to-one Bonding. However, if thousands of PVC requests are required, thousands of PVs need to be created. The maintenance cost is very high for operation and maintenance personnel. Kubernetes provides a mechanism for automatically creating PVs, called storageclass, which is used to create PV templates.

To create a StorageClass, you need to define the properties of the PV, such as storage type, size, etc.; in addition, you need to create the storage plug-ins for this PV, such as ceph.

With these two pieces of information, Kubernetes can find the corresponding storageClass based on the PVC submitted by the user, and then Kubernetes will call the storage plug-in declared by the storageClass to automatically create the required PV and bind it.

Therefore, K8s began to support PV dynamic supply, using StorageClass objects to achieve. Storage Class | Kubernetes

 PV dynamic supply based on NFS

Deploy NFS to automatically create PV plug-ins

# 授权访问apiserver
kubectl apply -f rbac.yaml
    
# 部署插件，需修改里面NFS服务器地址与共享目录
kubectl apply -f deployment.yaml 
    
# 创建存储类    
kubectl apply -f class.yaml  

Create rbac.yaml

---
kind: ServiceAccount
apiVersion: v1
metadata:
  name: nfs-client-provisioner
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    namespace: default
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: default
roleRef:
  kind: Role
  name: leader-locking-nfs-client-provisioner
  apiGroup: rbac.authorization.k8s.io

 create deployment.yaml

​
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
---
kind: Deployment
apiVersion: apps/v1 
metadata:
  name: nfs-client-provisioner
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: quay.io/external_storage/nfs-client-provisioner:latest
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: fuseim.pri/ifs
            - name: NFS_SERVER
              value: 192.168.2.118  ## NFS服务器 k8s-node1
            - name: NFS_PATH
              value: /ifs/kubernetes
      volumes:
        - name: nfs-client-root
          nfs:
            server: 192.168.2.118 
            path: /ifs/kubernetes    ## 数据共享目录

 Create class.yaml storage class

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-nfs-storage
provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
  archiveOnDelete: "true"

Create a storage class and view

kubectl apply  -f    rbac.yaml deployment.yaml class.yaml 
kubectl get sc  # 查看存储类

Create deployment-sc.yaml web example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: depployment-sc
spec:
  selector:
    matchLabels:
      app: nginx-sc
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx-sc
    spec:
      containers:
      - name: nginx
        image: nginx
        volumeMounts:
        - name: wwwroot
          mountPath: /usr/share/nginx/html
        ports:
        - containerPort: 80
      volumes:
      - name: wwwroot
        persistentVolumeClaim:  
          claimName: my-pvc3   ## 引用PVC

---
apiVersion: v1    
kind: PersistentVolumeClaim
metadata:
  name: my-pvc3
spec:
  storageClassName: "managed-nfs-storage" ## 通知  storageclass 自动创建 PV
  accessModes:      ## 读写权限
  - ReadWriteMany
  resources:
    requests:         ## 请求的资源大小
      storage: 2Gi    ## 资源大小 2G

 Dynamically create pv binding request my-pvc3, nginx-sc pod successfully applied for 2G resources

The storage path of my-pvc3 will be automatically created under the path of the NFS server server/ifs/kubernetes

We can successfully access index.html

delete the deployment-sc pod

kubectl delete  -f  deployment-sc.yaml

Use the Delete (delete) recycling policy, after the PVC is deleted, the dynamically created PV will also be deleted

summary

Dynamically create PV resources, apply for the use of PV resources through PVCs, and bind PVCs to PVs. Finally, Pods use PV resources by referencing PVCs (that is, pods use the underlying storage resources indirectly at the end)

That is, declare the PVC to be used in the pod, then mount the directory of the container to the PV specified by the PVC, and finally mount all the files in the container directory on the PV to the shared directory corresponding to nfs at the same time. middle. Because the mounted directory in the container is mounted on the PV through the PVC reference, and the PV is built on the nfs storage, that is, the final data will be persisted to the corresponding shared directory of the nfs server declared by the PV.

6.3 PV life cycle

6.3.1 ACCESS MODES (access mode)

AccessModes is used to set the access mode of PV, and is used to describe the access rights of user applications to storage resources. The access rights include the following methods:

• ReadWriteOnce (RWO): read and write permissions, but can only be mounted by a single node

• ReadOnlyMany (ROX): read-only permission, can be mounted by multiple nodes

• ReadWriteMany (RWX): read and write permissions, can be mounted by multiple nodes

6.3.2 RECLAIM POLICY (recycling policy)

There are currently three strategies supported by PV:

• Retain: Retain data, requiring administrators to manually clean up data

• Recycle (recycle): clear the data in the PV, the effect is equivalent to executing rm -rf /ifs/kuberneres/*

• Delete: Delete the backend storage connected to the PV at the same time

6.3.3 STATUS (Status)

In the life cycle of a PV, it may be in 4 different stages:

• Available (available): Indicates available status, not yet bound by any PVC

• Bound (bound): Indicates that the PV has been bound by the PVC

• Released: the PVC was deleted, but the resource has not yet been reclaimed by the cluster

• Failed: Indicates that the automatic recycling of the PV failed

6.3.4PV and PVC life cycle

The life cycle of PV and PVC is shown in the figure below:

There is a one-to-one correspondence between PVC and PV, and the interaction between PV and PVC follows the following life cycle:

• Resource supply : The administrator manually creates the underlying storage and PV, and the status is Available

• Resource binding : the user creates a PVC, and kubernetes is responsible for finding and binding the PV according to the declaration of the PVC, and the status is Bound

After the user defines a PVC, the system will select a satisfying condition among the existing PVs according to the PVC's request for storage resources.

       a. Once found, bind the PV to a user-defined PVC, and the user's application can use this PVC

       b. If not found, the PVC will be in the Pending state indefinitely until the system administrator creates a PV that meets its requirements

       Once a PV is bound to a PVC, it will be exclusively occupied by this PVC and cannot be bound to other PVCs.

• Resource usage : users can use pvc in the pod like a volume

        Pod uses the definition of Volume to mount PVC to a certain path in the container for use.

• Resource release : the user deletes pvc to release pv

        When the storage resource is used up, the user can delete the PVC, and the PV bound to the PVC will be marked as "Released", but it cannot be bound to other PVCs immediately. Data written through a previous PVC may also be left on the storage device, and the PV can only be used again after being cleared.

• Resource recycling : kubernetes recycles resources according to the recycling policy set by pv

For PV, the administrator can set recycling policy, which is used to set how to deal with the remaining data after the PVC bound to it releases resources. Only after the storage space of the PV has been reclaimed (in the Available state) can it be bound and used by a new PVC

 Reference: K8S data storage (PV, PVC, PV, and PVC life cycle of advanced storage)_Pathway does not appreciate at sunset, blog-CSDN blog_k8s View pv PV and PVC life cycle K8S data storage (PV, PVC, advanced storage , PV and PVC life cycle)_Pathway Sunset does not reward,'s blog-CSDN blog_k8s View pv