The relationship and practice of kubernetes storage class and PV and PVC

StorageClass & PV & PVC relationship diagram

• Volumes is the most basic storage abstraction, which supports multiple types, including local storage, NFS, FC and numerous cloud storages. We can also write our own storage plug-ins to support specific storage systems. Volume can be used directly by Pod or by PV. There is a static binding relationship between ordinary Volume and Pod. While defining the Pod, the storage type is defined by the volume attribute, and the mount point in the container is defined by volumeMount.

• PersistentVolume. Different from ordinary Volume, PV is a resource object in Kubernetes. Creating a PV is equivalent to creating a storage resource object. The use of this resource must be requested through PVC.

• PersistentVolumeClaim. PVC is a user's request for storage resource PV. Kubernetes dynamically searches for PV resources in the system and binds them according to the conditions specified in the PVC. Currently, PVC and PV can be matched by three methods: StorageClassName, matchLabels, or matchExpressions.

• StorageClass. Storage type, currently kubernetes supports many storages, such as ceph, nfs, glusterfs, etc. . .

Next, this article uses the ceph cluster created in the previous article "Teach you how to deploy a ceph cluster using rpm" to provide storage for kubernetes.

Create storage class

1. Get the admin key

grep key /etc/ceph/ceph.client.admin.keyring |awk '{printf "%s", $NF }'|base64

QVFCZ2ZZOWJ1dGdBQ0JBQXN5dGdLZ1BFOGlsblIzWjJqNVVKMUE9PQ==

2. Write ceph-secret-admin.yaml

apiVersion: v1
kind: Secret
metadata:
  name: ceph-secret-admin
type: "kubernetes.io/rbd"
data:
  key: QVFCZ2ZZOWJ1dGdBQ0JBQXN5dGdLZ1BFOGlsblIzWjJqNVVKMUE9PQ==

3. Create a secret

kubectl  apply -f ceph-secret-admin.yaml
kubectl  get secret
NAME                                 TYPE                                  DATA      AGE
ceph-secret-admin                    kubernetes.io/rbd                     1         6m

4. Modify rbd-storage-class.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: rbd
provisioner: kubernetes.io/rbd
parameters:
    monitors: 192.168.100.100:6789,192.168.100.101:6789,192.168.100.102:6789
    adminId: admin
    adminSecretName: ceph-secret-admin
    adminSecretNamespace: default
    pool: k8s
    userId: admin
    userSecretName: ceph-secret-admin
    userSecretNamespace: default
    fsType: ext4
    imageFormat: "2"
    imageFeatures: "layering"

5. Create a storage class

kubectl apply -f rbd-storage-class.yaml

[root@qd01-stop-cloud001 rbd]# kubectl  get sc
NAME      PROVISIONER         AGE
rbd         kubernetes.io/rbd   4m

Create PVC & PV

6. Create pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: claim
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: rbd
  resources:
    requests:
      storage: 1Gi

7. Check pvc, the display status is Bound to indicate success

kubectl apply -f pvc.yaml
kubectl  get pvc
NAME      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
claim    Bound     pvc-130c2445-b4a5-11e8-9d27-782bcb3bb379   1Gi        RWO            slow           13m

Test verification

8. Create pod.yaml

apiVersion: v1
kind: ReplicationController
metadata:
  name: server
spec:
  replicas: 1
  selector:
    role: server
  template:
    metadata:
      labels:
        role: server
    spec:
      containers:
      - name: server
        image: nginx
        volumeMounts:
          - mountPath: /var/lib/www/html
            name: mypvc
      volumes:
        - name: mypvc
          persistentVolumeClaim:
            claimName: claim

9. View mounted storage

/dev/rbd0       1G  9.1M  0.98G   1% /var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/k8s-image-kubernetes-dynamic-pvc-198f56b3-b4a5-11e8-97eb-782bcb3bb379

Error handling

如果出错failed to create rbd image: executable file not found in $PATH
Refer to https://blog.csdn.net/aixiaoyang168/article/details/79120095
You can use the following items to create storage classes

Use external-storage to create a storage class

$ git clone https://github.com/kubernetes-incubator/external-storage.git
$ tree external-storage/ceph/rbd/deploy/
├── README.md
├── non-rbac
│   └── deployment.yaml
└── rbac
    ├── clusterrole.yaml
    ├── clusterrolebinding.yaml
    ├── deployment.yaml
    └── serviceaccount.yaml

Install without RBAC roles:
cd $GOPATH/src/github.com/kubernetes-incubator/external-storage/ceph/rbd/deploy
kubectl apply -f ./non-rbac

Install with RBAC roles:
cd $GOPATH/src/github.com/kubernetes-incubator/external-storage/ceph/rbd/deploy
NAMESPACE=default     # change this if you want to deploy it in another namespace
sed -r -i "s/namespace: [^ ]+/namespace: $NAMESPACE/g" ./rbac/clusterrolebinding.yaml ./rbac/rolebinding.yaml
kubectl -n $NAMESPACE apply -f ./rbac

If the secret and provisioner are not in the same namespace, the secret permission is not enough.
Solution:
add secrets permissions to the following files
external-storage/ceph/rbd/deploy/rbac/clusterrole.yaml

  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "create", "delete"]