StatefulSet - Storage - Rook Cluster -

Scope & Not-in-scope

• document the setup process of PVC/PV upon Ceph Rook cluster, for learning purpose only
• list out some error met during #1
• K8S related concepts & knowledges is not in this blog scope, you can find it here(https://kubernetes.io/)

steps:

1. Create Ceph rook cluster

git clone https://github.com/rook/rook.git
cd /cluster/examples/kubernetes/ceph
kubectl create -f operator.yaml

# verify the rook-ceph-operator, rook-ceph-agent, and rook-discover pods are in the `Running` state before proceeding
kubectl -n rook-ceph-system get pod

#Create a Rook Cluster
kubectl create -f cluster.yaml

#Use kubectl to list pods in the rook namespace. You should be able to see the following pods once they are all running. The number of osd pods will depend on the number of nodes in the cluster and the number of devices and directories configured
kubectl -n rook-ceph get pod

Ceph-Rook-Cluster.png

2. Provision Storage - Block

Block storage allows you to mount storage to a single pod, and there are three types of storage exposed by Rook, in our case let try to use block storage for demostration purpose.

• Block: Create block storage to be consumed by a pod
• Object: Create an object store that is accessible inside or outside the Kubernetes cluster
• Shared File System: Create a file system to be shared across multiple pods

of course we can use the provided examples, after run the below command, k8s will help to create CephBlockPool - replicapool and StorageClass - rook-ceph-block

kubectl create -f storageclass.yaml

3. Create a Statefulset in which volumeClaimTemplates been predefined and to apply the block storage into pods going to create

to create yaml file - statefulset-pvc.yaml, it's content like below

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: "nginx"
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.15.4
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: web
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: web
    spec:
      storageClassName: rook-ceph-block
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi

use statefulset-pvc.yaml to create pods

kubectl create -f statefulset-pvc.yaml

4. Check pods, Persitent Volume Claim, Persistent Volume, and Stroage bind status

# check pod
kubectl get pod

# check persistent volume claim
kubectl get pvc

# check persitent volume
kubectl get pv

Status Result.png

Q&A

Rook cluster creation

• after exec the 'kubectl apply -f cluster.yaml', got 'Init:CrashLoopBackOff'
  CrashLoopBackOff.png
  [Solution]

1. check the pod status using 'kubectl describe pod rook-ceph-mon-a-788ff56499-87hzx --namespace=rook-ceph'

2. you will find the error message:
   The keyring does not match the existing keyring in /var/lib/rook/mon-a/data/keyring. You may need to delete the contents of dataDirHostPath on the host from a previous deployment.
   RootCause-CrashLoopBachOff.png

3. on all worker node,go to the dataDirHostPath , namely /var/lib/rook， clean this folder with below command, after removal, related pods/container will auto retry and recover

cd /var/lib/rook
rm -rf *

Create-RookCluster-Succ.png

btw, refer to (https://rook.github.io/docs/rook/master/ceph-quickstart.html)