Use rook to build a Ceph cluster

Host:

MacBook Pro（Apple M2 Max）

VMware Fusion Player version 13.0.2 

VM software and hardware:

ubuntu 22.04.2

4-core CPU, 5G memory, 40G hard drive

*It is important to allocate hardware resources to each machine. It can appropriately exceed the amount of resources of the host machine, otherwise it will cause various problems.

IP and role (163 is master, others are nodes):

192.168.19.163 k8smaster

192.168.19.161 k8sworker3

192.168.19.162 k8sworker2

192.168.19.164 k8sworker1

k8s suite:

Kubernetes v1.28.0

Docker version 24.0.5

containerd.io 1.6.22

Rook v1.12.2

1. Ensure that the time of each machine is synchronized

Configuration reference: Ubuntu turns on NTP time synchronization_ubuntu synchronization time_modern third-rate magician's blog-CSDN blog

2. Ensure that each worker node has a raw disk

After the VM is shut down, just add "new hard drive" in the settings.

After adding the new hard disk and then starting the virtual machine, execute the following lsblk -f to view the new hard disk name. The author here is nvme0n2 . Note that the value of the FSTYPE column is empty.

Make sure that all three machines, k8sworker1, k8sworker2, and k8sworker3 except the master, have new hard drives.

 lsblk -f

3. Download Rook v1.12.2 source code

git clone --single-branch --branch v1.12.2 ​​​​​​​https://github.com/rook/rook.git

  The directory I downloaded here is /home/zhangzhaokun/rook, and the next steps are based on this directory.

4. Modify Rook’s CSI mirror address

Enter the directory rook/deploy/examples and modify the CSI mirror address of Rook. The original address is registry.k8s.io, which cannot be accessed in China due to the wall. It needs to be replaced with registry.lank8s.cn.

As for what registry.lank8s.cn is, please refer to the document  What is Lank8s.cn

cat operator.yaml |grep IMAGE

The result is as follows:

​​​​​​​

 The revised content is referenced as follows:

5. Modify operator.yam configuration

cat operator.yaml |grep DISCOVERY

ROOK_ENABLE_DISCOVERY_DAEMON: "false"

Need to be modified to: true

6. Pull rook’s image in advance

1) Find the image in the operator.yaml file

  1.1) Find the image file in the "image:" format:

cat operator.yaml |grep "image:"

    The result is as follows:

 1.2) Find the image file in "_IMAGE" format:

 cat operator.yaml |grep "_IMAGE"

     The result is as follows:

2) Find the image in the cluster.yaml file

cat cluster.yaml |grep image

 The result is as follows:

3) Pull the image in advance through containerd

It depends on your character here. Sometimes the speed may be super slow and may even fail, and sometimes it may be very fast. When the speed is super slow, switching the network (such as switching from mobile to telecom WI-FI) may cause huge problems. Fast.

sudo ctr images pull docker.io/rook/ceph:v1.12.2
sudo ctr images pull quay.io/csiaddons/k8s-sidecar:v0.7.0
sudo ctr images pull quay.io/ceph/ceph:v17.2.6
sudo ctr images pull quay.io/cephcsi/cephcsi:v3.9.0
sudo ctr images pull registry.lank8s.cn/sig-storage/csi-node-driver-registrar:v2.8.0
sudo ctr images pull registry.lank8s.cn/sig-storage/csi-resizer:v1.8.0
sudo ctr images pull registry.lank8s.cn/sig-storage/csi-provisioner:v3.5.0
sudo ctr images pull registry.lank8s.cn/sig-storage/csi-snapshotter:v6.2.2
sudo ctr images pull registry.lank8s.cn/sig-storage/csi-attacher:v4.3.0

7. Deploy Rook

Note that this is executed only after entering the directory rook/deploy/examples.

kubectl create -f crds.yaml -f common.yaml -f operator.yaml

Note that all pods under the rook-ceph namespace must become 1/1 Running before it is considered OK and you can proceed to the next step.

kubectl get pods -n rook-ceph

NAME                                            READY    STATUS     RESTARTS     AGE

rook-ceph-operator-6ff688d999-wkhcb 1/1    Running 2 (93m ago) 11h

rook-discover-86vc8                              1/1    Running 1 (3h32m ago) 11h

rook-discover-b56f8                               1/1    Running 1 (3h32m ago) 11h

rook-discover-djfqw                                1/1    Running 2 (93m ago) 11h

8. Modify cluster.yaml

Modification No. 1:

useAllNodes: false

useAllDevices: false

Modification 2:

Specify the node and disk names used by Ceph, and the OSD nodes run on these nodes.

nodes:
  - name: "k8sworker1"
    devices:
      - name: "nvme0n2"
  - name: "k8sworker2"
    devices:
      - name: "nvme0n2"
  - name: "k8sworker3"
    devices:
      - name: "nvme0n2"

9. Deploy Ceph cluster

kubectl create -f cluster.yaml

After the creation is completed, you can check the status of the Pod. Note that osd-0/1/2, mon-a/b/c these Pods come out later, and when there is a problem, their status is often abnormal.

View ceph cluster status:

kubectl get cephcluster -n rook-ceph

10. Install the ceph client tool toolbox

kubectl apply -f toolbox.yaml

The results are as follows, find rook-ceph-tools

Enter the command line:

kubectl exec -it -n rook-ceph rook-ceph-tools-84f9854d5f-cr7kt -- bash

Check the status of ceph on the command line:

ceph -s
ceph osd status

The output is as follows:

11. Deploy Dashboard

Deploy dashboard:

kubectl create -f dashboard-external-https.yaml

View service:

kubectl get svc -n rook-ceph

The result is as follows:

You can access the dashboard by accessing any worker on the host:

https://192.168.19.161:31052

The default username is admin, and the password can be obtained through the following code:

kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}"|base64 --decode && echo

The result is as follows:

a's?6XlvZk'75dc7*X[~