Cloud native in-depth analysis of Ceph deployment and functional testing in K8S

I. Introduction

• Ceph is very convenient to use as shared storage in k8s. Ceph is an old-fashioned distributed storage system, which is very mature and powerful. It supports three modes (fast storage, file system storage, and object storage), so how to use it in k8s What about using ceph?
• For the deployment of the k8s environment, please refer to: Cloud Native In-depth Analysis of the High Availability Environment Deployment of K8S 1.24 .
• For the environment deployment and actual operation of Ceph, please refer to: Cloud native in-depth analysis of the environment deployment and actual operation of the distributed storage system Ceph .

2. Introduction to Ceph Rook

• Rook is an open source cloud-native storage orchestration tool that provides a platform, framework, and support for various storage solutions for native integration with cloud-native environments. Rook turns storage software into a self-managing, self-scaling, and self-healing storage service by automating deployment, startup, configuration, provisioning, scaling, upgrades, migrations, disaster recovery, monitoring, and resource management. Rook's underlying layer uses the capabilities provided by cloud-native container management, scheduling, and orchestration platforms to provide these capabilities.
• Rook uses extended functions to deeply integrate it into the cloud-native environment, and provides a seamless experience for scheduling, lifecycle management, resource management, security, monitoring, etc. Rook currently supports Ceph, NFS, Minio Object Store and CockroachDB .

• Rook official website ;
• Rook Git address .

3. Deploy Ceph in k8s through Rook

① Download the deployment package

• The command is as follows:

git clone --single-branch --branch v1.10.8 https://github.com/rook/rook.git

• The images used for deployment are as follows:

• Since the source of the image is abroad, it cannot be downloaded in China. Here you need to modify some images or download tags in advance. The operation is as follows:

cd rook/deploy/examples/

#(registry.aliyuncs.com/google_containers/<image>:<tag>)，后四个镜像我FQ下
docker pull registry.aliyuncs.com/google_containers/csi-node-driver-registrar:v2.5.1

docker tag registry.aliyuncs.com/google_containers/csi-node-driver-registrar:v2.5.1 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1

docker pull registry.aliyuncs.com/google_containers/csi-snapshotter:v6.1.0
docker tag registry.aliyuncs.com/google_containers/csi-snapshotter:v6.1.0 registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0

docker pull registry.aliyuncs.com/google_containers/csi-attacher:v4.0.0
docker tag registry.aliyuncs.com/google_containers/csi-attacher:v4.0.0 registry.k8s.io/sig-storage/csi-attacher:v4.0.0

docker pull registry.aliyuncs.com/google_containers/csi-resizer:v1.6.0
docker tag registry.aliyuncs.com/google_containers/csi-resizer:v1.6.0 registry.k8s.io/sig-storage/csi-resizer:v1.6.0

docker pull registry.aliyuncs.com/google_containers/csi-resizer:v1.6.0
docker tag registry.aliyuncs.com/google_containers/csi-resizer:v1.6.0 registry.k8s.io/sig-storage/csi-resizer:v1.6.0

docker pull registry.aliyuncs.com/google_containers/csi-provisioner:v3.3.0
docker tag registry.aliyuncs.com/google_containers/csi-provisioner:v3.3.0 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0

② Deploy Rook Operator

cd rook/deploy/examples
kubectl create -f crds.yaml -f common.yaml -f operator.yaml
# 检查
kubectl -n rook-ceph get pod

• It can also be deployed through helm:

helm repo add rook-release https://charts.rook.io/release
helm install --create-namespace --namespace rook-ceph rook-ceph rook-release/rook-ceph -f values.yaml

③ Create a Rook Ceph cluster

• Now that the Rook Operator is in the Running state, you can create a Ceph cluster next. In order to make the cluster unaffected after restarting, please ensure that the set dataDirHostPath attribute value is a valid host path:

cd rook/deploy/examples
kubectl apply -f cluster.yaml

④ Deploy the Rook Ceph tool

cd rook/deploy/examples
kubectl create -f toolbox.yaml

⑤ Deploy Ceph Dashboard

cd rook/deploy/examples
kubectl apply -f dashboard-external-https.yaml

# 获取 dashboard admin密码
kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 -d

• View Ceph cluster status through Ceph Dashboard:

# 查看对外端口
kubectl get svc -n rook-ceph

⑥ check

kubectl get pods,svc -n rook-ceph

⑦ View the ceph cluster status through the ceph-tool tool pod

kubectl exec -it `kubectl get pods -n rook-ceph|grep rook-ceph-tools|awk '{
    
    print $1}'` -n rook-ceph -- bash

ceph -s

4. Test verification

① Block storage (RBD) test

• Create StorageClass:

cd rook/deploy/examples
# 创建一个名为replicapool的rbd pool
kubectl apply -f csi/rbd/storageclass.yaml

• Deploy WordPress:

kubectl apply -f mysql.yaml
kubectl apply -f wordpress.yaml

② File system (CephFS) test

• Create StorageClass:

kubectl apply -f csi/cephfs/storageclass.yaml

• Deploy the application:

kubectl apply -f filesystem.yaml

③ Object storage (RGW) test

• Create an object store:

kubectl create -f object.yaml

# 验证rgw pod正常运行
kubectl -n rook-ceph get pod -l app=rook-ceph-rgw

• Create an object storage user:

kubectl create -f object-user.yaml

• Get accesskey secretkey:

# 获取AccessKey
kubectl -n rook-ceph get secret rook-ceph-object-user-my-store-my-user -o yaml | grep AccessKey | awk '{
    
    print $2}' | base64 --decode

# 获取 SecretKey
kubectl -n rook-ceph get secret rook-ceph-object-user-my-store-my-user -o yaml | grep SecretKey | awk '{
    
    print $2}' | base64 --decode

• Deploy rgw nodeport:

kubectl apply -f rgw-external.yaml

kubectl -n rook-ceph get service rook-ceph-rgw-my-store rook-ceph-rgw-my-store-external

• Use Ceph object storage through the api interface:

#首先，需要安装 python-boto 包，用于测试连接 S3
yum install python-boto -y

# 然后，编写 python 测试脚本
# cat s3.py
#!/usr/bin/python

import boto
import boto.s3.connection
access_key = 'C7492VVSL8O11NZBK3GT'
secret_key = 'lo8IIwMfmow4fjkSOMbjebmgjzTRBQSO7w83SvBd'
conn = boto.connect_s3(
    aws_access_key_id = access_key,
    aws_secret_access_key = secret_key,
    host = '192.168.182.110', port=30369,
    is_secure=False,
    calling_format = boto.s3.connection.OrdinaryCallingFormat(),
)
bucket = conn.create_bucket('my-first-s3-bucket')
for bucket in conn.get_all_buckets():
        print "{name}\t{created}".format(
                name = bucket.name,
                created = bucket.creation_date,
)