这份文档各组件系统
- ceph:15.2.5
- k8s: 1.19.2
- os: centos7 7.8.2003
在所有k8s node节点上安装ceph-common
不需要节点有/etc/ceph/目录下的文件
yum install -y ceph-common
创建pool
ceph osd pool create kube rbd pool init kube
创建用户
ceph auth get-or-create \
client.kube mon 'allow r' \
osd 'allow class-read object_prefix rbd_children, allow rwx pool=kube' \
-o /etc/ceph/ceph.client.kube.keyring
创建namespace rbd用于部署csi
kubectl create namespace rbd
创建csi configmap
ceph-csi当前仅支持旧版V1协议。
- 查看mon和clusterID
ceph mon dump
- 创建configmap
cat <<EOF > csi-config-map.yaml
---
apiVersion: v1
kind: ConfigMap
data:
config.json: |-
[
{
"clusterID": "01c1e960-fbce-11ea-a90e-52540079f755",
"monitors": [
"192.168.254.103:6789",
"192.168.254.103:6789",
"192.168.254.103:6789"
]
}
]
metadata:
name: ceph-csi-config
namespace: rbd
EOF
创建ksm configmap,新版本需要这个
参看https://github.com/ceph/ceph-csi/issues/834
cat <<EOF>>kms-config.yaml
---
apiVersion: v1
kind: ConfigMap
data:
config.json: |-
{
}
metadata:
name: ceph-csi-encryption-kms-config
EOF
kubectl -n rbd create -f kms-config.yaml
创建csi访问ceph的secret
cat <<EOF > csi-rbd-secret.yaml
---
apiVersion: v1
kind: Secret
metadata:
name: csi-rbd-secret
namespace: rbd
stringData:
userID: kube
userKey: AQDFhmlfVphsGhAAh8f0Ck3gREjilODu4Sz/Zw==
EOF
配置csi插件
创建rbac
- 下载rbac文件
wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-provisioner-rbac.yaml
wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-nodeplugin-rbac.yaml
- 将rbac中的namespace: default替换为namespace: rbd
sed -i -E 's/(^\s+namespace:)(.*)/\1 rbd/g' csi-provisioner-rbac.yaml
sed -i -E 's/(^\s+namespace:)(.*)/\1 rbd/g' csi-nodeplugin-rbac.yaml
- apply rbac
kubectl -n rbd apply -f csi-provisioner-rbac.yaml
kubectl -n rbd apply -f csi-nodeplugin-rbac.yaml
创建ceph-csi Provisioner和Node插件
- 下载插件文件
wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin-provisioner.yaml
wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin.yaml
- apply plugin
kubectl -n rbd apply -f csi-rbdplugin-provisioner.yaml
kubectl -n rbd apply -f csi-rbdplugin.yaml
创建storageClass
注意增加fsType、imageFormat、imageFeatures
imageFeatures是一定要增加的,否则centos7上不能正常挂载rbd
cat <<EOF > csi-rbd-sc.yaml
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
parameters:
clusterID: 01c1e960-fbce-11ea-a90e-52540079f755
pool: kube
fsType: xfs
imageFormat: "2"
imageFeatures: "layering"
csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
csi.storage.k8s.io/provisioner-secret-namespace: rbd
csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
csi.storage.k8s.io/node-stage-secret-namespace: rbd
reclaimPolicy: Delete
mountOptions:
- discard
EOF
kubectl apply -f storageClass.yaml
以block模式挂载
这种模式是直接将rbd设备以block模式挂载容器中,使用时需要进入容器格式,这种例子中将rbd设备挂载到/dev/xvda
这种模式下,进入容器后是没有权限在/dev/xvda上创建文件系统、执行mount操作
创建pvc
cat <<EOF > raw-block-pvc.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: raw-block-pvc
spec:
accessModes:
- ReadWriteOnce
volumeMode: Block
resources:
requests:
storage: 1Gi
storageClassName: csi-rbd-sc
EOF
kubectl -n rbd apply -f raw-block-pvc.yaml
创建pod
cat <<EOF > raw-block-pod.yaml
---
apiVersion: v1
kind: Pod
metadata:
name: pod-with-raw-block-volume
spec:
containers:
- name: fc-container
image: centos:7
command: ["/bin/sh", "-c"]
args: ["tail -f /dev/null"]
volumeDevices:
- name: data
devicePath: /dev/xvda
volumes:
- name: data
persistentVolumeClaim:
claimName: raw-block-pvc
EOF
kubectl -n rbd apply -f raw-block-pod.yaml
以文件系统方式使用
这种模式会先将rbd设备格式化后再挂载给容器
创建pvc
cat <<EOF > pvc.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: rbd-pvc
spec:
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
resources:
requests:
storage: 1Gi
storageClassName: csi-rbd-sc
EOF
kubectl -n rbd apply -f pvc.yaml
创建Pod
cat <<EOF > pod.yaml
---
apiVersion: v1
kind: Pod
metadata:
name: csi-rbd-demo-pod
spec:
containers:
- name: web-server
image: nginx
volumeMounts:
- name: mypvc
mountPath: /var/lib/www/html
volumes:
- name: mypvc
persistentVolumeClaim:
claimName: rbd-pvc
readOnly: false
EOF
kubectl -n rbd apply -f pod.yaml