前言
本文介绍基于银河麒麟高级服务器操作系统V10已安装部署的k8s单机集群上部署GlusterFS、Heketi
本文涉及部署脚本主要基于gluster官方项目https://github.com/gluster/gluster-kubernetes在arm64上的迁移适配项目https://github.com/hknarutofk/gluster-kubernetes
前置条件
银河麒麟高级服务器操作系统V10上安装k8s单机集群： https://blog.csdn.net/m0_46573967/article/details/112935319
一、准备一个块独立的硬盘
在长城云上为银河麒麟高级服务器V10所在虚拟机分配一个独立的云盘。其他虚拟化平台类似，物理机则直接插到硬盘接口上。
新增的云盘会被内核自动识别
二、下载gluster-kubernetes脚本
切换到root用户，下载脚本
[yeqiang@192-168-110-185 桌面]$ sudo su
[root@192-168-110-185 桌面]# cd ~
[root@192-168-110-185 ~]# git clone --depth=1 https://github.com/hknarutofk/gluster-kubernetes.git
正克隆到 'gluster-kubernetes'...
remote: Enumerating objects: 157, done.
remote: Counting objects: 100% (157/157), done.
remote: Compressing objects: 100% (132/132), done.
remote: Total 157 (delta 21), reused 95 (delta 14), pack-reused 0
接收对象中: 100% (157/157), 659.85 KiB | 7.00 KiB/s, 完成.
处理 delta 中: 100% (21/21), 完成.
三、部署单点GlusterFS及heketi服务
由于资源有限，我们直接在当前服务器节点上部署一个GlusterFS实例、一个heketi实例
获取节点信息
[root@192-168-110-185 ~]# kubectl get nodes --show-labels
NAME STATUS ROLES AGE VERSION LABELS
192.168.110.185 Ready master 3h19m v1.18.6 beta.kubernetes.io/arch=arm64,beta.kubernetes.io/os=linux,kubernetes.io/arch=arm64,kubernetes.io/hostname=192.168.110.185,kubernetes.io/os=linux,kubernetes.io/role=master节点直接是名与ip地址相同
创建topology.json文件
[root@192-168-110-185 ~]# cd gluster-kubernetes/
[root@192-168-110-185 gluster-kubernetes]# ^C
[root@192-168-110-185 gluster-kubernetes]# cd ..
[root@192-168-110-185 ~]# cd gluster-kubernetes/deploy/
[root@192-168-110-185 deploy]# vim topology.json
{
"clusters": [
{
"nodes": [
{
"node": {
"hostnames": {
"manage": [
"192.168.110.185"
],
"storage": [
"192.168.110.185"
]
},
"zone": 1
},
"devices": [
"/dev/sda"
]
}
]
}
]
}注意：
node.hostnames.manage数组内填写的是节点名称，不是ip地址！只是我们当前安装的节点是以ip命名
node.hostnames.storage数组内填写的是GlusterFS安装目标节点ip地址
devices数组填写的是目前ip节点上为glusterfs准备的磁盘路径（空硬盘）
在目标节点服务器安装glusterfs-fuse
[root@192-168-110-185 ~]# yum install glusterfs-fuse -y
Last metadata expiration check: 0:11:06 ago on 2021年01月22日 星期五 14时43分51秒.
Dependencies resolved.
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
glusterfs aarch64 7.0-4.ky10 ks10-adv-os 3.5 M
Installing dependencies:
python3-gluster aarch64 7.0-4.ky10 ks10-adv-os 15 k
python3-prettytable noarch 0.7.2-18.ky10 ks10-adv-os 33 k
rdma-core aarch64 20.1-6.ky10 ks10-adv-os 494 k
Transaction Summary
================================================================================
Install 4 Packages
Total download size: 4.0 M
Installed size: 23 M
Downloading Packages:
(1/4): python3-gluster-7.0-4.ky10.aarch64.rpm 90 kB/s | 15 kB 00:00
(2/4): python3-prettytable-0.7.2-18.ky10.noarch 116 kB/s | 33 kB 00:00
(3/4): rdma-core-20.1-6.ky10.aarch64.rpm 402 kB/s | 494 kB 00:01
(4/4): glusterfs-7.0-4.ky10.aarch64.rpm 767 kB/s | 3.5 MB 00:04
--------------------------------------------------------------------------------
Total 883 kB/s | 4.0 MB 00:04
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : rdma-core-20.1-6.ky10.aarch64 1/4
Running scriptlet: rdma-core-20.1-6.ky10.aarch64 1/4
Installing : python3-prettytable-0.7.2-18.ky10.noarch 2/4
Installing : python3-gluster-7.0-4.ky10.aarch64 3/4
Running scriptlet: glusterfs-7.0-4.ky10.aarch64 4/4
Installing : glusterfs-7.0-4.ky10.aarch64 4/4
警告：/etc/glusterfs/glusterd.vol 已建立为 /etc/glusterfs/glusterd.vol.rpmnew
警告：/etc/glusterfs/glusterfs-logrotate 已建立为 /etc/glusterfs/glusterfs-logrotate.rpmnew
警告：/etc/glusterfs/gsyncd.conf 已建立为 /etc/glusterfs/gsyncd.conf.rpmnew
Running scriptlet: glusterfs-7.0-4.ky10.aarch64 4/4
Verifying : glusterfs-7.0-4.ky10.aarch64 1/4
Verifying : python3-gluster-7.0-4.ky10.aarch64 2/4
Verifying : python3-prettytable-0.7.2-18.ky10.noarch 3/4
Verifying : rdma-core-20.1-6.ky10.aarch64 4/4
Installed:
glusterfs-7.0-4.ky10.aarch64 python3-gluster-7.0-4.ky10.aarch64
python3-prettytable-0.7.2-18.ky10.noarch rdma-core-20.1-6.ky10.aarch64
Complete!
删除掉原项目中的污点容忍配置
由于原项目基于独立部署节点部署GlusterFS，所有GlusterFS上均设置了污点。本文只采用了一个节点安装所有组建，因此需要删除这部分配置
编辑/root/gluster-kubernetes/deploy/kube-templates/glusterfs-daemonset.yaml
去掉末尾的
tolerations:
- key: glusterfs
operator: Exists
effect: NoSchedule
保存文件
修改后的完整内容如下
---
kind: DaemonSet
apiVersion: apps/v1
metadata:
name: glusterfs
labels:
glusterfs: daemonset
annotations:
description: GlusterFS DaemonSet
tags: glusterfs
spec:
selector:
matchLabels:
glusterfs: pod
glusterfs-node: pod
template:
metadata:
name: glusterfs
labels:
glusterfs: pod
glusterfs-node: pod
spec:
nodeSelector:
storagenode: glusterfs
hostNetwork: true
containers:
- image: registry.cn-hangzhou.aliyuncs.com/hknaruto/glusterfs-gluster_centos-arm64:latest
imagePullPolicy: Always
name: glusterfs
env:
# alternative for /dev volumeMount to enable access to *all* devices
- name: HOST_DEV_DIR
value: "/mnt/host-dev"
# set GLUSTER_BLOCKD_STATUS_PROBE_ENABLE to "1" so the
# readiness/liveness probe validate gluster-blockd as well
- name: GLUSTER_BLOCKD_STATUS_PROBE_ENABLE
value: "1"
- name: GB_GLFS_LRU_COUNT
value: "15"
- name: TCMU_LOGDIR
value: "/var/log/glusterfs/gluster-block"
resources:
requests:
memory: 100Mi
cpu: 100m
volumeMounts:
- name: glusterfs-heketi
mountPath: "/var/lib/heketi"
- name: glusterfs-run
mountPath: "/run"
- name: glusterfs-lvm
mountPath: "/run/lvm"
- name: glusterfs-etc
mountPath: "/etc/glusterfs"
- name: glusterfs-logs
mountPath: "/var/log/glusterfs"
- name: glusterfs-config
mountPath: "/var/lib/glusterd"
- name: glusterfs-host-dev
mountPath: "/mnt/host-dev"
- name: glusterfs-misc
mountPath: "/var/lib/misc/glusterfsd"
- name: glusterfs-block-sys-class
mountPath: "/sys/class"
- name: glusterfs-block-sys-module
mountPath: "/sys/module"
- name: glusterfs-cgroup
mountPath: "/sys/fs/cgroup"
readOnly: true
- name: glusterfs-ssl
mountPath: "/etc/ssl"
readOnly: true
- name: kernel-modules
mountPath: "/lib/modules"
readOnly: true
securityContext:
capabilities: {}
privileged: true
readinessProbe:
timeoutSeconds: 3
initialDelaySeconds: 40
exec:
command:
- "/bin/bash"
- "-c"
- "if command -v /usr/local/bin/status-probe.sh; then /usr/local/bin/status-probe.sh readiness; else systemctl status glusterd.service; fi"
periodSeconds: 25
successThreshold: 1
failureThreshold: 50
livenessProbe:
timeoutSeconds: 3
initialDelaySeconds: 40
exec:
command:
- "/bin/bash"
- "-c"
- "if command -v /usr/local/bin/status-probe.sh; then /usr/local/bin/status-probe.sh liveness; else systemctl status glusterd.service; fi"
periodSeconds: 25
successThreshold: 1
failureThreshold: 50
volumes:
- name: glusterfs-heketi
hostPath:
path: "/var/lib/heketi"
- name: glusterfs-run
- name: glusterfs-lvm
hostPath:
path: "/run/lvm"
- name: glusterfs-etc
hostPath:
path: "/etc/glusterfs"
- name: glusterfs-logs
hostPath:
path: "/var/log/glusterfs"
- name: glusterfs-config
hostPath:
path: "/var/lib/glusterd"
- name: glusterfs-host-dev
hostPath:
path: "/dev"
- name: glusterfs-misc
hostPath:
path: "/var/lib/misc/glusterfsd"
- name: glusterfs-block-sys-class
hostPath:
path: "/sys/class"
- name: glusterfs-block-sys-module
hostPath:
path: "/sys/module"
- name: glusterfs-cgroup
hostPath:
path: "/sys/fs/cgroup"
- name: glusterfs-ssl
hostPath:
path: "/etc/ssl"
- name: kernel-modules
hostPath:
path: "/lib/modules"
执行部署
[root@192-168-110-185 ~]# cd gluster-kubernetes/deploy/
[root@192-168-110-185 deploy]# sh deploy.sh
Using Kubernetes CLI.
Checking status of namespace matching 'default':
default Active 3h34m
Using namespace "default".
Checking for pre-existing resources...
GlusterFS pods ...
Checking status of pods matching '--selector=glusterfs=pod':
Timed out waiting for pods matching '--selector=glusterfs=pod'.
not found.
deploy-heketi pod ...
Checking status of pods matching '--selector=deploy-heketi=pod':
Timed out waiting for pods matching '--selector=deploy-heketi=pod'.
not found.
heketi pod ...
Checking status of pods matching '--selector=heketi=pod':
Timed out waiting for pods matching '--selector=heketi=pod'.
not found.
gluster-s3 pod ...
Checking status of pods matching '--selector=glusterfs=s3-pod':
Timed out waiting for pods matching '--selector=glusterfs=s3-pod'.
not found.
Creating initial resources ... /opt/kube/bin/kubectl -n default create -f /root/gluster-kubernetes/deploy/kube-templates/heketi-service-account.yaml 2>&1
serviceaccount/heketi-service-account created
/opt/kube/bin/kubectl -n default create clusterrolebinding heketi-sa-view --clusterrole=edit --serviceaccount=default:heketi-service-account 2>&1
clusterrolebinding.rbac.authorization.k8s.io/heketi-sa-view created
/opt/kube/bin/kubectl -n default label --overwrite clusterrolebinding heketi-sa-view glusterfs=heketi-sa-view heketi=sa-view
clusterrolebinding.rbac.authorization.k8s.io/heketi-sa-view labeled
OK
Marking '192.168.110.185' as a GlusterFS node.
/opt/kube/bin/kubectl -n default label nodes 192.168.110.185 storagenode=glusterfs --overwrite 2>&1
node/192.168.110.185 not labeled
Deploying GlusterFS pods.
sed -e 's/storagenode\: glusterfs/storagenode\: 'glusterfs'/g' /root/gluster-kubernetes/deploy/kube-templates/glusterfs-daemonset.yaml | /opt/kube/bin/kubectl -n default create -f - 2>&1
daemonset.apps/glusterfs created
Waiting for GlusterFS pods to start ...
Checking status of pods matching '--selector=glusterfs=pod':
glusterfs-ndxgd 1/1 Running 0 91s
OK
/opt/kube/bin/kubectl -n default create secret generic heketi-config-secret --from-file=private_key=/dev/null --from-file=./heketi.json --from-file=topology.json=topology.json
secret/heketi-config-secret created
/opt/kube/bin/kubectl -n default label --overwrite secret heketi-config-secret glusterfs=heketi-config-secret heketi=config-secret
secret/heketi-config-secret labeled
sed -e 's/\${HEKETI_EXECUTOR}/kubernetes/' -e 's#\${HEKETI_FSTAB}#/var/lib/heketi/fstab#' -e 's/\${HEKETI_ADMIN_KEY}/admin/' -e 's/\${HEKETI_USER_KEY}/user/' /root/gluster-kubernetes/deploy/kube-templates/deploy-heketi-deployment.yaml | /opt/kube/bin/kubectl -n default create -f - 2>&1
service/deploy-heketi created
deployment.apps/deploy-heketi created
Waiting for deploy-heketi pod to start ...
Checking status of pods matching '--selector=deploy-heketi=pod':
deploy-heketi-59d9fdff68-kpr87 1/1 Running 0 12s
OK
Determining heketi service URL ... OK
/opt/kube/bin/kubectl -n default exec -i deploy-heketi-59d9fdff68-kpr87 -- heketi-cli -s http://localhost:8080 --user admin --secret 'admin' topology load --json=/etc/heketi/topology.json 2>&1
Creating cluster ... ID: 5c8d62cb9f57df17d7a55cc60d5fe5ca
Allowing file volumes on cluster.
Allowing block volumes on cluster.
Creating node 192.168.110.185 ... ID: a326a80bfc84137329503683869d044e
Adding device /dev/sda ... OK
heketi topology loaded.
/opt/kube/bin/kubectl -n default exec -i deploy-heketi-59d9fdff68-kpr87 -- heketi-cli -s http://localhost:8080 --user admin --secret 'admin' setup-openshift-heketi-storage --help --durability=none >/dev/null 2>&1
/opt/kube/bin/kubectl -n default exec -i deploy-heketi-59d9fdff68-kpr87 -- heketi-cli -s http://localhost:8080 --user admin --secret 'admin' setup-openshift-heketi-storage --listfile=/tmp/heketi-storage.json --durability=none 2>&1
Saving /tmp/heketi-storage.json
/opt/kube/bin/kubectl -n default exec -i deploy-heketi-59d9fdff68-kpr87 -- cat /tmp/heketi-storage.json | sed 's/heketi\/heketi:dev/registry.cn-hangzhou.aliyuncs.com\/hknaruto\/heketi-arm64:v10.2.0/g' | /opt/kube/bin/kubectl -n default create -f - 2>&1
secret/heketi-storage-secret created
endpoints/heketi-storage-endpoints created
service/heketi-storage-endpoints created
job.batch/heketi-storage-copy-job created
Checking status of pods matching '--selector=job-name=heketi-storage-copy-job':
heketi-storage-copy-job-f85d7 0/1 Completed 0 2m24s
/opt/kube/bin/kubectl -n default label --overwrite svc heketi-storage-endpoints glusterfs=heketi-storage-endpoints heketi=storage-endpoints
service/heketi-storage-endpoints labeled
/opt/kube/bin/kubectl -n default delete all,service,jobs,deployment,secret --selector="deploy-heketi" 2>&1
pod "deploy-heketi-59d9fdff68-kpr87" deleted
service "deploy-heketi" deleted
deployment.apps "deploy-heketi" deleted
replicaset.apps "deploy-heketi-59d9fdff68" deleted
job.batch "heketi-storage-copy-job" deleted
secret "heketi-storage-secret" deleted
sed -e 's/\${HEKETI_EXECUTOR}/kubernetes/' -e 's#\${HEKETI_FSTAB}#/var/lib/heketi/fstab#' -e 's/\${HEKETI_ADMIN_KEY}/admin/' -e 's/\${HEKETI_USER_KEY}/user/' /root/gluster-kubernetes/deploy/kube-templates/heketi-deployment.yaml | /opt/kube/bin/kubectl -n default create -f - 2>&1
service/heketi created
deployment.apps/heketi created
Waiting for heketi pod to start ...
Checking status of pods matching '--selector=heketi=pod':
heketi-bc754bf5d-lm2z2 1/1 Running 0 9s
OK
Determining heketi service URL ... OK
heketi is now running and accessible via http://172.20.0.17:8080 . To run
administrative commands you can install 'heketi-cli' and use it as follows:
# heketi-cli -s http://172.20.0.17:8080 --user admin --secret '<ADMIN_KEY>' cluster list
You can find it at https://github.com/heketi/heketi/releases . Alternatively,
use it from within the heketi pod:
# /opt/kube/bin/kubectl -n default exec -i heketi-bc754bf5d-lm2z2 -- heketi-cli -s http://localhost:8080 --user admin --secret '<ADMIN_KEY>' cluster list
For dynamic provisioning, create a StorageClass similar to this:
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: glusterfs-storage
provisioner: kubernetes.io/glusterfs
parameters:
resturl: "http://172.20.0.17:8080"
restuser: "user"
restuserkey: "user"
Deployment complete!
[root@192-168-110-185 deploy]#
创建StorageClass
部署日志最后打印出来的StorageClass只是一个固定的例子，其中的配置需要随当前项目修改
查看heketi服务地址
[root@192-168-110-185 deploy]# kubectl get svc | grep heketi
heketi ClusterIP 10.68.93.101 <none> 8080/TCP 3m
heketi-storage-endpoints ClusterIP 10.68.28.192 <none> 1/TCP 5m30s
得到heketi服务地址：http://10.68.93.101:8080
查看管理员口令
管理员口令是安装是指令参数输入的，位于deploy.sh脚本中
[root@192-168-110-185 deploy]# cat deploy.sh
#!/bin/bash
bash ./gk-deploy -v -g --admin-key=admin --user-key=user --single-node -l /tmp/gk-deploy.log -y
得到admin-key：admin
修改storageclass-singlenode.yaml
[root@192-168-110-185 deploy]# vim storageclass-singlenode.yamlapiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: glusterfs-storage
annotations:
# 由于系统存在其他默认存储，此处设置为false（默认也是false）
storageclass.kubernetes.io/is-default-class: "false"
provisioner: kubernetes.io/glusterfs
parameters:
resturl: "http://10.68.93.101:8080"
restuser: "admin"
restuserkey: "admin"
volumetype: none说明：
1. 此处设置的是非默认存储，可以根据实际情况修改。如果设置为默认存储，则k8s所有pvc将到此节点消耗存储空间
2. 单节点glusterfs部署必须指定volumetype:none，否则默认是需要三个节点才能成功绑定pvc
部署StorageClass
[root@192-168-110-185 deploy]# kubectl apply -f storageclass-singlenode.yaml
storageclass.storage.k8s.io/glusterfs-storage created
[root@192-168-110-185 deploy]# kubectl get storageclasses
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
glusterfs-storage kubernetes.io/glusterfs Delete Immediate false 7s总结
k8s集成heketi、glusterfs后，可以自动根据pvc创建、管理对应的pv，无需人工提前准备。
本文涉及的arm64镜像制作涉及项目
https://github.com/hknarutofk/gluster-containers/tree/master/CentOS