k8s数据存储的方式很多：临时存储（emptyDir），半持久化存储（hostpath），持久化存储（文件、块、对象存存储等）。公有云各厂商几乎都支持云存储服务集成k8s托管，k8s系统本身通过两个概念PersistentVolume（PV）和 PersistentVolumeClaim(PVC)实现存储持久化的生命周期管理（申请、保留、删除、废弃等），pod根据需求向集群系统申请数据卷并使用，如果pod一旦释放，应用消失，数据卷相应分离解绑，数据会保留下来，在使用中pod以及应用对存储的后端基础架构并无感知。 这种设计很好地把应用和数据做了解藕。 至于存储资源的底层是选择ceph、portwx、弹性云存储、云对象存储等等，这就取决于业务具体需求和厂商的支持了。
闲话少说，切入今天的主题，既然有很多后端选择，用户在试用一款k8s集群产品，存储持久化以及应用层面的读写能力是必测一关，这里我们以ibm cloud kubernetes service为例，看看IBM云上的三种常见外挂存储的测试方法以及性能比较。
大致分为三个步骤：
- 创建K8S集群并通过kubectl连接使用 (部署在日本东京)
- 确认和安装IBM云存储k8s供应插件(CFS、CBS、COS部署在东京)
- dbench(fio)测试上述三种存储卷
1. k8s集群创建、连接和使用
登陆cloud.ibm.com, 在商品目录里搜索关键字“kubernetes”，进入服务创建向导，本次测试将用标准资费（免费集群的worker节点只有一个，无法满足测试要求），可以注意到的是ibm k8s的在线版本很新，选择最近的稳定版本1.17.6
接下来选择经典基础架构（覆盖全），vpc基础架构在六大区（东京，悉尼，法兰克福，伦敦，华盛顿，达拉斯）也已完全就绪，考虑vpc高可用区集群的worker成本，在单区开一套经典做测试性价比会高一点。
最后我们设置worker pool，选择三个2c*4g虚拟机作为工作节点，若考虑安全可以打开节点磁盘加密（ibm这点真的没话说），定义完集群名字，点创建
大约五六分钟，集群初始化完成，进入“Normal”状态
点击创建好的集群，进入“访问”，参考步骤在自己的笔记本上安装ibmcloud cli （k8s相关工具比如kubectl，docker等如果当前系统没有，会一并安装）
danws-MacBook-Pro:k8s-playground danw$ ibmcloud login --sso -a cloud.ibm.com -r jp-tok -g danw-group
danws-MacBook-Pro:k8s-playground danw$ ibmcloud ks cluster config --cluster bro62jit0fjj42dqsh7g
danws-MacBook-Pro:k8s-playground danw$ export KUBECONFIG=/Users/danw/.bluemix/plugins/container-service/clusters/bro62jit0fjj42dqsh7g/kube-config-tok05-demo-k8s-tok05.yml
也可将集群配置文件添加到～/.bash_profile中,
测试连接成功！
danws-MacBook-Pro:k8s-playground danw$ k get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
10.193.33.31 Ready <none> 26m v1.17.6+IKS 10.193.33.31 165.192.96.2 Ubuntu 18.04.4 LTS 4.15.0-101-generic containerd://1.3.4
10.193.33.33 Ready <none> 26m v1.17.6+IKS 10.193.33.33 165.192.96.12 Ubuntu 18.04.4 LTS 4.15.0-101-generic containerd://1.3.4
10.193.33.44 Ready <none> 27m v1.17.6+IKS 10.193.33.44 165.192.96.7 Ubuntu 18.04.4 LTS 4.15.0-101-generic containerd://1.3.4
2. 安装IBM Cloud云存储插件
a. IBM Cloud File Storage
默认file stroage class（存储插件）在集群创建过程中预装好了，可直接使用
danws-MacBook-Pro:k8s-playground danw$ k get storageclasses | grep file
default ibm.io/ibmc-file Delete Immediate false 37m
ibmc-file-bronze ibm.io/ibmc-file Delete Immediate false 37m
ibmc-file-bronze-gid ibm.io/ibmc-file Delete Immediate false 37m
ibmc-file-custom ibm.io/ibmc-file Delete Immediate false 37m
ibmc-file-gold (default) ibm.io/ibmc-file Delete Immediate false 37m
ibmc-file-gold-gid ibm.io/ibmc-file Delete Immediate false 37m
ibmc-file-retain-bronze ibm.io/ibmc-file Retain Immediate false 37m
ibmc-file-retain-custom ibm.io/ibmc-file Retain Immediate false 37m
ibmc-file-retain-gold ibm.io/ibmc-file Retain Immediate false 37m
ibmc-file-retain-silver ibm.io/ibmc-file Retain Immediate false 37m
ibmc-file-silver ibm.io/ibmc-file Delete Immediate false 37m
ibmc-file-silver-gid ibm.io/ibmc-file Delete Immediate false 37m
这里测试我们选择“silver”（ 按小时计费，4 IOPS/GB）
danws-MacBook-Pro:k8s-playground danw$ kubectl describe storageclass ibmc-file-bronze
Name: ibmc-file-bronze
IsDefaultClass: No
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"labels":{"kubernetes.io/cluster-service":"true"},"name":"ibmc-file-bronze"},"parameters":{"billingType":"hourly","classVersion":"2","iopsPerGB":"2","sizeRange":"[20-12000]Gi","type":"Endurance"},"provisioner":"ibm.io/ibmc-file","reclaimPolicy":"Delete"}
Provisioner: ibm.io/ibmc-file
Parameters: billingType=hourly,classVersion=2,iopsPerGB=2,sizeRange=[20-12000]Gi,type=Endurance
AllowVolumeExpansion: <unset>
MountOptions: <none>
ReclaimPolicy: Delete
VolumeBindingMode: Immediate
Events: <none>
b. IBM Cloud Block Storage
集群默认没有预装“块存储“插件，这里需要先安装，官方在线文档可参考具体步骤。
https://cloud.ibm.com/docs/containers?topic=containers-block_storage#install_block
首先确认集群工作节点状态
danws-MacBook-Pro:k8s-playground danw$ ibmcloud ks worker ls --cluster demo-k8s-tok05
OK
ID Public IP Private IP Flavor State Status Zone Version
kube-bro62jit0fjj42dqsh7g-demok8stok0-default-000001b3 165.192.96.7 10.193.33.44 u3c.2x4.encrypted normal Ready tok05 1.17.6_1527
kube-bro62jit0fjj42dqsh7g-demok8stok0-default-0000022e 165.192.96.2 10.193.33.31 u3c.2x4.encrypted normal Ready tok05 1.17.6_1527
kube-bro62jit0fjj42dqsh7g-demok8stok0-default-000003fe 165.192.96.12 10.193.33.33 u3c.2x4.encrypted normal Ready tok05 1.17.6_1527
块存储和后面的COS插件安装需要依赖helm chart，这里我们先安装helm v3,这里helm安装过程已省略，请参考以下连接：
https://github.com/helm/helm/releases/tag/v3.2.4
danws-MacBook-Pro:k8s-playground danw$ helm version
version.BuildInfo{Version:"v3.2.4", GitCommit:"0ad800ef43d3b826f31a5ad8dfbb4fe05d143688", GitTreeState:"clean", GoVersion:"go1.13.12"}
danws-MacBook-Pro:k8s-playground danw$ helm repo add iks-charts https://icr.io/helm/iks-charts
"iks-charts" has been added to your repositories
danws-MacBook-Pro:k8s-playground danw$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "istio.io" chart repository
...Successfully got an update from the "iks-charts" chart repository
Update Complete. ⎈ Happy Helming!⎈
danws-MacBook-Pro:k8s-playground danw$ helm search repo iks-charts
NAME CHART VERSION APP VERSION DESCRIPTION
iks-charts/ibm-block-storage-attacher 1.1.4 A Helm chart for installing ibmcloud block stor...
iks-charts/ibm-iks-cluster-autoscaler 1.1.7 A Helm chart for installing the IBM Cloud clust...
iks-charts/ibm-object-storage-plugin 1.0.7 1.0.7 A Helm chart for installing ibmcloud object sto...
iks-charts/ibm-worker-recovery 1.10.71 IBM Autorecovery system allows automatic recove...
iks-charts/ibmcloud-alb-metrics-exporter 1.0.25 25.0 A Helm chart to deploy IBM Cloud ALB metrics ex...
iks-charts/ibmcloud-backup-restore 1.0.1 A Helm chart for taking backup of pvc data and ...
iks-charts/ibmcloud-block-storage-plugin 1.7.0 A Helm chart for installing ibmcloud block stor...
iks-charts/ibmcloud-data-shield 1.18.731 1.0 Protect data in use with IBM Cloud Data Shield
iks-charts/ibmcloud-image-enforcement 0.2.10 A Helm chart to install IBM Container Image Sec...
iks-charts/strongswan 2.6.3 20.05.15 A strongSwan IPSec VPN service to securely conn...
danws-MacBook-Pro:k8s-playground danw$ helm install 1.7.0 iks-charts/ibmcloud-block-storage-plugin -n default
danws-MacBook-Pro:k8s-playground danw$ kubectl get pods -n kube-system | grep ibmcloud-block-storage
ibmcloud-block-storage-driver-792wk 1/1 Running 0 62s
ibmcloud-block-storage-driver-m4phr 1/1 Running 0 62s
ibmcloud-block-storage-driver-n9pxv 1/1 Running 0 62s
ibmcloud-block-storage-plugin-665f4b5fb5-4fn9t 1/1 Running 0 61s
danws-MacBook-Pro:k8s-playground danw$ kubectl get storageclasses | grep block
ibmc-block-bronze ibm.io/ibmc-block Delete Immediate true 2m50s
ibmc-block-custom ibm.io/ibmc-block Delete Immediate true 2m50s
ibmc-block-gold ibm.io/ibmc-block Delete Immediate true 2m50s
ibmc-block-retain-bronze ibm.io/ibmc-block Retain Immediate true 2m50s
ibmc-block-retain-custom ibm.io/ibmc-block Retain Immediate true 2m50s
ibmc-block-retain-gold ibm.io/ibmc-block Retain Immediate true 2m50s
ibmc-block-retain-silver ibm.io/ibmc-block Retain Immediate true 2m50s
ibmc-block-silver ibm.io/ibmc-block Delete Immediate true 2m50s
注：下篇我们再探讨COS pvc
3. 使用dbench对上述两种类型的pvc进行fio测试
github一个开源的容器fio测试image，可以拿来直接使用，clone repo先
danws-MacBook-Pro:dbench danw$ git clone https://github.com/leeliu/dbench.git
⚠️： 这里有个小雷，dbench image目前移动到这个repo “ndrpnt/dbench:1.0.0”
fio执行的脚本文件内容可参考下面链接,
https://github.com/leeliu/dbench/blob/master/docker-entrypoint.sh
虽然脚本已经打包到镜像中，但用户可以通过pod环境变量对部分参数进行传参调整比如快速测试，测试文件大小，数据偏移等。
编辑dbench.yaml，因为集群三个worker节点（2c4g）计算资源有限，担心三种存储同时执行fio测试，会影响到彼此的I/O效率，所以我们可以更改yaml分别挂载不同存储类型的pvc分开测试。
danws-MacBook-Pro:dbench danw$ ls -lhrt *-dbench*
-rw-r--r-- 1 danw staff 977B Jun 24 00:27 cfs-dbench.yaml
-rw-r--r-- 1 danw staff 977B Jun 24 00:27 cbs-dbench.yaml
-rw-r--r-- 1 danw staff 977B Jun 24 00:27 cos-dbench.yaml
Example: cfs-dbench.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: cfs-dbench
spec:
template:
spec:
containers:
- name: cfs-dbench
image: ndrpnt/dbench:1.0.0
imagePullPolicy: Always
env:
- name: DBENCH_MOUNTPOINT
value: /data
- name: DBENCH_QUICK
value: "no"
- name: FIO_SIZE
value: 10G
- name: FIO_OFFSET_INCREMENT
value: 256M
- name: FIO_DIRECT
value: "1"
volumeMounts:
- name: cfs-vol
mountPath: /data
restartPolicy: Never
volumes:
- name: cfs-vol
persistentVolumeClaim:
claimName: cfs-pvc
backoffLimit: 4
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: cfs-pvc
labels:
billingType: "hourly"
region: jp-tok
zone: tok05
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 500Gi
storageClassName: ibmc-file-silver
danws-MacBook-Pro:dbench danw$ k get po
NAME READY STATUS RESTARTS AGE
cfs-dbench-m6ksp 1/1 Running 0 7m20s
danws-MacBook-Pro:dbench danw$ k top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
10.193.33.31 1223m 63% 714Mi 25%
10.193.33.33 117m 6% 1204Mi 42%
10.193.33.44 153m 7% 1102Mi 38%
danws-MacBook-Pro:dbench danw$ k logs -f job/cfs-dbench
Working dir: /data
Testing Read IOPS...
fio: posix_fallocate fails: Not supported
read_iops: (g=0): rw=randread, bs=4096B-4096B,4096B-4096B,4096B-4096B, ioengine=libaio, iodepth=64
fio-2.17-45-g06cb
Starting 1 process
read_iops: Laying out IO file(s) (1 file(s) / 10240MiB)
...
文件存储测试结果：
接下来看看块存储，
Example: cbs-dbench.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: cbs-dbench
spec:
template:
spec:
containers:
- name: cbs-dbench
image: ndrpnt/dbench:1.0.0
imagePullPolicy: Always
env:
- name: DBENCH_MOUNTPOINT
value: /data
- name: DBENCH_QUICK
value: "no"
- name: FIO_SIZE
value: 10G
- name: FIO_OFFSET_INCREMENT
value: 256M
- name: FIO_DIRECT
value: "1"
volumeMounts:
- name: cbs-vol
mountPath: /data
restartPolicy: Never
volumes:
- name: cbs-vol
persistentVolumeClaim:
claimName: cbs-pvc
backoffLimit: 4
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: cbs-pvc
labels:
billingType: "hourly"
region: jp-tok
zone: tok05
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 500Gi
storageClassName: ibmc-block-silver
danws-MacBook-Pro:dbench danw$ k get po
NAME READY STATUS RESTARTS AGE
cbs-dbench-dv59g 1/1 Running 0 102s
cfs-dbench-m6ksp 0/1 Completed 0 20m
danws-MacBook-Pro:dbench danw$ k get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
cbs-pvc Bound pvc-ce517510-d529-4f66-a5b9-a0d2d9a58502 500Gi RWO ibmc-block-silver 11m
cfs-pvc Bound pvc-55ad15de-2deb-4bbd-a805-d45e14831ed1 500Gi RWX ibmc-file-silver 30m
danws-MacBook-Pro:dbench danw$ k top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
10.193.33.31 1173m 61% 718Mi 25%
10.193.33.33 98m 5% 1206Mi 42%
10.193.33.44 160m 8% 1104Mi 38%
块存储测试结果：
测试完毕，清空退掉资源
danws-MacBook-Pro:dbench danw$ k delete -f cbs-dbench.yaml
job.batch "cbs-dbench" deleted
persistentvolumeclaim "cbs-pvc" deleted
danws-MacBook-Pro:dbench danw$ k delete -f cfs-dbench.yaml
job.batch "cfs-dbench" deleted
persistentvolumeclaim "cfs-pvc" deleted
总结： 500G * 4 IOPS tier 的文件和块存储，最后测试的IOPS（read+write）都符合承诺的2000，带宽分别是128MiB/S，说明k8s容器虚拟化的I/O消耗并没有想象中那么大，值得一提的是IBM Cloud的文件和块存储在集群中供应速度很快，而且相应的storage class，包括动态PV和k8s volume yaml定义支持都不错。 当然不同的业务属性、不同结构的数据，数据冷热度、访问要求决定了持久化存储的选择，如何针对自身业务做好k8s存储规划，可以看看下面这个链接。
https://cloud.ibm.com/docs/containers?topic=containers-storage_planning
下一篇我们接着探讨COS的挂载以及fio测试。