Environment Description:
| CPU name | Operating system version | ip | docker version | omelets version | Configuration | Remark |
|---|---|---|---|---|---|---|
| master | Centos 06/07/1810 | 172.27.9.131 | Docker 18.09.6 | V1.14.2 | 2C2G | master host |
| node01 | Centos 06/07/1810 | 172.27.9.135 | Docker 18.09.6 | V1.14.2 | 2C2G | node node |
| node02 | Centos 06/07/1810 | 172.27.9.136 | Docker 18.09.6 | V1.14.2 | 2C2G | node node |
| centos7 | Centos 07/03/1611 | 172.27.9.181 | × | × | 1C1G | nfs server |
k8s cluster deployment see: Centos7.6 deployment k8s (v1.14.2) cluster
k8s learning materials See: basic concepts, kubectl command and information sharing
A, Volume
1. Concept
Kubernetes volume is a part of the pod, so as in the specification as the container is defined in the pod. They are not independent Kubernetes objects, can not be created or deleted separately. All containers pod can be used in the volume, but it must first be mounted on each of the containers need to access it in. In each container, the volume can be mounted anywhere in its file system.
2. Why Volume
The life cycle of the file on the disk container is short, which makes there will be some problems when running important applications in the container. First, when the container crashes, kubelet will restart it, but the container file will be lost - the container with a clean state (mirroring the initial state) restart. Secondly, when the plurality of containers simultaneously run Pod usually need to share files between the containers. Volume abstract Kubernetes of a good solution to these problems.
3. Volume Type
Currently, Kubernetes Volume supports the following types:
This paper will emptyDir, hostPath, shared storage NFS, PV and PVC were tested practice.
Two, emptyDir
1. emptyDir concept
emptyDir Volume is the most basic type for easy empty directory to store temporary data. If the Pod set emptyDir type Volume, Pod was assigned to the Node time, creates emptyDir, as long as the Pod run on the Node, emptyDir there will be (container hung up does not result in loss of data emptyDir), but if the Pod is deleted from the Node (Pod is deleted, or Pod migrate), emptyDir will be deleted and permanently lost.
It will be used to achieve the same pod emptyDir volumes between two container file sharing
2. Create pod emptyDir-fortune
[root@master ~]# more emptyDir-pod.yaml
apiVersion: v1
kind: Pod
metadata:
labels:
app: prod #pod标签
name: emptydir-fortune
spec:
containers:
- image: loong576/fortune
name: html-generator
volumeMounts: #名为html的卷挂载至容器的/var/htdocs目录
- name: html
mountPath: /var/htdocs
- image: nginx:alpine
name: web-server
volumeMounts: #挂载相同的卷至容器/usr/share/nginx/html目录且设置为只读
- name: html
mountPath: /usr/share/nginx/html
readOnly: true
ports:
- containerPort: 80
protocol: TCP
volumes:
- name: html #卷名为html的emptyDir卷同时挂载至以上两个容器
emptyDir: {}
[root@master ~]# kubectl apply -f emptyDir-pod.yaml
pod/emptydir-fortune created
[root@master ~]# kubectl get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
emptydir-fortune 2/2 Running 0 9s 10.244.2.140 node02 <none> <none>
Creating pod emptydir-fortune, the pod has two vessels, while the volume mount emptyDir, container html-generator random write content to the volume, by accessing the web-server to verify that the vessel sharing files.
2.1 loong576 / fortune mirroring
root@master ~]# more Dockerfile
[root@master ~]# more fortune/Dockerfile
FROM ubuntu:latest
RUN apt-get update ; apt-get -y install fortune
ADD fortuneloop.sh /bin/fortuneloop.sh
E*TRYPOINT /bin/fortuneloop.shThe mirror base image is ubuntu, will perform fortuneloop.sh script mirrored boot
fortuneloop.sh script:
[root@master ~]# more fortuneloop.sh
#!/bin/bash
trap "exit" SIGINT
mkdir /var/htdocs
while :
do
echo $(date) Writing fortune to /var/htdocs/index.html
/usr/games/fortune > /var/htdocs/index.html
sleep 10
doneThe script is mainly output every 10 seconds a random phrase to the index.html file.
3. Access nginx
3.1 Creating service
[root@master ~]# more service-fortune.yaml
apiVersion: v1
kind: Service
metadata:
name: my-service #service名
spec:
type: NodePort
selector:
app: prod #pod标签,由此定位到pod emptydir-fortune
ports:
- protocol: TCP
nodePort: 30002 #节点监听端口，暴露静态端口30002对外提供服务
port: 8881 #ClusterIP监听的端口
targetPort: 80 #容器端口
sessionAffinity: ClientIP #是否支持Session,同一个客户端的访问请求都转发到同一个后端Pod
[root@master ~]# kubectl apply -f service-fortune.yaml
service/my-service created
[root@master ~]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3d17h
my-service NodePort 10.102.191.57 <none> 8881:30002/TCP 9s3.2 nginx access
[root@master ~]# curl 10.102.191.57:8881
Writing is easy; all you do is sit staring at the blank sheet of paper until
drops of blood form on your forehead.
-- Gene Fowler
[root@master ~]# curl 172.27.9.135:30002
Don't Worry, Be Happy.
-- Meher Baba
in conclusion:
-
Container nginx successfully read the contents of the container fortune written to storage, emptyDir volume file sharing between the container can be achieved.
- Life cycle life cycle emptyDir volumes associated with the pod, so when deleting pod, contents of the volume will be lost
Three, hostPath
1. Concept
hostPath able to mount the file system on Pod Node to go inside. If you need to use Pod file on the Node, you can use hostPath. Running on a node and use the same path in the same pod hostPath its volume can be seen in the same file.
2. Create pod hostpath-nginx
2.1 Create a mount directory
Create mount directory on node node, the following operations are performed on the master node and each
[root@master ~]# mkdir /data && cd /data && echo `hostname` > index.html2.2 Creating pod
[root@master ~]# more hostPath-pod.yaml
apiVersion: v1
kind: Pod
metadata:
labels:
app: prod
name: hostpath-nginx
spec:
containers:
- image: nginx
name: nginx
volumeMounts:
- mountPath: /usr/share/nginx/html #容器挂载点
name: nginx-volume #挂载卷nginx-volume
volumes:
- name: nginx-volume #卷名
hostPath:
path: /data #准备挂载的node上的文件系统
[root@master ~]# kubectl apply -f hostPath-pod.yaml
pod/hostpath-nginx created
[root@master ~]# kubectl get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
emptydir-fortune 2/2 Running 0 40m 10.244.2.140 node02 <none> <none>
hostpath-nginx 1/1 Running 0 16s 10.244.1.140 node01 <none> <none>3. Access pod hostpath-nginx
[root@master ~]# curl 10.244.1.140
node01
in conclusion:
-
pod运行在node01上，访问的内容为'node01'，为挂载的文件系统/data下index.html内容，容器成功读取到挂载的节点文件系统里的内容。
-
仅当需要在节点上读取或写入系统文件时才使用hostPath , 切勿使用它们来持久化跨pod的数据。
- hostPath可以实现持久存储，但是在node节点故障时，也会导致数据的丢失。
四、NFS共享存储
1. 概念
NFS是Network File System的缩写，即网络文件系统。Kubernetes中通过简单地配置就可以挂载NFS到Pod中，而NFS中的数据是可以永久保存的，同时NFS支持同时写操作。
emptyDir可以提供不同容器间的文件共享，但不能存储；hostPath可以为不同容器提供文件的共享并可以存储，但受制于节点限制，不能跨节点共享；这时需要网络存储 (NAS)，即既可以方便存储容器又可以从任何集群节点访问，本文以NFS为例做测试。
2. nfs搭建及配置
nfs搭建详见：Centos7下NFS服务器搭建及客户端连接配置
完成nfs服务器搭建和客户端nfs软件安装安装后，可在master和各node节点检查nfs服务是否正常
[root@master ~]# showmount -e 172.27.9.181
Export list for 172.27.9.181:
/backup 172.27.9.0/24master和node01、node02节点都执行showmount命令，用于验证nfs服务是否正常，/backup为nfs服务器对外提供的共享目录。
本文测试的NFS内容：
3. 新建pod mongodb-nfs
[root@master ~]# more mongodb-pod-nfs.yaml
apiVersion: v1
kind: Pod
metadata:
name: mongodb-nfs
spec:
containers:
- image: mongo
name: mongodb
volumeMounts:
- name: nfs-data #挂载的卷名，与上面的mongodb-data保持一致
mountPath: /data/db #MongoDB数据存放的路径
ports:
- containerPort: 27017
protocol: TCP
volumes:
- name: nfs-data #卷名
nfs:
server: 172.27.9.181 #nfs服务器ip
path: /backup #nfs服务器对外提供的共享目录
[root@master ~]# kubectl apply -f mongodb-pod-nfs.yaml
pod/mongodb-nfs created
[root@master ~]# kubectl get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
mongodb-nfs 1/1 Running 0 23s 10.244.2.142 node02 <none> <none>注意此时pod的ip为10.244.2.142
4. nfs共享存储测试
4.1 向MongoDB写入数据
[root@master ~]# kubectl exec -it mongodb-nfs mongo
> use loong
switched to db loong
> db.foo.insert({name:'loong576'})
WriteResult({ "nInserted" : 1 })切换至db loong，插入JSON文档(name:'loong576')
4.2 查看写入的数据
> db.foo.find()
{ "_id" : ObjectId("5d6e17b018651a21e0063641"), "name" : "loong576" }
4.3 删除pod并重建
[root@master ~]# kubectl delete pod mongodb-nfs
pod "mongodb-nfs" deleted
[root@master ~]# kubectl apply -f mongodb-pod-nfs.yaml
pod/mongodb-nfs created
[root@master ~]# kubectl get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
mongodb-nfs 1/1 Running 0 22s 10.244.2.143 node02 <none> <none>删除pod mongodb-nfs并重建，此时podip变为10.244.2.143，再次访问MongoDB验证之前写入的文档是否还存在。
4.4 新pod读取共享存储数据
[root@master ~]# kubectl exec -it mongodb-nfs mongo
> use loong
switched to db loong
> db.foo.find()
{ "_id" : ObjectId("5d6e17b018651a21e0063641"), "name" : "loong576" }
即使pod被删除重建仍然能访问共享数据。
结论：
- NFS共享存储可持久化数据
- NFS共享存储可跨节点提供数据共享
五、PV and PVC
1. 概念
PersistentVolume (持久卷， 简称 PV)和Persistent VolumeClaim(持久卷声明，简称 PVC)使得K8s集群具备了存储的逻辑抽象能力，使得在配置Pod的逻辑里可以忽略对实际后台存储技术的配置，而把这项配置的工作交给PV的配置者，即集群的管理者。存储的PV和PVC的这种关系，跟计算的Node和Pod的关系是非常类似的；PV和Node是资源的提供者，根据集群的基础设施变化而变化，由K8s集群管理员配置；而PVC和Pod是资源的使用者，根据业务服务的需求变化而变化，由K8s集群的使用者即服务的管理员来配置。
当集群用户需要在其pod中使用持久化存储时，他们首先创建PVC清单，指定所需要的最低容量要求和访问模式，然后用户将待久卷声明清单提交给Kubernetes API服务器，Kubernetes将找到可匹配的PV并将其绑定到PVC。PVC可以当作pod中的一个卷来使用，其他用户不能使用相同的PV，除非先通过删除PVC绑定来释放。
2. 创建PV
2.1 nfs配置
nfs服务器共享目录配置：
[root@centos7 ~]# exportfs
/backup/v1 172.27.9.0/24
/backup/v2 172.27.9.0/24
/backup/v3 172.27.9.0/24master和各node节点检查nfs配置：
[root@master ~]# showmount -e 172.27.9.181
Export list for 172.27.9.181:
/backup/v3 172.27.9.0/24
/backup/v2 172.27.9.0/24
/backup/v1 172.27.9.0/242.2 PV创建
[root@master ~]# more pv-nfs.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv001
spec:
capacity:
storage: 2Gi #指定PV容量为2G
volumeMode: Filesystem #卷模式，默认为Filesystem，也可设置为'Block'表示支持原始块设备
accessModes:
- ReadWriteOnce #访问模式，该卷可以被单个节点以读/写模式挂载
persistentVolumeReclaimPolicy: Retain #回收策略，Retain（保留），表示手动回收
storageClassName: nfs #类名，PV可以具有一个类，一个特定类别的PV只能绑定到请求该类别的PVC
nfs: #指定NFS共享目录和IP信息
path: /backup/v1
server: 172.27.9.181
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv002
spec:
capacity:
storage: 2Gi #指定PV容量为2G
volumeMode: Filesystem #卷模式，默认为Filesystem，也可设置为'Block'表示支持原始块设备
accessModes:
- ReadOnlyMany #访问模式，该卷可以被多个节点以只读模式挂载
persistentVolumeReclaimPolicy: Retain #回收策略，Retain（保留），表示手动回收
storageClassName: nfs #类名，PV可以具有一个类，一个特定类别的PV只能绑定到请求该类别的PVC
nfs: #指定NFS共享目录和IP信息
path: /backup/v2
server: 172.27.9.181
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv003
spec:
capacity:
storage: 1Gi #指定PV容量为1G
volumeMode: Filesystem #卷模式，默认为Filesystem，也可设置为'Block'表示支持原始块设备
accessModes:
- ReadWriteOnce #访问模式，该卷可以被单个节点以读/写模式挂载
persistentVolumeReclaimPolicy: Retain #回收策略，Retain（保留），表示手动回收
storageClassName: nfs #类名，PV可以具有一个类，一个特定类别的PV只能绑定到请求该类别的PVC
nfs: #指定NFS共享目录和IP信息
path: /backup/v3
server: 172.27.9.181
[root@master ~]# kubectl apply -f pv-nfs.yaml
persistentvolume/pv001 created
persistentvolume/pv002 created
persistentvolume/pv003 created
[root@master ~]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLA*S REASON AGE
pv001 2Gi RWO Retain Available nfs 26s
pv002 2Gi ROX Retain Available nfs 26s
pv003 1Gi RWO Retain Available nfs 26s创建pv001、pv002、pv003，分别对应nfs的共享目录/backup/v1、/backup/v2、/backup/v2。
卷可以处于以下的某种状态：
- Available（可用），一块空闲资源还没有被任何声明绑定
- Bound（已绑定），卷已经被声明绑定
- Released（已释放），声明被删除，但是资源还未被集群重新声明
- Failed（失败），该卷的自动回收失败
PV的访问模式有三种：
- 第一种，ReadWriteOnce：是最基本的方式，可读可写，但只支持被单个Pod挂载。
- 第二种，ReadOnlyMany：可以以只读的方式被多个Pod挂载。
- Third, ReadWriteMany: This memory can be read in a shared manner plurality Pod. Not every store support all three ways, like sharing, support is still relatively small at present, more commonly used is NFS.
PV does not belong to any namespace, it is like a cluster node-level resources, different from the pod and PVC.
3. Create PVC
3.1 PVC creation
[root@master ~]# more pvc-nfs.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: mypvc #声明的名称，当做pod的卷使用时会用到
spec:
accessModes:
- ReadWriteOnce #访问卷模式，筛选PV条件之一
volumeMode: Filesystem #卷模式，与PV保持一致，指示将卷作为文件系统或块设备使用
resources: #声明可以请求特定数量的资源,筛选PV条件之一
requests:
storage: 2Gi
storageClassName: nfs #请求特定的类，与PV保持一致，否则无法完成绑定
[root@master ~]# kubectl apply -f pvc-nfs.yaml
persistentvolumeclaim/mypvc created
[root@master ~]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mypvc Bound pv001 2Gi RWO nfs 22sCreate PVC mypvc, access volume mode ReadWriteOnce, size 2G; WO, ROX, RWX, RWO represents the number of nodes may simultaneously use the work roll and not the number of the pod.
3.2 View selected PV
PVC Filters:
| PV | accessModes | storage |
|---|---|---|
| pv001 | √ | √ |
| pv002 | × | √ |
| pv003 | √ | × |
PV View:
[root@master ~]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLA*S REASON AGE
pv001 2Gi RWO Retain Bound default/mypvc nfs 12m
pv002 2Gi ROX Retain Available nfs 12m
pv003 1Gi RWO Retain Available nfs 12m
pv001 was selected, in line with the definition of PVC, pv002 access patterns do not match, pv003 size mismatch.
4. pod used in PVC
[root@master ~]# more mongodb-pod-pvc.yaml
apiVersion: v1
kind: Pod
metadata:
name: mongodb-pvc
spec:
containers:
- image: mongo
name: mongodb
volumeMounts:
- name: pvc-data
mountPath: /data/db
ports:
- containerPort: 27017
protocol: TCP
volumes:
- name: pvc-data
persistentVolumeClaim:
claimName: mypvc #与pvc中声明的name保持一致
[root@master ~]# kubectl apply -f mongodb-pod-pvc.yaml
pod/mongodb-pvc created
[root@master ~]# kubectl get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
mongodb-pvc 1/1 Running 0 16s 10.244.2.144 node02 <none> <none>
Creating pod mongodb-pvc, use PVC mypvc, test nfs shared storage test and the four -4, and not repeat them.
In this paper all scripts and configuration files have been uploaded: K8S practice (VII): data storage volume and persistence (Volumes and Persistent Storage)