This article has been included in the column " Learn k8s from scratch "
Previous article: Click
under persistent storage
k8s persistent storage: hostPath
hostPath Volume refers to the directory or file on the host that the Pod mounts. hostPath Volume enables the container to use the host's file system for storage, hostpath (host path): a node-level storage volume, when the pod is deleted, the storage volume still exists and will not be deleted, so as long as the same pod is deleted Scheduled to the same node, after the pod is deleted and rescheduled to this node, the corresponding data still exists.
#查看 hostPath 存储卷的用法
[root@k8smaster cjh]# kubectl explain pods.spec.volumes.hostPath
KIND: Pod
VERSION: v1
RESOURCE: hostPath <Object>
DESCRIPTION:
HostPath represents a pre-existing file or directory on the host machine
that is directly exposed to the container. This is generally used for
system agents or other privileged things that are allowed to see the host
machine. Most containers will NOT need this. More info:
https://kubernetes.io/docs/concepts/storage/volumes#hostpath
Represents a host path mapped into a pod. Host path volumes do not support
ownership management or SELinux relabeling.
FIELDS:
path <string> -required-
type <string>
node1和2下载tomcat
#创建一个 pod,挂载 hostPath 存储卷
[root@k8smaster cjh]# vim hostpath.yaml
apiVersion: v1
kind: Pod
metadata:
name: test-hostpath
spec:
containers:
- image: nginx
imagePullPolicy: IfNotPresent
name: test-nginx
volumeMounts:
- mountPath: /test-nginx
name: test-volume
- image: tomcat
imagePullPolicy: IfNotPresent
name: test-tomcat
volumeMounts:
- mountPath: /test-tomcat
name: test-volume
volumes:
- name: test-volume
hostPath:
path: /data1 #宿主机的目录 根下的data1
type: DirectoryOrCreate # DirectoryOrCreate 表示本地有/data1 目录,就用本地的,本地没有就会在 pod 调度到的节点自动创建一个
#更新资源清单文件
[root@k8smaster cjh]# kubectl apply -f hostpath.yaml
pod/test-hostpath created
#查看 pod 调度到了哪个物理节点
[root@k8smaster cjh]# kubectl get pods -o wide | grep hostpath
test-hostpath 2/2 Running 0 31s 10.244.2.8 k8snode <none> <none>
#由上面可以知道 pod 调度到了 node1 上,登录到 node1 机器,查看是否在这台机器创建了存储目录
[root@k8snode ~]# cd /data1/
[root@k8snode data1]# ll
total 0
#上面可以看到已经创建了存储目录/data1,这个/data1 会作为 pod 的持久化存储目录
#在 node1 上的/data1 下创建一个目录
[root@k8snode data1]# mkdir paopao
#测试存储卷是否可以正常使用,登录到 nginx 容器
[root@k8smaster cjh]# kubectl exec -it test-hostpath -c test-nginx -- /bin/bash
root@test-hostpath:/# cd /test-nginx/
#/test-nginx/目录存在,说明已经把宿主机目录挂载到了容器里
root@test-hostpath:/test-nginx# ls
paopao
#测试存储卷是否可以正常使用,登录到 tomcat 容器
[root@k8smaster cjh]# kubectl exec -it test-hostpath -c test-tomcat -- /bin/bash
bash-4.4# cd /test-tomcat/
#/test-tomcat/目录存在,说明已经把宿主机目录挂载到了容器里
bash-4.4# ls
paopao
#通过上面测试可以看到,同一个 pod 里的 test-nginx 和 test-tomcat 这两个容器是共享存储卷的。
hostpath 存储卷缺点:
单节点
pod 删除之后重新创建必须调度到同一个 node 节点,数据才不会丢失
可以用分布式存储:
nfs,cephfs,glusterfs
k8s persistent storage: nfs
The hostPath storage mentioned in the previous section has a single point of failure. When the pod is mounted on the hostPath, the data will not be lost unless it is scheduled to the same node. That can use nfs as persistent storage.
1、搭建 nfs 服务
以 k8s 的控制节点作为 NFS 服务端
[root@k8smaster cjh]# yum install nfs-utils -y
#在宿主机创建 NFS 需要的共享目录
[root@k8smaster cjh]# mkdir /data/volumes -pv pv是层级创建以及信息
mkdir: created directory ‘/data’
mkdir: created directory ‘/data/volumes’
#配置 nfs 共享服务器上的/data/volumes 目录
[root@k8smaster cjh]# systemctl start nfs
[root@k8smaster cjh]# vim /etc/exports
/data/volumes 192.168.11.0/24(rw,no_root_squash)
#no_root_squash: 用户具有根目录的完全管理访问权限
#使 NFS 配置生效
[root@k8smaster cjh]# exportfs -arv
exporting 192.168.11.0/24:/data/volumes
[root@k8smaster cjh]# service nfs start
Redirecting to /bin/systemctl start nfs.service
#设置成开机自启动
[root@k8smaster cjh]# systemctl enable nfs
#查看 nfs 是否启动成功
[root@k8smaster cjh]# systemctl status nfs
Active: active (exited) since Tue 2022-07-12 23:48:29 PDT; 1min 33s ago
看到 nfs 是 active,说明 nfs 正常启动了
1 2节点也要下载nfs
yum install nfs-utils -y
systemctl enable nfs
在 node1 上手动挂载试试:
[root@k8snode data1]# mkdir /test
[root@k8snode data1]# mount 192.168.11.139:/data/volumes /test/
[root@k8snode data1]# df -h
192.168.11.139:/data/volumes 32G 6.4G 25G 21% /test
#nfs 可以被正常挂载
#手动卸载:
[root@k8snode data1]# umount /test
#创建 Pod,挂载 NFS 共享出来的目录
Pod 挂载 nfs 的官方地址:
https://kubernetes.io/zh/docs/concepts/storage/volumes/
[root@k8smaster cjh]# vim nfs.yaml
apiVersion: v1
kind: Pod
metadata:
name: test-nfs-volume
spec:
containers:
- name: test-nfs
image: nginx
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
protocol: TCP
volumeMounts:
- name: nfs-volumes
mountPath: /usr/share/nginx/html
volumes:
- name: nfs-volumes
nfs:
path: /data/volumes
server: 192.168.11.139
注:path: /data/volumes #nfs 的共享目录
server:192.168.40.180 是 master1 机器的 ip,这个是安装 nfs 服务的地址
#更新资源清单文件
[root@k8smaster cjh]# kubectl apply -f nfs.yaml
pod/test-nfs-volume created
#查看 pod 是否创建成功
[root@k8smaster cjh]# kubectl get pods -o wide | grep nfs
test-nfs-volume 1/1 Running 0 10s 10.244.1.10 k8snode2 <none> <none>
#登录到 nfs 服务器,在共享目录创建一个 index.html
[root@k8smaster ~]# cd /data/volumes/
[root@k8smaster volumes]# ls
[root@k8smaster volumes]# pwd
/data/volumes
[root@k8smaster volumes]# vim index.html
Hello, Everyone
My name is paopao
#请求 pod,看结果
[root@k8smaster volumes]# curl 10.244.1.10
Hello, Everyone
My name is paopao
#通过上面可以看到,在共享目录创建的 index.html 已经被 pod 挂载了
#登录到 pod 验证下
[root@k8smaster volumes]# kubectl exec -it test-nfs-volume -- /bin/bash
root@test-nfs-volume:/# cat /usr/share/nginx/html/index.html
\Hello, Everyone
My name is paopao
#上面说明挂载 nfs 存储卷成功了,nfs 支持多个客户端挂载,可以创建多个 pod,挂载同一个 nfs服务器共享出来的目录;但是 nfs 如果宕机了,数据也就丢失了,所以需要使用分布式存储,常见的分布式存储有 glusterfs 和 cephfs
k8s persistent storage: PVC
Reference official website: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes
What is k8s PV?
A PersistentVolume (PV) is a piece of storage in a cluster that is configured by an administrator or dynamically using storage classes. It is a resource in the cluster, just like a pod is a k8s cluster resource. PVs are capacity plugins, like Volumes, whose lifecycle is independent of any single pod using the PV.
What is k8s PVC?
PersistentVolumeClaim (PVC) is a persistent storage volume, we can define this type of storage volume when we create a pod. It is similar to a pod. Pods consume node resources, and PVCs consume PV resources. Pods can request certain levels of resources (CPU and memory). The pvc can also request a specific size and access mode when requesting the pv (for example, read-write once or read-only many times).
How k8s PVC and PV work
PVs are resources in the cluster. PVCs are requests for these resources. The interaction between PV and PVC follows the following lifecycle:
(1) Supply method of pv
PV can be configured in two ways: static or dynamic.
Static: Cluster administrators create many PVs. They contain details of the actual storage available to cluster users. They exist in the Kubernetes API and can be used.
Dynamic: When none of the static PVs created by the administrator match the user's PersistentVolumeClaim, the cluster may try to dynamically provision volumes specifically for PVCs. This configuration is based on StorageClasses, the PVC must request a storage class, which must be created and configured by the administrator for dynamic configuration. (Based on storage class, must be created in advance)
(2) Binding
The user creates the pvc and specifies the required resources and access mode. The pvc remains unbound until an available pv is found
(3) use
a) You need to find a storage server and divide it into multiple storage spaces;
b) k8s administrators can define these storage spaces as multiple PVs;
c) Before using a pvc-type storage volume in a pod, you need to create a pvc first, and find the appropriate pv by defining the size of the pv to be used and the corresponding access mode;
d) After the pvc is created, it can be used as a storage volume. We can use this pvc storage volume when defining a pod
e) There is a one-to-one correspondence between pvc and pv. If pv is bound by pvc, it cannot be used by other pvcs;
f) When we create pvc, we should ensure that it can be bound to the underlying pv. If there is no suitable pv, then the pvc will be in the pending state.
(4) Recycling strategy
When we create a pod, if we use pvc as a storage volume, it will be bound to pv. When the pod is deleted, the binding between pvc and pv will be released. After the release, what should be done with the data in the pv volume bound to pvc? Currently, volumes can be retained, reclaimed or deleted:
Retain
Recycle (deprecated, 1.15 may be deprecated)
Delete
Retain When the pvc is deleted, the pv still exists and is in the released state, but it cannot be used by other pvc bindings, and the data in it still exists. When we use it next time, the data still exists. This is the default recycling strategy
Delete removes the PV from Kubernetes when the pvc is deleted, and also deletes the storage asset from the related external facility
Create a pod and use pvc as a persistent storage volume
1、创建 nfs 共享目录
#在宿主机创建 NFS 需要的共享目录
[root@k8smaster ~]# mkdir /data/volume_test/v{1,2,3,4,5,6,7,8,9,10} -p
#配置 nfs 共享宿主机上的/data/volume_test/v1..v10 目录
[root@k8smaster ~]# vim /etc/exports
/data/volumes 192.168.11.0/24(rw,no_root_squash)
/data/volume_test/v1 192.168.11.139/24(rw,no_root_squash)
/data/volume_test/v2 192.168.11.139/24(rw,no_root_squash)
/data/volume_test/v3 192.168.11.139/24(rw,no_root_squash)
/data/volume_test/v4 192.168.11.139/24(rw,no_root_squash)
/data/volume_test/v5 192.168.11.139/24(rw,no_root_squash)
/data/volume_test/v6 192.168.11.139/24(rw,no_root_squash)
/data/volume_test/v7 192.168.11.139/24(rw,no_root_squash)
/data/volume_test/v8 192.168.11.139/24(rw,no_root_squash)
/data/volume_test/v9 192.168.11.139/24(rw,no_root_squash)
/data/volume_test/v10 192.168.11.139/24(rw,no_root_squash)
#重新加载配置,使配置成效
[root@k8smaster ~]# exportfs -arv
exporting 192.168.11.139/24:/data/volume_test/v10
exporting 192.168.11.139/24:/data/volume_test/v9
exporting 192.168.11.139/24:/data/volume_test/v8
exporting 192.168.11.139/24:/data/volume_test/v7
exporting 192.168.11.139/24:/data/volume_test/v6
exporting 192.168.11.139/24:/data/volume_test/v5
exporting 192.168.11.139/24:/data/volume_test/v4
exporting 192.168.11.139/24:/data/volume_test/v3
exporting 192.168.11.139/24:/data/volume_test/v2
exporting 192.168.11.139/24:/data/volume_test/v1
exporting 192.168.11.0/24:/data/volumes
2、如何编写 pv 的资源清单文件
#查看定义 pv 需要的字段
[root@k8smaster ~]# kubectl explain pv
KIND: PersistentVolume
VERSION: v1
DESCRIPTION:
PersistentVolume (PV) is a storage resource provisioned by an
administrator. It is analogous to a node. More info:
https://kubernetes.io/docs/concepts/storage/persistent-volumes
FIELDS:
apiVersion <string>s
kind <string>
metadata <Object>
spec <Object>
#查看定义 nfs 类型的 pv 需要的字段
[root@k8smaster ~]# kubectl explain pv.spec.nfs
KIND: PersistentVolume
VERSION: v1
RESOURCE: nfs <Object>
DESCRIPTION:
NFS represents an NFS mount on the host. Provisioned by an admin. More
info: https://kubernetes.io/docs/concepts/storage/volumes#nfs
Represents an NFS mount that lasts the lifetime of a pod. NFS volumes do
not support ownership management or SELinux relabeling.
FIELDS:
path <string> -required-
readOnly <boolean>
server <string> -required-
3、创建 pv
参考:https://kubernetes.io/zh/docs/concepts/storage/persistent-volumes/#reclaiming
ReadWriteOnce
卷可以被一个节点以读写方式挂载。 ReadWriteOnce 访问模式也允许运行在同一节点上的多个 Pod 访问卷。
ReadOnlyMany
卷可以被多个节点以只读方式挂载。
ReadWriteMany
卷可以被多个节点以读写方式挂载。
ReadWriteOncePod
卷可以被单个 Pod 以读写方式挂载。 如果你想确保整个集群中只有一个 Pod 可以读取或写入该 PVC, 请使用ReadWriteOncePod 访问模式。这只支持 CSI 卷以及需要 Kubernetes 1.22 以上版本。
[root@xianchaomaster1 ~]# vim pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: v1
spec:
capacity:
storage: 1Gi #pv 的存储空间容量
accessModes: ["ReadWriteOnce"] #访问模式
nfs:
path: /data/volume_test/v1 #把 nfs 的存储空间创建成 pv
server: 192.168.11.139 #nfs 服务器的地址
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: v2
spec:
capacity:
storage: 2Gi
accessModes: ["ReadWriteMany"]
nfs:
path: /data/volume_test/v1
server: 192.168.11.139
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: v3
spec:
capacity:
storage: 3Gi
accessModes: ["ReadWriteMany"]
nfs:
path: /data/volume_test/v1
server: 192.168.11.139
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: v4
spec:
capacity:
storage: 4Gi
accessModes: ["ReadWriteOnce","ReadWriteMany"]
nfs:
path: /data/volume_test/v1
server: 192.168.11.139
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: v5
spec:
capacity:
storage: 5Gi
accessModes: ["ReadWriteOnce","ReadWriteMany"]
nfs:
path: /data/volume_test/v1
server: 192.168.11.139
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: v6
spec:
capacity:
storage: 6Gi
accessModes: ["ReadWriteOnce","ReadWriteMany"]
nfs:
path: /data/volume_test/v1
server: 192.168.11.139
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: v7
spec:
capacity:
storage: 7Gi
accessModes: ["ReadWriteOnce","ReadWriteMany"]
nfs:
path: /data/volume_test/v1
server: 192.168.11.139
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: v8
spec:
capacity:
storage: 8Gi
accessModes: ["ReadWriteOnce","ReadWriteMany"]
nfs:
path: /data/volume_test/v1
server: 192.168.11.139
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: v9
spec:
capacity:
storage: 9Gi
accessModes: ["ReadWriteOnce","ReadWriteMany"]
nfs:
path: /data/volume_test/v1
server: 192.168.11.139
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: v10
spec:
capacity:
storage: 10Gi
accessModes: ["ReadWriteOnce","ReadWriteMany"]
nfs:
path: /data/volume_test/v1
server: 192.168.11.139
#更新资源清单文件
[root@k8smaster ~]# kubectl apply -f pv.yaml
persistentvolume/v1 created
persistentvolume/v2 created
persistentvolume/v3 created
persistentvolume/v4 created
persistentvolume/v5 created
persistentvolume/v6 created
persistentvolume/v7 created
persistentvolume/v8 created
persistentvolume/v9 created
persistentvolume/v10 created
#查看 pv 资源
[root@k8smaster ~]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
v1 1Gi RWO Retain Available 29s
v10 10Gi RWO,RWX Retain Available 29s
v2 2Gi RWX Retain Available 29s
v3 3Gi RWX Retain Available 29s
v4 4Gi RWO,RWX Retain Available 29s
v5 5Gi RWO,RWX Retain Available 29s
v6 6Gi RWO,RWX Retain Available 29s
v7 7Gi RWO,RWX Retain Available 29s
v8 8Gi RWO,RWX Retain Available 29s
v9 9Gi RWO,RWX Retain Available 29s
#STATUS 是 Available,表示 pv 是可用的
[root@k8smaster ~]# cd /data/volume_test/ #pv目录
[root@k8smaster volume_test]# ls
v1 v10 v2 v3 v4 v5 v6 v7 v8 v9
4、创建 pvc,和符合条件的 pv 绑定
[root@k8smaster ~]# vim pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: my-pvc
spec:
accessModes: ["ReadWriteMany"]
resources:
requests:
storage: 2Gi
#更新资源清单文件
[root@k8smaster ~]# kubectl apply -f pvc.yaml
persistentvolumeclaim/my-pvc created
#查看 pv 和 pvc
[root@k8smaster ~]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
v1 1Gi RWO Retain Available 5m8s
v10 10Gi RWO,RWX Retain Available 5m8s
v2 2Gi RWX Retain Bound default/my-pvc 5m8s
v3 3Gi RWX Retain Available 5m8s
v4 4Gi RWO,RWX Retain Available 5m8s
v5 5Gi RWO,RWX Retain Available 5m8s
v6 6Gi RWO,RWX Retain Available 5m8s
v7 7Gi RWO,RWX Retain Available 5m8s
v8 8Gi RWO,RWX Retain Available 5m8s
v9 9Gi RWO,RWX Retain Available 5m8s
#STATUS 是 Bound,表示这个 pv 已经被 my-pvc 绑定了
[root@k8smaster ~]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
my-pvc Bound v2 2Gi RWX 32s
pvc 的名字-绑定到 pv-绑定的是 v2 这个 pv-pvc 可使用的容量是 2G
5、创建 pod,挂载 pvc
[root@k8smaster ~]# vim pod_pvc.yaml
apiVersion: v1
kind: Pod
metadata:
name: pod-pvc
spec:
containers:
- name: nginx
image: nginx
imagePullPolicy: IfNotPresent
volumeMounts:
- name: nginx-html
mountPath: /usr/share/nginx/html
volumes:
- name: nginx-html
persistentVolumeClaim:
claimName: my-pvc
#更新资源清单文件
[root@k8smaster ~]# kubectl apply -f pod_pvc.yaml
pod/pod-pvc created
#查看 pod 状态
[root@k8smaster ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
pod-pvc 0/1 Running 0 5s
#通过上面可以看到 pod 处于 running 状态,正常运行
#每次pvc和pv绑定,pv只认哪一个,如果pvc被删除,那么pv会进入释放状态(有数据)并且不让其他pvc绑定。如果原来的pv不能绑定,会自动找最接近需求的新的pv,如果找不到就会进入pending状态。
注:使用 pvc 和 pv 的注意事项
1、我们每次创建 pvc 的时候,需要事先有划分好的 pv,这样可能不方便,那么可以在创建 pvc 的时候直接动态创建一个 pv 这个存储类,pv 事先是不存在的
2、pvc 和 pv 绑定,如果使用默认的回收策略 retain,那么删除 pvc 之后,pv 会处于 released 状态,我们想要继续使用这个 pv,需要手动删除 pv,kubectl delete pv pv_name,删除 pv,不会删除 pv里的数据,当我们重新创建 pvc 时还会和这个最匹配的 pv 绑定,数据还是原来数据,不会丢失。
#先删除pod 不然删不掉pvc 删掉pvc之后再删pv 是这个顺序 如果不删除pod在用pvc的话是不让删除的
[root@k8smaster ~]# kubectl delete pods pod-pvc --grace-period=0 --force
[root@k8smaster ~]# kubectl delete -f pvc.yaml
persistentvolumeclaim "my-pvc" deleted
[root@k8smaster ~]# kubectl delete -f pv.yaml
[root@k8smaster ~]# kubectl apply -f pv.yaml
[root@k8smaster ~]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
v1 1Gi RWO Retain Available 5s
v10 10Gi RWO,RWX Retain Available 4s
v2 2Gi RWX Retain Available 5s
v3 3Gi RWX Retain Available 4s
v4 4Gi RWO,RWX Retain Available 4s
v5 5Gi RWO,RWX Retain Available 4s
v6 6Gi RWO,RWX Retain Available 4s
v7 7Gi RWO,RWX Retain Available 4s
v8 8Gi RWO,RWX Retain Available 4s
v9 9Gi RWO,RWX Retain Available 4s
[root@k8smaster ~]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
my-pvc Bound v2 2Gi RWX 7s
#还是和2绑定
#修改回收策略
[root@k8smaster ~]# kubectl explain pv.spec.persistentVolumeReclaimPolicy
KIND: PersistentVolume
VERSION: v1
FIELD: persistentVolumeReclaimPolicy <string>
[root@k8smaster ~]# vim pv.yaml
#修改第二个
apiVersion: v1
kind: PersistentVolume
metadata:
name: v2
spec:
persistentVolumeReclaimPolicy: Delete
[root@k8smaster ~]# kubectl apply -f pv.yaml
[root@k8smaster ~]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
v1 1Gi RWO Retain Available 7s
v10 10Gi RWO,RWX Retain Available 7s
v2 2Gi RWX Delete Available 7s
v3 3Gi RWX Retain Available 7s
v4 4Gi RWO,RWX Retain Available 7s
v5 5Gi RWO,RWX Retain Available 7s
v6 6Gi RWO,RWX Retain Available 7s
v7 7Gi RWO,RWX Retain Available 7s
v8 8Gi RWO,RWX Retain Available 7s
v9 9Gi RWO,RWX Retain Available 7s
#变成delete了 测试一下更新pvc然后是否绑定等
[root@k8smaster ~]# kubectl apply -f pvc.yaml
persistentvolumeclaim/my-pvc created
[root@k8smaster ~]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
my-pvc Bound v2 2Gi RWX 11s
#如果此时删除了pvc和pv,pv存储目录的数据文件不会丢失,如果想彻底清理,删除pvc,pv和目录存储的数据删除
write at the end
It is not easy to create, if you think the content is helpful to you, please give me a three-link follow to support me! If there are any mistakes, please point them out in the comments and I will change them in time!
The series that is currently being updated: learn k8s from scratch.
Thank you for watching. The article is mixed with personal understanding. If there is any error, please contact me and point it out~