Centos7 mounting GlusterFS (copy mode)

Centos7 mounting GlusterFS (copy mode)

Under Tucao, being unfamiliar street of benefits to sour, everybody has a iphone 11 256G, but also recruit people do not? Ha ha ha

1. Background

公司目前k8s用到的存储开源软件有2个，一个是FastDFS(10台集群)，一个是NFS(2台，一主一备)；FastDFS主要用来存合同、图片、语音文件等，但这个主要是通过api来写入和读取文件，不能像NFS或者GlusterFs一样可以挂载到容器内；NFS目前用来做prometheus-operator的后端存储以及前端文件的共享存储，考虑到NFS不太容易支持高可用，做起来很费劲，年后计划用GlusterFS替换NFS，年前先调研测试下。

2, Environment Description

3台机器安装 GlusterFS 组成一个集群。

服务器和hosts：
192.168.206.148 glusterfs01
192.168.206.149 glusterfs02
192.168.206.150 glusterfs03

服务器版本：
[root@glusterfs01 glusterd]# cat /etc/redhat-release 
CentOS Linux release 7.7.1908 (Core)

[root@glusterfs01 glusterd]# uname -r
3.10.0-1062.9.1.el7.x86_64

3, installation GluesterFS

• Net official document as follows:
• Fedora 26 (or later) on 3 nodes named "server1", "server2" and "server3"
• A working network connection
• At least two virtual disks, one for the OS installation, and one to be used to serve GlusterFS storage (sdb), on each of these VMs. This will emulate a real-world deployment, where you would want to separate GlusterFS storage from the OS install.
• Setup NTP on each of these servers to get the proper functioning of many applications on top of filesystem.

三台节点各增加一块单独的磁盘并格式化（在三个节点都执行）

# mkfs.xfs -i size=512 /dev/sdb1
# mkdir -p /data/brick1
# echo '/dev/sdb1 /data/brick1 xfs defaults 1 2' >> /etc/fstab
# mount -a && mount

在三个节点都安装glusterts（在三个节点都执行）

yum install centos-release-gluster -y

yum install -y glusterfs glusterfs-server glusterfs-fuse glusterfs-rdma

配置 GlusterFS 集群：

启动 glusterFS（在三个节点都执行）

systemctl start glusterd.service

systemctl enable glusterd.service


在glusterfs01节点上配置，将节点加入到 集群中。（在其中一个节点执行，这里选择glusterfs01节点执行）

[root@glusterfs01 ~]#gluster peer probe glusterfs01
peer probe: success. Probe on localhost not needed

[root@glusterfs01 ~]#gluster peer probe glusterfs02
peer probe: success.

[root@glusterfs01 ~]#gluster peer probe glusterfs03
peer probe: success.


查看集群状态：
[root@glusterfs01 ~]# gluster peer status
Number of Peers: 2

Hostname: glusterfs02
Uuid: 4acc65b4-9f5a-427c-827b-564d5f60d153
State: Peer in Cluster (Connected)

Hostname: glusterfs03
Uuid: 45f475ba-945d-4f57-98e1-d195d17bf97a
State: Peer in Cluster (Connected)



创建数据存储目录（在三个节点都执行）
[root@glusterfs01 ~]# mkdir -p /data/brick1/gv0
[root@glusterfs02 ~]# mkdir -p /data/brick1/gv0
[root@glusterfs03 ~]# mkdir -p /data/brick1/gv0

查看volume 状态：
[root@glusterfs01 ~]# gluster volume info
No volumes present


创建GlusterFS磁盘(在其中一个节点执行，这里选择glusterfs01)
gluster volume create gv0 replica 3 glusterfs01:/data/brick1/gv0 glusterfs02:/data/brick1/gv0 glusterfs03:/data/brick1/gv0

4, GlusterFS several volume Mode Description

First, the default mode, both DHT, also known as distributed volumes: the file hash algorithm randomly distributed to one server node is stored.
gluster volume create test-volume server1: / exp1 server2: / exp2

Second, copy mode, both AFR, when the volume is created with replica number x: Copy the file to a replica x node.
gluster volume create test-volume replica 2 transport tcp server1: / exp1 server2: / exp2

Third, the banding patterns, either Striped, when creating a volume with a stripe number x: cut file data blocks are stored in the nodes in X stripe (similar raid 0).
gluster volume create test-volume stripe 2 transport tcp server1: / exp1 server2: / exp2

Fourth, the band distributed mode (combined type), requires a minimum of four servers can be created. Stripe 2 server = 4 nodes when creating the volume: DHT is combined with a Striped type.
gluster volume create test-volume stripe 2 transport tcp server1: / exp1 server2: / exp2 server3: / exp3 server4: / exp4

V. distributed copy mode (combined type), it requires a minimum of four servers can be created. When creating a replica volume 2 server = 4 nodes: DHT is a combination of the AFR type.
gluster volume create test-volume replica 2 transport tcp server1: / exp1 server2: / exp2 server3: / exp3 server4: / exp4

Sixth, striped volume copy mode (combined type), requires a minimum of four servers can be created. Stripe 2 replica 2 server = 4 nodes when creating the volume: Striped and AFR is the combination type.
gluster volume create test-volume stripe 2 replica 2 transport tcp server1: / exp1 server2: / exp2 server3: / exp3 server4: / exp4

Seven, three models mixed, at least eight servers can be created. stripe 2 replica 2, each composed of a group of four nodes.
gluster volume create test-volume stripe 2 replica 2 transport tcp server1: / exp1 server2: / exp2 server3: / exp3 server4: / exp4 server5: / exp5 server6: / exp6 server7: / exp7 server8: / exp8

Eight, and then view the volume status:

[root@glusterfs01 ~]#gluster volume info

Volume Name: gv0
Type: Replicate
Volume ID: bc0efcd9-e799-401a-b1fc-7ff73086d7bc
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: glusterfs01:/data/brick1/gv0
Brick2: glusterfs02:/data/brick1/gv0
Brick3: glusterfs03:/data/brick1/gv0
Options Reconfigured:
transport.address-family: inet
storage.fips-mode-rchecksum: on
nfs.disable: on
performance.client-io-threads: off

Nine, start models

[root@glusterfs01 ~]# gluster volume start gv0
volume start: models: success

5, test GlusterFS volume

As a client for testing glusterfs01

mount -t glusterfs glusterfs01:/gv0 /mnt
for i in `seq -w 1 100`; do cp -rp /var/log/messages /mnt/copy-test-$i; done

分别在在3台机器上查看，这里选择glusterfs01一台查看
[root@glusterfs01 ~]# ls -lA /data/brick1/gv0/copy*
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-001
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-002
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-003
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-004
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-005
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-006
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-007
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-008
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-009
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-010
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-011
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-012
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-013
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-014
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-015
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-016
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-017
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-018
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-019
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-020
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-021
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-022
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-023
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-024
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-025
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-026
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-027
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-028
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-029
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-030
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-031
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-032
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-033
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-034
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-035
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-036
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-037
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-038
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-039
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-040
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-041
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-042
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-043
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-044
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-045
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-046
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-047
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-048
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-049
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-050
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-051
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-052
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-053
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-054
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-055
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-056
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-057
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-058
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-059
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-060
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-061
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-062
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-063
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-064
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-065
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-066
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-067
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-068
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-069
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-070
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-071
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-072
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-073
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-074
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-075
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-076
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-077
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-078
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-079
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-080
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-081
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-082
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-083
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-084
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-085
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-086
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-087
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-088
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-089
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-090
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-091
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-092
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-093
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-094
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-095
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-096
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-097
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-098
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-099
-rw------- 2 root root 565327 1月  11 14:54 /data/brick1/gv0/copy-test-100

6, reference

https://docs.gluster.org/en/latest/Quick-Start-Guide/Quickstart/
https://www.cnblogs.com/jicki/p/5801712.html