Brief description of GFS distributed file system and Gluster cluster deployment

1. Overview of GFS

1.1 Introduction to Gluster

• Open source distributed file system
• Composed of storage server, client and NFS/Samba storage network management
• No metadata server
• GlusterFS features
  Scalability and high performance
  High availability
  Global unified namespace
  Flexible volume management
  Based on standard protocols

1.2 Gluster commonly used terms

• Brick: The storage unit in GFS, through an export directory of a server in a trusted storage pool. Can be identified by host name and directory name, such as'SERVER:EXPORT'
• Volume: Volume
• FUSE: Kernel file system, Filesystem Userspace is a loadable kernel module, which supports non-privileged users to create their own file system without modifying the kernel code. By running the file system code in the user space, the FUSE code is bridged with the kernel.
• VFS: Virtual File System
• Glusterd: Gluster management daemon, to run on all servers in the trusted storage pool.

1.3 Gluster structure and working principle

Structure
Modular stacking architecture
Modular and stacking architecture
Through the combination of modules, complex functional
working principles are realized
1. Clients or applications access data through the mount point of GlusterFS
2. Linux system kernel receives data through VFS API Request and process
3. The VFS submits the data to the FUSE kernel file system, and the fuse file system submits the data to the GlusterFS client through the /dev/fuse device file.
4. After the GlusterFS client receives the data, the client is configured according to the configuration file Process the data
5. Pass the data to the remote GlusterFS Server through the network, and write the data to the server storage device

1.4 GlusterFS algorithm

• Get a 32-bit integer through the HASH algorithm
• Divided into N continuous subspaces, each space corresponds to a Brick
• Advantages of flexible hash algorithm
• Ensure that the data is evenly distributed in each Brick
• Solve the dependence on the metadata server, thereby solving the single point of failure and access bottleneck

Two, GlusterFS volume type

• Distributed volume
• Striped roll
• Copy volume
• Distributed stripe volume
• Distributed replication volume
• Striped replication volume
• Distributed striped replicated volume

2.1 Distributed volume

• The file is not divided into blocks
• Save Hash value through extended file attributes
• The supported underlying file systems are EXT3, EXT4, ZFS, XFS, etc.
• Without block processing, files can only be stored in one server, and efficiency is not improved.
  Distributed characteristics.
  Files are distributed on different servers and do not have redundancy.
  Easier and cheap. Low-expansion volume.
  Single point of failure can cause data loss
  . Data protection
  creates distributed volumes

gluster volume  create  dis-volume  server1:/dir1   server2:/dir2   s erver3:/dir3
## 指定名称为 dis-volume    包含 server1:/dir1   server2:/dir2   s erver3:/dir3

2.2 Strip roll

• Divide the file into N blocks (N strip nodes) according to the offset, and store the polled ones in each Brick Server node
• Performance is particularly outstanding when storing large files
• No redundancy, similar to Raid0
  .
  Data is divided into smaller pieces and distributed to different striped areas in the block server group.
  Distribution reduces the load and smaller files accelerate the speed of access.
  No data redundancy
  creates stripes. volume

gluster volume  create  stripe-volume stripe 2   transport   tcp   server1:/dir1  server2:/dir2
## 名称为 stripe-volume    每个文件分为两块（stripe  2） ，  指定传输协议为tcp（transport   tcp）  包含server1:/dir1  server2:/dir2

2.3 Copy volume

• Save one or more copies of the same file
• Because of the need to save copies, disk utilization is low
• If the storage space on multiple nodes is the same, the capacity of the lowest node will be taken as the total capacity
  feature of the
  volume to be modified according to the barrel effect . All servers in the
  volume store a complete copy . The number of copies of the volume can be determined
  at least when the customer creates There are two block servers or more servers
  with redundancy to
  create replicated volumes

gluster volume  create  rep-volume replica 2   transport   tcp   server1:/dir1  server2:/dir2  
# 创建名为rep-volume的复制卷，文件将同时保存两份副本，分别再Server1：/dir1 和Server2：/dir2 中

2.4 Distributed striped volume

• Take into account the functions of distributed volumes and striped volumes
• Mainly used for large file access processing
• At least 4 servers are required to
  create distributed striped volumes

创建了名为dis-stripe的分布式条带卷，配置分布式的条带卷时，卷中Brick所包含的存储服务器必须是条带数的倍数（>=2倍）
gluster volume create dis-stripe stripe 2  node1:/data/sdd1 node2:/data/sdd1 node3:/data/sdd1 node4:/data/sdd1 force

2.5 Distributed replicated volumes

• Take into account the functions of distributed and replicated volumes
• Used when redundancy is required

Create command

创建了名为dis-rep 的分布式复制卷，配置分布式的条带卷时，卷中Brick所包含的存储服务器必须是条带数的倍数（>=2倍）
gluster volume create dis-rep replica 2  node1:/data/sde1 node2:/data/sde1 node3:/data/sde1  node4:/data/sde1  force

Three, Gluster cluster deployment

3.1 Experimental environment

The IP addresses of the 4 virtual machines are
node1 192.168.233.100
node2 192.168.233.100
node3 192.168.233.180
node4 192.168.233.50
Each virtual machine adds 4 hard disks with a size of 20G

3.2 Experiment preparation

root@localhost ~]# hostnamectl set-hostname node1  ## 4台服务器依次修改主机名
[root@localhost ~]# su
[root@node1 ~]# vim /etc/hosts     ## 四台都要写入 主机名要和IP地址对应   用 scp /etc/hosts  [email protected]:/etc/hosts  为了方便  可以传输过去
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.233.100 node1
192.168.233.200 node2
192.168.233.180 node3
192.168.233.50 node4
[root@node1 ~]# systemctl stop firewalld ## 四台服务器都关闭防火墙   清空规则没用  还是会有干扰
[root@node1 ~]# setenforce 0

3.3 Configure yum source to install Gluster

• Each node must be installed

root@promote ~]# mount -t cifs -o username=wang,password=248569 //192.168.1.162/gfsrepo /mnt

[root@localhost ~]# cd /etc/yum.repos.d/
[root@localhost yum.repos.d]# mkdir bak
[root@localhost yum.repos.d]# mv C* bak
[root@localhost yum.repos.d]# vim GLFS.repo
[GLFS]
name=glfs
baseurl=file:///mnt/
gpgcheck=0
enable=1
[root@localhost ~]# yum clean all
[root@localhost ~]#  yum list
[root@node1 mnt]# yum -y install glusterfs-server  glusterfs-rdma glusterfs-fuse
[root@node1 ~]# systemctl start  glusterd.service    ## gluster服务启动 
[root@node1 ~]# systemctl enable glusterd.service  ## 开机自启动
[root@node1 ~]# systemctl status   glusterd.service  ## 查看状态

3.4 Check the time, add gluster neighbor

[root@node1 ~]# ntpdate ntp1.aliyun.com    ## 确保后面数据是正的
13 Sep 22:20:58 ntpdate[20333]: adjust time server 120.25.115.20 offset 0.000040 sec
#####添加入存储信任池只要在一台主机上添加其他节点即可
[root@node1 ~]# gluster peer probe node2
peer probe: success. 
[root@node1 ~]# gluster peer probe node3
peer probe: success. 
[root@node1 ~]# gluster peer probe node4
peer probe: success. 

3.5 View gluster nodes

[root@node1 ~]# gluster peer status    ## 查看所有节点
Number of Peers: 3

Hostname: node2
Uuid: 2cfcfa43-89ba-40ec-ba5e-e6873a40b649
State: Peer in Cluster (Connected)

Hostname: node3
Uuid: b7c73acb-a677-45a4-9e14-6cec62f1cc86
State: Peer in Cluster (Connected)

Hostname: node4
Uuid: d2a4f351-3e22-416b-9a4a-b6713dc6c5c7
State: Peer in Cluster (Connected)

3.6 Client operation

[root@client ~]# systemctl stop firewalld
[root@client ~]# setenforce 0

[root@client  ~]# mount -t cifs -o username=wang,password=248569 //192.168.1.162/gfsrepo /mnt

[root@client  ~]# cd /etc/yum.repos.d/
[root@client  yum.repos.d]# mkdir bak
[root@client  yum.repos.d]# mv C* bak
[root@client  yum.repos.d]# vim GLFS.repo
[GLFS]
name=glfs
baseurl=file:///mnt/
gpgcheck=0
enable=1

[root@client ~]# yum clean all
[root@client ~]#  yum list
[root@client ~]# yum remove glusterfs-libs.x86_64
[root@client mnt]# yum -y install glusterfs-server  glusterfs-rdma glusterfs-fuse
[root@client ~]# vim /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.233.100 node1
192.168.233.200 node2
192.168.233.180 node3
192.168.233.50 node4

3.7 Creation of various volumes

3.7.1 Verification of the creation and storage of striped volumes

########创建 条带卷 随便哪个node节点都可以
[root@node1 ~]# gluster volume create stripe-vol stripe 2 node1:/data/sdc1 node2:/data/sdc1 force ## 创建条带卷
volume create: stripe-vol: success: please start the volume to access data
[root@node1 ~]# gluster volume list  ## 查看卷
stripe-vol
[root@node1 ~]# gluster volume info stripe-vol   ## 查看stripe-vol
 
Volume Name: stripe-vol
Type: Stripe
Volume ID: 164d4e23-51af-47b6-8b77-a8c50b267e52
Status: Created
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: node1:/data/sdc1
Brick2: node2:/data/sdc1
Options Reconfigured:
transport.address-family: inet
nfs.disable: on

[root@node1 ~]# gluster volume start stripe-vol  ## 开启卷 才能挂载
volume start: stripe-vol: success

verification

### 客户端验证条带卷
dd if=/dev/zero of=/demo1.log bs=1M count=40 ## 创建5个用于测试的包，每个40M
dd if=/dev/zero of=/demo2.log bs=1M count=40
dd if=/dev/zero of=/demo3.log bs=1M count=40
dd if=/dev/zero of=/demo4.log bs=1M count=40
dd if=/dev/zero of=/demo5.log bs=1M count=40
[root@client ~]# mkdir -p /test/stripe-vol
[root@client ~]# mount.glusterfs node1:stripe-vol  /test/stripe-vol/ ## 这里node用哪个都可以 挂载 stripe-vol 到  /test/stripe-vol/

[root@client ~]#  cd /
[root@client /]# ls
bin   demo1.log  demo3.log  demo5.log  etc   lib    media  opt   root  sbin  sys   tmp  var
boot  demo2.log  demo4.log  dev        home  lib64  mnt    proc  run   srv   test  usr
[root@client /]# cp -p demo* /test/stripe-vol/   ## 将测试的文件复制到/test/stripe-vol/ 


查看node2
[root@node2 ~]# cd /data/sdc1/
[root@node2 sdc1]# ls -lh
总用量 100M
-rw-r--r--. 2 root root 20M 9月  14 09:51 demo1.log
-rw-r--r--. 2 root root 20M 9月  14 09:51 demo2.log
-rw-r--r--. 2 root root 20M 9月  14 09:51 demo3.log
-rw-r--r--. 2 root root 20M 9月  14 09:51 demo4.log
-rw-r--r--. 2 root root 20M 9月  14 09:51 demo5.log
[root@node1 ~]# cd /data/sdc1/
[root@node1 sdc1]# ls -lh
总用量 100M
-rw-r--r--. 2 root root 20M 9月  14 09:51 demo1.log
-rw-r--r--. 2 root root 20M 9月  14 09:51 demo2.log
-rw-r--r--. 2 root root 20M 9月  14 09:51 demo3.log
-rw-r--r--. 2 root root 20M 9月  14 09:51 demo4.log
-rw-r--r--. 2 root root 20M 9月  14 09:51 demo5.log

3.7.2 Verification of the creation and storage of distributed volumes

• Node creates distributed volume

创建 分布式卷
[root@node1 sdc1]# gluster volume create dis-vol  node1:/data/sdb1 node2:/data/sdb1 force
volume create: dis-vol: success: please start the volume to access data
[root@node1 sdc1]# gluster volume start dis-vol 
volume start: dis-vol: success

• Verification
  -Client

[root@client /]# mount.glusterfs node1:dis-vol  /test/dis-vol/
[root@client /]# cp -p demo* /test/dis-vol/   ## 将测试的文件复制到/test/dis -vol/

[root@node1 sdc1]# cd /data/sdb1/
您在 /var/spool/mail/root 中有新邮件
[root@node1 sdb1]# ls -lh
总用量 160M
-rw-r--r--. 2 root root 40M 9月  14 09:51 demo1.log
-rw-r--r--. 2 root root 40M 9月  14 09:51 demo2.log
-rw-r--r--. 2 root root 40M 9月  14 09:51 demo3.log
-rw-r--r--. 2 root root 40M 9月  14 09:51 demo4.log


[root@node2 sdc1]# cd /data/sdb1/
[root@node2 sdb1]# ls -lh
总用量 40M
-rw-r--r--. 2 root root 40M 9月  14 09:51 demo5.log

3.7.3 Creation and storage mode verification of replicated volumes

• Create a replicated volume

[root@node1 sdb1]# gluster volume create rep-vol replica 2  node3:/data/sdb1 node4:/data/sdb1 force
volume create: rep-vol: success: please start the volume to access data
[root@node1 sdb1]# gluster volume start rep-vol 
volume start: rep-vol: success

• verification

####### 客户端
[root@client test]# mkdir -p /test/rep-vol
[root@client test]# mount.glusterfs node1:rep-vol  /test/rep-vol/
[root@client /]#  cp -p demo* /test/rep-vol/

## node 3
[root@node3 ~]# cd /data/sdb1/
[root@node3 sdb1]# ls -lh
总用量 200M
-rw-r--r--. 2 root root 40M 9月  14 09:51 demo1.log
-rw-r--r--. 2 root root 40M 9月  14 09:51 demo2.log
-rw-r--r--. 2 root root 40M 9月  14 09:51 demo3.log
-rw-r--r--. 2 root root 40M 9月  14 09:51 demo4.log
-rw-r--r--. 2 root root 40M 9月  14 09:51 demo5.log
#### node 4
[root@node4 ~]# cd /data/sdb1/
[root@node4 sdb1]# ls -lh
总用量 200M
-rw-r--r--. 2 root root 40M 9月  14 09:51 demo1.log
-rw-r--r--. 2 root root 40M 9月  14 09:51 demo2.log
-rw-r--r--. 2 root root 40M 9月  14 09:51 demo3.log
-rw-r--r--. 2 root root 40M 9月  14 09:51 demo4.log
-rw-r--r--. 2 root root 40M 9月  14 09:51 demo5.log

3.7.4 Verification of the creation and storage mode of distributed striped volumes

• Create a distributed striped volume

[root@node1 sdb1]# gluster volume create dis-stripe stripe 2  node1:/data/sdd1 node2:/data/sdd1 node3:/data/sdd1 node4:/data/sdd1 force
volume create: dis-stripe: success: please start the volume to access data
[root@node1 sdb1]# gluster volume start dis-stripe 
volume start: dis-stripe: success

• verification

客户端
[root@client /]# mkdir -p /test/dis-stripe
[root@client /]# mount.glusterfs node1:dis-stripe  /test/dis-stripe/
[root@client /]#  cp -p demo* /test/dis-stripe/


验证
[root@node1 sdb1]# cd /data/sdd1/
[root@node1 sdd1]# ls -lh
总用量 80M
-rw-r--r--. 2 root root 20M 9月  14 09:51 demo1.log
-rw-r--r--. 2 root root 20M 9月  14 09:51 demo2.log
-rw-r--r--. 2 root root 20M 9月  14 09:51 demo3.log
-rw-r--r--. 2 root root 20M 9月  14 09:51 demo4.log

[root@node2 sdb1]# cd /data/sdd1/
[root@node2 sdd1]# ls -lh
总用量  80M
-rw-r--r--. 2 root root 20M 9月  14 09:51 demo1.log
-rw-r--r--. 2 root root 20M 9月  14 09:51 demo2.log
-rw-r--r--. 2 root root 20M 9月  14 09:51 demo3.log
-rw-r--r--. 2 root root 20M 9月  14 09:51 demo4.log

[root@node3 sdb1]# cd /data/sdd1/
[root@node3 sdd1]# ls -lh
总用量 20M
-rw-r--r--. 2 root root 20M 9月  14 09:51 demo5.log


[root@node4 sdb1]# cd /data/sdd1/
[root@node4 sdd1]# ls -lh
总用量 20M
-rw-r--r--. 2 root root 20M 9月  14 09:51 demo5.log

3.7.5 Verification of the creation and storage mode of distributed replication volumes

• Create distributed replicated volumes

###########################分布式复制卷#######
[root@node1 sdd1]# gluster volume create dis-rep replica 2  node1:/data/sde1 node2:/data/sde1 node3:/data/sde1  node4:/data/sde1  force
volume create: dis-rep: success: please start the volume to access data
[root@node1 sdd1]# gluster volume start dis-rep 
volume start: dis-rep: success

• verification

#######客户端#############

[root@client /]# mkdir -p /test/dis-rep
[root@client /]# mount.glusterfs node1:dis-rep  /test/dis-rep/


验证
[root@node1 sdd1]# cd /data/sde1/
[root@node1 sde1]# ls -lh
总用量 160M
-rw-r--r--. 2 root root 40M 9月  14 09:51 demo1.log
-rw-r--r--. 2 root root 40M 9月  14 09:51 demo2.log
-rw-r--r--. 2 root root 40M 9月  14 09:51 demo3.log
-rw-r--r--. 2 root root 40M 9月  14 09:51 demo4.log

[root@node2 sdd1]# cd /data/sde1/
[root@node2 sde1]# ls -lh
总用量 160M
-rw-r--r--. 2 root root 40M 9月  14 09:51 demo1.log
-rw-r--r--. 2 root root 40M 9月  14 09:51 demo2.log
-rw-r--r--. 2 root root 40M 9月  14 09:51 demo3.log
-rw-r--r--. 2 root root 40M 9月  14 09:51 demo4.log


[root@node3 sdd1]# cd /data/sde1/
[root@node3 sde1]# ls -lh
总用量 120M
-rw-r--r--. 2 root root 40M 9月  14 09:51 demo5.log
-rw-r--r--. 2 root root 40M 9月  14 10:36 demo6.log
-rw-r--r--. 2 root root 40M 9月  14 10:37 demo7.log

[root@node4 sdb1]# cd /data/sde1/
[root@node4 sde1]# ls -lh
总用量 120M
-rw-r--r--. 2 root root 40M 9月  14 09:51 demo5.log
-rw-r--r--. 2 root root 40M 9月  14 10:36 demo6.log
-rw-r--r--. 2 root root 40M 9月  14 10:37 demo7.log

3.8 Destruction test

Shut down the node2 node.
In addition to copying the volume, there will be more or less problems.
The striped volume is wiped out, and the data on the distributed node is gone.

3.9 Access control

[root@node1 sde1]# gluster volume set dis-rep auth.reject 192.168.233.30  ## 禁止 192.168.233.30 使用dis-rep卷（黑名单）
volume set: success
### 客户端 挂载这个卷失败
[root@client /]# mount.glusterfs node1:dis-rep  /test/dis-rep/
Mount failed. Please check the log file for more details.
[root@node1 sde1]# gluster volume set dis-rep auth.allow  192.168.233.30  ## 仅允许 192.168.233.30 使用dis-rep卷（白名单）