实验环境
使用了两台虚拟主机做实验
192.168.56.7
192.168.56.8
环境准备
/usr/sbin/ntpdate ntp.api.bz
关闭防火墙
/etc/init.d/iptables stop
关闭SElinux
setenforce 0
安装相关依赖包
yum install -y libibverbs librdmacm xfsprogs nfs-utils rpcbind libaio liblvm2app lvm2-devel
安装服务器端软件包
glusterfs-3.4.6-1.el6.x86_64.rpm
glusterfs-api-3.4.6-1.el6.x86_64.rpm
glusterfs-cli-3.4.6-1.el6.x86_64.rpm
glusterfs-fuse-3.4.6-1.el6.x86_64.rpm
glusterfs-libs-3.4.6-1.el6.x86_64.rpm
glusterfs-server-3.4.6-1.el6.x86_64.rpm
cd /tools/glusterfs
[root@tom01 install]# rpm -ivh *.rpm
warning: glusterfs-3.4.6-1.el6.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID 4ab22bb3: NOKEY
Preparing… ########################################### [100%]
1:glusterfs-libs ########################################### [ 9%]
2:glusterfs ########################################### [ 18%]
3:glusterfs-api ########################################### [ 27%]
4:glusterfs-devel ########################################### [ 36%]
5:glusterfs-fuse ########################################### [ 45%]
6:glusterfs-cli ########################################### [ 55%]
7:glusterfs-server ########################################### [ 64%]
8:glusterfs-geo-replicati########################################### [ 73%]
9:glusterfs-api-devel ########################################### [ 82%]
10:glusterfs-rdma ########################################### [ 91%]
11:glusterfs-debuginfo ########################################### [100%]
安装测试工具
yum -y install atop fio iperf
rpm -ivh iozone-3.394-1.el6.rf.x86_64.rpm
Gluster服务启动
/etc/init.d/glusterd start
chkconfig glusterd on
创建信任池 （因为是无中心，所有在任意节点操作都可以）
默认端口24007
gluster peer probe 192.168.56.8
查看信任池
[root@tom01 tools]# gluster peer status
Number of Peers: 1
Hostname: 192.168.56.8
Port: 24007
Uuid: cf764db6-9b6d-444b-9cf1-3544e7c38849
State: Peer in Cluster (Connected)
集群扩展节点（信任池）
[root@tom01 tools]# gluster peer probe 192.168.56.9
peer probe: success
[root@tom01 tools]# gluster peer status
Number of Peers: 2
Hostname: 192.168.56.8
Port: 24007
Uuid: cf764db6-9b6d-444b-9cf1-3544e7c38849
State: Peer in Cluster (Connected)
Hostname: 192.168.56.9
Port: 24007
Uuid: ec99080e-64dc-4034-8285-58d38deb0b61
State: Peer in Cluster (Connected)
删除节点
[root@tom01 tools]# gluster peer detach 192.168.56.9
peer detach: success
[root@tom01 tools]# gluster peer status
Number of Peers: 1
Hostname: 192.168.56.8
Port: 24007
Uuid: cf764db6-9b6d-444b-9cf1-3544e7c38849
State: Peer in Cluster (Connected)
创建卷
这里做实验没有单独挂硬盘，所以在本地硬盘上创建一个目录模拟
mkdir -p /glufs/exp1 192.168.56.7
mkdir -p /glufs/exp2 192.168.56.8
因为是无中心在那台机器上操作都可以
[root@tom01 tools]# gluster volume create test-volume 192.168.56.7:/glufs/exp1 192.168.56.8:/glufs/exp2 force
volume create: test-volume: success: please start the volume to access data
命令注释：test-volume 为卷名 默认不添加类型为分布式卷
注：一块硬盘下使用force参数 建议单独挂载块硬盘
启动卷
[root@tom01 tools]# gluster volume start test-volume
volume start: test-volume: success
查看卷信息
[root@tom01 tools]# gluster volume info
Volume Name: test-volume
Type: Distribute
Volume ID: 846d46b7-e2e2-4792-90d2-df7d15132dbd
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: 192.168.56.7:/glufs/exp1
Brick2: 192.168.56.8:/glufs/exp2
查看目录扩展属性（在目录所在机器上查看）
[root@tom01 tools]# getfattr -d -m . -e hex /glufs/exp1
getfattr: Removing leading ‘/’ from absolute path names
#file: glufs/exp1
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000000000007ffffffe
trusted.glusterfs.volume-id=0x846d46b7e2e2479290d2df7d15132dbd
[root@tom02 tools]# getfattr -d -m . -e hex /glufs/exp2
getfattr: Removing leading ‘/’ from absolute path names
#file: glufs/exp2
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.volume-id=0x846d46b7e2e2479290d2df7d15132dbd
新开一台机器挂载查看 192.168.56.9
[root@tom01 tools]# mount -t glusterfs 192.168.56.7:/test-volume /mnt
[root@tom01 tools]# df -TH
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/vg_tom01-lv_root
ext4 19G 1.5G 17G 9% /
tmpfs tmpfs 519M 0 519M 0% /dev/shm
/dev/sda1 ext4 500M 35M 439M 8% /boot
192.168.56.7:/test-volume
fuse.glusterfs 37G 3.4G 32G 10% /mnt
向卷中添加节点 192.168.56.9
1 先把新机器加入信任池
[root@tom01 tools]# gluster peer probe 192.168.56.9
peer probe: success
[root@tom01 tools]# gluster peer status
Number of Peers: 2
Hostname: 192.168.56.8
Port: 24007
Uuid: cf764db6-9b6d-444b-9cf1-3544e7c38849
State: Peer in Cluster (Connected)
Hostname: 192.168.56.9
Port: 24007
Uuid: ec99080e-64dc-4034-8285-58d38deb0b61
State: Peer in Cluster (Connected)
只有这一步要在新添加的主机上操作
把新节点加入卷
[root@tom01 glufs]# gluster volume add-brick test-volume 192.168.56.9:/glufs/exp3 force
volume add-brick: success
添加后查看
[root@tom01 glusterd]# gluster volume info
Volume Name: test-volume
Type: Distribute
Volume ID: 846d46b7-e2e2-4792-90d2-df7d15132dbd
Status: Started
Number of Bricks: 3
Transport-type: tcp
Bricks:
Brick1: 192.168.56.7:/glufs/exp1
Brick2: 192.168.56.8:/glufs/exp2
Brick3: 192.168.56.9:/glufs/exp3
重新分配数据（添加删除节点后必须要这么做）
[root@tom01 tools]# gluster volume rebalance test-volume start
查看rebalance状态（重新分配数据状态）
[root@tom01 tools]# gluster volume rebalance test-volume status
Node Rebalanced-files size scanned failures skipped status run time in secs
--------- ----------- ----------- ----------- ----------- ----------- ------------ --------------
localhost 0 0Bytes 0 0 0 completed 0.00
192.168.56.7 0 0Bytes 0 0 0 completed 0.00
192.168.56.7 0 0Bytes 0 0 0 completed 0.00
192.168.56.9 0 0Bytes 0 0 0 completed 0.00
volume rebalance: test-volume: success:
移除节点
[root@tom01 tools]# gluster volume remove-brick test-volume 192.168.56.9:/glufs/exp3
Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y
volume remove-brick commit force: success
查看卷内节点状态
[root@tom01 tools]# gluster volume status
Status of volume: test-volume
Gluster process Port Online Pid
Brick 192.168.56.7:/glufs/exp1 49152 Y 1980
Brick 192.168.56.8:/glufs/exp2 49152 Y 1911
NFS Server on localhost 2049 Y 2098
NFS Server on 192.168.56.7 2049 Y 2377
NFS Server on 192.168.56.9 2049 Y 2233
Task ID Status
---- -- ------
Rebalance 8d55f66e-19c4-42d2-a82c-5aa0b1be448d completed
删除卷
gluster volume stop test-volume
gluster volume delete test-volume