GlusterFS分布式存储系统中更换故障Brick的操作记录1

前面已经介绍了GlusterFS分布式存储集群环境部署记录,现在模拟下更换故障Brick的操作:

1)GlusterFS集群系统一共有4个节点,集群信息如下:

分别在各个节点上配置hosts、同步好系统时间,关闭防火墙和selinux
[root@GlusterFS-slave data]# cat /etc/hosts
192.168.10.239 GlusterFS-master 192.168.10.212 GlusterFS-slave 192.168.10.204 GlusterFS-slave2 192.168.10.220 GlusterFS-slave3 分别在各个节点上创建存储目录 首先新建分区 # fdisk /dev/sdb //依次输入p->n->1->回车->回车->w 发现并校验 # partx /dev/sdb # ls /dev/sdb* 创建文件系统 # mkfs.xfs -i size=1024 /dev/sdb1 配置挂载 # mkdir -p /data # echo '/dev/sdb1 /data xfs defaults 1 2' >> /etc/fstab # mount -a && mount 配置存储位置 # mkdir /data/gluster 部署glusterfs集群的中间部分操作在此省略,具体可参考:http://www.cnblogs.com/kevingrace/p/8743812.html [root@GlusterFS-master ~]# gluster peer status Number of Peers: 3 Hostname: 192.168.10.212 Uuid: f8e69297-4690-488e-b765-c1c404810d6a State: Peer in Cluster (Connected) Hostname: 192.168.10.204 Uuid: a989394c-f64a-40c3-8bc5-820f623952c4 State: Peer in Cluster (Connected) Hostname: 192.168.10.220 Uuid: dd99743a-285b-4aed-b3d6-e860f9efd965 State: Peer in Cluster (Connected) [root@GlusterFS-master ~]# gluster volume info Volume Name: models Type: Distributed-Replicate Volume ID: f1945b0b-67d6-4202-9198-639244ab0a6a Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: 192.168.10.239:/data/gluster Brick2: 192.168.10.212:/data/gluster Brick3: 192.168.10.204:/data/gluster Brick4: 192.168.10.220:/data/gluster Options Reconfigured: auth.allow: 192.168.* performance.write-behind: on performance.io-thread-count: 32 performance.flush-behind: on performance.cache-size: 128MB features.quota: on 客户端挂载GlusterFS存储 [root@Client ~]# mount -t glusterfs 192.168.10.239:models /data/gluster/

2)测试Gluster卷

写入测试数据
[root@Client ~]# for i in `seq -w 1 100`; do cp -rp /var/log/messages /opt/gfsmount/copy-test-$i; done 写入确认 [root@Client ~]# ls -lA /opt/gfsmount|wc -l 101 在各节点机器上也确认下,发现这100个文件随机地各自分为了两个50份的文件(均衡),分别同步到了第1-2节点和第3-4节点上了。 [root@GlusterFS-master ~]# ll /opt/gluster/data/|wc -l 51 [root@GlusterFS-master ~]# ls /opt/gluster/data/ copy-test-001 copy-test-016 copy-test-028 copy-test-038 copy-test-054 copy-test-078 copy-test-088 copy-test-100 copy-test-004 copy-test-017 copy-test-029 copy-test-039 copy-test-057 copy-test-079 copy-test-090 copy-test-006 copy-test-019 copy-test-030 copy-test-041 copy-test-060 copy-test-081 copy-test-093 copy-test-008 copy-test-021 copy-test-031 copy-test-046 copy-test-063 copy-test-082 copy-test-094 copy-test-011 copy-test-022 copy-test-032 copy-test-048 copy-test-065 copy-test-083 copy-test-095 copy-test-012 copy-test-023 copy-test-033 copy-test-051 copy-test-073 copy-test-086 copy-test-098 copy-test-015 copy-test-024 copy-test-034 copy-test-052 copy-test-077 copy-test-087 copy-test-099 [root@GlusterFS-slave ~]# ll /opt/gluster/data/|wc -l 51 [root@GlusterFS-slave ~]# ls /opt/gluster/data/ copy-test-001 copy-test-016 copy-test-028 copy-test-038 copy-test-054 copy-test-078 copy-test-088 copy-test-100 copy-test-004 copy-test-017 copy-test-029 copy-test-039 copy-test-057 copy-test-079 copy-test-090 copy-test-006 copy-test-019 copy-test-030 copy-test-041 copy-test-060 copy-test-081 copy-test-093 copy-test-008 copy-test-021 copy-test-031 copy-test-046 copy-test-063 copy-test-082 copy-test-094 copy-test-011 copy-test-022 copy-test-032 copy-test-048 copy-test-065 copy-test-083 copy-test-095 copy-test-012 copy-test-023 copy-test-033 copy-test-051 copy-test-073 copy-test-086 copy-test-098 copy-test-015 copy-test-024 copy-test-034 copy-test-052 copy-test-077 copy-test-087 copy-test-099 [root@GlusterFS-slave2 ~]# ll /opt/gluster/data/|wc -l 51 [root@GlusterFS-slave2 ~]# ls /opt/gluster/data/ copy-test-002 copy-test-014 copy-test-036 copy-test-047 copy-test-059 copy-test-069 copy-test-080 copy-test-097 copy-test-003 copy-test-018 copy-test-037 copy-test-049 copy-test-061 copy-test-070 copy-test-084 copy-test-005 copy-test-020 copy-test-040 copy-test-050 copy-test-062 copy-test-071 copy-test-085 copy-test-007 copy-test-025 copy-test-042 copy-test-053 copy-test-064 copy-test-072 copy-test-089 copy-test-009 copy-test-026 copy-test-043 copy-test-055 copy-test-066 copy-test-074 copy-test-091 copy-test-010 copy-test-027 copy-test-044 copy-test-056 copy-test-067 copy-test-075 copy-test-092 copy-test-013 copy-test-035 copy-test-045 copy-test-058 copy-test-068 copy-test-076 copy-test-096 [root@GlusterFS-slave3 ~]# ll /opt/gluster/data/|wc -l 51 [root@GlusterFS-slave3 ~]# ls /opt/gluster/data/ copy-test-002 copy-test-014 copy-test-036 copy-test-047 copy-test-059 copy-test-069 copy-test-080 copy-test-097 copy-test-003 copy-test-018 copy-test-037 copy-test-049 copy-test-061 copy-test-070 copy-test-084 copy-test-005 copy-test-020 copy-test-040 copy-test-050 copy-test-062 copy-test-071 copy-test-085 copy-test-007 copy-test-025 copy-test-042 copy-test-053 copy-test-064 copy-test-072 copy-test-089 copy-test-009 copy-test-026 copy-test-043 copy-test-055 copy-test-066 copy-test-074 copy-test-091 copy-test-010 copy-test-027 copy-test-044 copy-test-056 copy-test-067 copy-test-075 copy-test-092 copy-test-013 copy-test-035 copy-test-045 copy-test-058 copy-test-068 copy-test-076 copy-test-096

3)模拟brick故障

3.1)查看当前存储状态
在GlusterFS-slave3节点机器上操作
[root@GlusterFS-slave3 ~]# gluster volume status Status of volume: models Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick 192.168.10.239:/data/gluster 49152 Y 6016 Brick 192.168.10.212:/data/gluster 49152 Y 2910 Brick 192.168.10.204:/data/gluster 49153 Y 9030 Brick 192.168.10.220:/data/gluster 49153 Y 12363 NFS Server on localhost N/A N N/A Self-heal Daemon on localhost N/A Y 12382 Quota Daemon on localhost N/A Y 12389 NFS Server on 192.168.10.204 N/A N N/A Self-heal Daemon on 192.168.10.204 N/A Y 9049 Quota Daemon on 192.168.10.204 N/A Y 9056 NFS Server on GlusterFS-master N/A N N/A Self-heal Daemon on GlusterFS-master N/A Y 6037 Quota Daemon on GlusterFS-master N/A Y 6042 NFS Server on 192.168.10.212 N/A N N/A Self-heal Daemon on 192.168.10.212 N/A Y 2930 Quota Daemon on 192.168.10.212 N/A Y 2936 Task Status of Volume models ------------------------------------------------------------------------------ Task : Rebalance ID : f7bc799f-d8a8-488e-9c38-dc1f2c685a99 Status : completed 注:注意到Online项全部为"Y" 2)制造故障(注意这里模拟的是文件系统故障,假设物理硬盘没有问题或已经更换阵列中的硬盘) 在GlusterFS-slave3节点机器上操作 [root@GlusterFS-slave3 ~]# vim /etc/fstab //注释掉如下行 ...... #/dev/sdb1 /data xfs defaults 1 2 重启服务器 [root@GlusterFS-slave3 ~]# reboot 3)查看当前存储状态 [root@GlusterFS-slave3 ~]# gluster volume status Status of volume: models Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick 192.168.10.239:/data/gluster 49152 Y 6016 Brick 192.168.10.212:/data/gluster 49152 Y 2910 Brick 192.168.10.204:/data/gluster 49153 Y 9030 Brick 192.168.10.220:/data/gluster N/A N N/A NFS Server on localhost N/A N N/A Self-heal Daemon on localhost N/A Y 12382 Quota Daemon on localhost N/A Y 12389 NFS Server on GlusterFS-master N/A N N/A Self-heal Daemon on GlusterFS-master N/A Y 6037 Quota Daemon on GlusterFS-master N/A Y 6042 NFS Server on 192.168.10.204 N/A N N/A Self-heal Daemon on 192.168.10.204 N/A Y 9049 Quota Daemon on 192.168.10.204 N/A Y 9056 NFS Server on 192.168.10.212 N/A N N/A Self-heal Daemon on 192.168.10.212 N/A Y 2930 Quota Daemon on 192.168.10.212 N/A Y 2936 Task Status of Volume models ------------------------------------------------------------------------------ Task : Rebalance ID : f7bc799f-d8a8-488e-9c38-dc1f2c685a99 Status : completed 注意:发现GlusterFS-slave3节点(192.168.10.220)的Online项状态为"N"了! 4)恢复故障brick方法 4.1)结束故障brick的进程 如上通过"gluster volume status"命令,如果查看到状态Online项为"N"的GlusterFS-slave3节点存在PID号(不显示N/A),则应当使用"kill -15 pid"杀死它! 一般当Online项为"N"时就不显示pid号了。 4.2)创建新的数据目录(注意绝不可以与之前目录一样) [root@GlusterFS-slave3 ~]# mkfs.xfs -i size=1024 /dev/sdb1 [root@GlusterFS-slave3 ~]# vim /etc/fstab //去掉下面注释 ...... /dev/sdb1 /data xfs defaults 1 2 重新挂载文件系统: [root@GlusterFS-slave3 ~]# mount -a 增加新的数据存放文件夹(不可以与之前目录一样) [root@GlusterFS-slave3 ~]# mkdir -p /data/gluster1 4.3)查询故障节点的备份节点(GlusterFS-slave2)目录的扩展属性(使用"yum search getfattr"命令getfattr工具的安装途径) [root@GlusterFS-slave2 ~]# yum install -y attr.x86_64 [root@GlusterFS-slave2 ~]# getfattr -d -m. -e hex /data/gluster getfattr: Removing leading '/' from absolute path names # file: opt/gluster/data trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000007fff7f58ffffffff trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.size=0x0000000003c19000 trusted.glusterfs.volume-id=0xf1945b0b67d642029198639244ab0a6a 4.4)挂载卷并触发自愈 在客户端先卸载掉之前的挂载 [root@Client ~]# umount /data/gluster 然后重新挂载GlusterFS-slave3(其实挂载哪一个节点的都可以) [root@Client ~]# mount -t glusterfs 192.168.10.220:models /data/gluster/ 新建一个卷中不存在的目录并删除 [root@Client gfsmount]# mkdir testDir001 [root@Client gfsmount]# rm -rf testDir001 设置扩展属性触发自愈 [root@Client gfsmount]# setfattr -n trusted.non-existent-key -v abc /data/gluster [root@Client gfsmount]# setfattr -x trusted.non-existent-key /data/gluster 4.5)检查当前节点是否挂起xattrs 再次查询故障节点的备份节点(GlusterFS-slave2)目录的扩展属性 [root@GlusterFS-slave2 ~]# getfattr -d -m. -e hex /data/gluster getfattr: Removing leading '/' from absolute path names # file: opt/gluster/data trusted.afr.dirty=0x000000000000000000000000 trusted.afr.models-client-2=0x000000000000000000000000 trusted.afr.models-client-3=0x000000000000000200000002 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000007fff7f58ffffffff trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.size=0x0000000003c19000 trusted.glusterfs.volume-id=0xf1945b0b67d642029198639244ab0a6a 注意:留意第5行,表示xattrs已经将源标记为GlusterFS-slave3:/data/gluster 4.6)检查卷的状态是否显示需要替换 [root@GlusterFS-slave3 data]# gluster volume heal models info Brick GlusterFS-master:/data/gluster/ Number of entries: 0 Brick GlusterFS-slave:/data/gluster/ Number of entries: 0 Brick GlusterFS-slave2:/data/gluster/ / Number of entries: 1 Brick 192.168.10.220:/data/gluster Status: Transport endpoint is not connected 注:状态提示传输端点未连接(最后一行) 4.7)使用强制提交完成操作 [root@GlusterFS-slave3 data]# gluster volume replace-brick models 192.168.10.220:/data/gluster 192.168.10.220:/data/gluster1 commit force 提示如下表示正常完成: volume replace-brick: success: replace-brick commit force operation successful 注意:也可以将数据恢复到另外一台服务器,详细命令如下(192.168.10.230为新增的另一个glusterfs节点)(可选): # gluster peer probe 192.168.10.230 # gluster volume replace-brick models 192.168.10.220:/data/gluster 192.168.10.230:/data/gluster commit force 4.8)检查存储的在线状态 [root@GlusterFS-slave3 ~]# gluster volume status Status of volume: models Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick 192.168.10.239:/data/gluster1 49152 Y 6016 Brick 192.168.10.212:/data/gluster 49152 Y 2910 Brick 192.168.10.204:/data/gluster 49153 Y 9030 Brick 192.168.10.220:/data/gluster 49153 Y 12363 NFS Server on localhost N/A N N/A Self-heal Daemon on localhost N/A Y 12382 Quota Daemon on localhost N/A Y 12389 NFS Server on 192.168.10.204 N/A N N/A Self-heal Daemon on 192.168.10.204 N/A Y 9049 Quota Daemon on 192.168.10.204 N/A Y 9056 NFS Server on GlusterFS-master N/A N N/A Self-heal Daemon on GlusterFS-master N/A Y 6037 Quota Daemon on GlusterFS-master N/A Y 6042 NFS Server on 192.168.10.212 N/A N N/A Self-heal Daemon on 192.168.10.212 N/A Y 2930 Quota Daemon on 192.168.10.212 N/A Y 2936 Task Status of Volume models ------------------------------------------------------------------------------ Task : Rebalance ID : f7bc799f-d8a8-488e-9c38-dc1f2c685a99 Status : completed 另外,如果更换到其他服务器状态显示如下: [root@GlusterFS-slave ~]# gluster volume status Status of volume: models Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick 192.168.10.239:/data/gluster 49152 Y 6016 Brick 192.168.10.212:/data/gluster 49152 Y 2910 Brick 192.168.10.204:/data/gluster 49153 Y 9030 Brick 192.168.10.220:/data/gluster 49153 Y 12363 NFS Server on localhost N/A N N/A Self-heal Daemon on localhost N/A Y 12382 Quota Daemon on localhost N/A Y 12389 NFS Server on 192.168.10.204 N/A N N/A Self-heal Daemon on 192.168.10.204 N/A Y 9049 Quota Daemon on 192.168.10.204 N/A Y 9056 NFS Server on GlusterFS-master N/A N N/A Self-heal Daemon on GlusterFS-master N/A Y 6037 Quota Daemon on GlusterFS-master N/A Y 6042 NFS Server on 192.168.10.212 N/A N N/A Self-heal Daemon on 192.168.10.212 N/A Y 2930 Quota Daemon on 192.168.10.212 N/A Y 2936 Task Status of Volume models ------------------------------------------------------------------------------ Task : Rebalance ID : f7bc799f-d8a8-488e-9c38-dc1f2c685a99 Status : completed ====================================================================== 注意:上面是新建的独立分区,在这个独立分区上创建存储目录。如果不新建独立分区,直接在/分区上创建存储目录, 如文档http://www.cnblogs.com/kevingrace/p/8743812.html中的四个节点的存储目录是/opt/gluster/data。 如果是GlusterFS-slave3节点的这个存储目录/opt/gluster/data不小心误删除了。 最简单直接的方法可以是: 1)如上面操作,将删除的/opt/gluster/data目录重新mkdir新建出来 2)停止复制卷磁盘:gluster volume stop models 3)删除复制卷磁盘:gluster volume delete models 4)重新创建复制卷(副本卷)。卷名还是之前的models。这里选择4份副本。 # gluster volume create models replica 4 192.168.10.239:/opt/gluster/data 192.168.10.212:/opt/gluster/data 192.168.10.204:/opt/gluster/data 192.168.10.220:/opt/gluster/dataforce 5)删除复制卷磁盘后:gluster volume info 可以查看到四个节点的Bricks信息 6)客户端重新挂载glusterfs存储即可。 这样,发生故障的GlusterFS-slave3节点的存储目录下的数据就会跟另外一个replica组GlusterFS-master、GlusterFS-slave的数据一致。 由于GlusterFS-slave2是GlusterFS-slave3的备份节点,所以GlusterFS-slave2的存储目录下数据会涵盖所有节点的数据之和! 或者在上面第4步中重新创建副本卷的时候,还是和之前一样创建2个副本 # gluster volume create models replica 2 192.168.10.239:/opt/gluster/data 192.168.10.212:/opt/gluster/data force 然后将另外两个节点添加到复制卷里面 # gluster volume stop models # gluster volume status models # gluster volume add-brick models 192.168.10.204:/opt/gluster/data 192.168.10.220:/opt/gluster/data force

猜你喜欢

转载自www.cnblogs.com/zhengchunyuan/p/9560186.html
今日推荐