模拟在线更换硬盘

•  
1. 踢盘；
1. 确定故障磁盘；
2. 剔出磁盘组z；
3. 重平衡完毕；
1. 更换；
1. 存储节点上从qlink中剔除故障磁盘；
2. 将坏盘从操作系统detach掉；
3. 点亮故障硬盘槽位指示灯；
4. 更换；
5. attach上新盘；
6. 添加新盘到原有qlink位置；
1. 加回磁盘组；
1. 计算节点确定原有故障盘正常；
2. 加回磁盘组；
•  
• ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
•  
• 1.确定损坏磁盘在ASM磁盘组中的磁盘名
• set line 1000
• col name for a30;
• col failgroup for a50;
• col path for a50;
• select b.name,a.name,a.failgroup,a.path from v$asm_disk a,v$asm_diskgroup b where a.group_number = b.group_number order by 2;
•  
• 确定磁盘组信息，磁盘路径，把插到的坏的磁盘组那一行全部复制出来：
• NAME                   NAME                  FAILGROUP                      PATH
• SSDDG                   SSDDG_0057              SDDDGSTO3                      /dev/qdata/mpath-1s03.3263.01.P3B00S21
•  
• 2.从ASM中删除该ssd盘
• alter diskgroup <dgname> drop disk <disk_name> force;
•  
• alter diskgroup ssddg drop disk ssddg_0057 force;
•  
• 3.重平衡数据
• alter diskgroup <dgname> rebalance power 11;
•  
• alter diskgroup ssddg rebalance power 11;
•  
• 4.确认重平衡进度
• asmcmd lsop    该命令返回无结果，则可进行下一步
• crsctl status res -t   //确认集群状态
•  
•  
• 5.存储节点上qlink中删除该盘
• qdatamgr qlink del_lun -p 3261 -t 1 -b /dev/qdisk/Pxxx
• qdatamgr qlink show -t   //确认存储节点的target
•  
• qdatamgr qlink show -t
• qdatamgr qlink del_lun -p 3263 -t 1 -b /dev/qdisk/P3B00S21
• qdatamgr qlink show -t
•  
• 计算节点查看该盘的状态：
• qdatamgr qlink show -c | grep -P -A7 P3B00S21
•  
• 6.将坏盘从操作系统detach
• qdatamgr media detach_disk -s Pxxxx
•  
• qdatamgr media detach_disk -s P3B00S21
•  
•  
• 7.将磁盘槽位灯点亮，确定磁盘的物理位置，更换磁盘
• qdatamgr media lighton -s Pxxxx
•  
• qdatamgr qlink show -t
• qdatamgr media lighton -s P3B00S21
•  
•  
• 8.新的SSD重新插入
• 9.初始化新的磁盘
• qdatamgr media attach_disk -s Pxxxx
•  
• qdatamgr media attach_disk -s P3B00S21
•  
•  
• 10.qlink中添加新盘
• qdatamgr qlink add_lun -p 3261 -t 1 -b /dev/qdisk/Pxxxx
•  
• qdatamgr qlink add_lun -p 3263 -t 1 -b /dev/qdisk/P3B00S21
•  
•  
• 11.qlink中确认加载的新盘
• qdatamgr qlink show -c | grep -P -A7 Pxxxx
• qdatamgr qlink load -r     //计算节点重新load
•  
• qdatamgr qlink show -c | grep -P -A7 P3B00S21
•  
•  
• 12.ASM中重新加入新盘
• alter diskgroup ssddg add failgroup datasto1 disk '/dev/qdata/mpath-1s01.3261.01.P0B00S02' force;
•  
• alter diskgroup ssddg add failgroup ssddgsto3 disk '/dev/qdata/mpath-1s03.3263.01.P3B00S21' force;
• alter diskgroup ssddg rebalance power 11;
•  
• select b.name,a.name,a.failgroup,a.path from v$asm_disk a,v$asm_diskgroup b where a.group_number = b.group_number order by 2;
•  
•  
• +++++++++++++++++++++++++++++++++++++++++++++++++++++==
•  
• 确定磁盘位置：
• set line 1000
• col name for a30;
• col failgroup for a50;
• col path for a50;
• select b.name,a.name,a.failgroup,a.path from v$asm_disk a,v$asm_diskgroup b where a.group_number = b.group_number order by 2;
• 
•  
•  
• 
•  
•  
• 
•  
•  
• 删除磁盘后，查看asm磁盘组的状态：
• su – grid
• asmcmd lsdg
• asmcmd lsop
• 
• 节点2上的磁盘组状态：
• 
• 重平衡数据：
• 
•  
• 
•  
• 
•  
• 
•  
• 
•  
• 节点2：
• 
•  
• 
• 到对应的存储节点确定磁盘位置：
• qdatamgr media show
• 
• 从存储节点上删除该磁盘：
•  
• 
• ++++++++++++++++++++++  额外信息  ++++++++++++++++++++++++++++
• 当存储节点删除掉硬盘以后，从计算节点查看：
• 
•  
• 当存储节点添加完硬盘，假如到原来的qlink中后，从计算节点查看，qlink会自动挂载：
• 
• +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
• 
• 将坏盘从操作系统detach掉：
• 
•  
• 
• 将硬盘槽位灯点亮，确定磁盘位置，更换磁盘：
• 
•  
• 
•  
• 
• 更换上新的磁盘后，硬盘槽位灯恢复正常；
•  
•  
•  
• 将新的盘attach上：
• 
•  
• 
•  
•  
•  
• 将新的磁盘添加到qlink中：
• 
•  
• 
•  
• 
•  
• 
• 将磁盘加到磁盘组：
• 
•  
• 
•  
• 
•  
• 
•  
• 
•  
• 
•  
• 
•  
•  
•  
•  
•  
•  
•  
•  
•  
•  
•  
•  
•  
•  
•  
•  
• 相关日志输出：
• 
•  
• 
•  
• 
•  
• 
•  
• 
•  
• 
•  
• 
•  
• 
•  
• ++++++++++++++++++++++++++++++
• 在计算节点2上操作更换硬盘的操作，grid报的完整日志信息如下：
• Thu Nov 30 11:53:00 2017
• SQL> alter diskgroup ssddg drop disk ssddg_0027 force
• Thu Nov 30 11:53:00 2017
• NOTE: GroupBlock outside rolling migration privileged region
• NOTE: requesting all-instance membership refresh for group=4
• NOTE: process _user86851_+asm2 (86851) initiating offline of disk 27.3915936961 (SSDDG_0027) with mask 0x7e in group 4 (SSDDG) without client assisting
• NOTE: initiating PST update: grp 4 (SSDDG), dsk = 27/0xe96874c1, mask = 0x6a, op = clear
• Thu Nov 30 11:53:00 2017
• GMON updating disk modes for group 4 at 26 for pid 33, osid 86851
• Thu Nov 30 11:53:00 2017
• NOTE: PST update grp = 4 completed successfully
• NOTE: initiating PST update: grp 4 (SSDDG), dsk = 27/0xe96874c1, mask = 0x7e, op = clear
• Thu Nov 30 11:53:00 2017
• GMON updating disk modes for group 4 at 27 for pid 33, osid 86851
• Thu Nov 30 11:53:00 2017
• NOTE: cache closing disk 27 of grp 4: SSDDG_0027
• Thu Nov 30 11:53:00 2017
• NOTE: PST update grp = 4 completed successfully
• Thu Nov 30 11:53:00 2017
• GMON updating for reconfiguration, group 4 at 28 for pid 33, osid 86851
• Thu Nov 30 11:53:00 2017
• NOTE: cache closing disk 27 of grp 4: (not open) SSDDG_0027
• Thu Nov 30 11:53:00 2017
• NOTE: group 4 PST updated.
• Thu Nov 30 11:53:00 2017
• NOTE: membership refresh pending for group 4/0x605884d1 (SSDDG)
• Thu Nov 30 11:53:00 2017
• GMON querying group 4 at 29 for pid 22, osid 21402
• Thu Nov 30 11:53:00 2017
• NOTE: cache closing disk 27 of grp 4: (not open) _DROPPED_0027_SSDDG
• Thu Nov 30 11:53:00 2017
• SUCCESS: refreshed membership for 4/0x605884d1 (SSDDG)
• Thu Nov 30 11:53:00 2017
• SUCCESS: alter diskgroup ssddg drop disk ssddg_0027 force
• NOTE: Attempting voting file refresh on diskgroup SSDDG
• NOTE: Refresh completed on diskgroup SSDDG. No voting file found.
• Thu Nov 30 11:53:01 2017
• NOTE: starting rebalance of group 4/0x605884d1 (SSDDG) at power 1
• Starting background process ARB0
• Thu Nov 30 11:53:01 2017
• ARB0 started with pid=37, OS id=88936
• NOTE: assigning ARB0 to group 4/0x605884d1 (SSDDG) with 1 parallel I/O
• Thu Nov 30 11:53:44 2017
• SQL> alter diskgroup ssddg rebalance power 11
• Thu Nov 30 11:53:44 2017
• NOTE: cache closing disk 27 of grp 4: (not open) _DROPPED_0027_SSDDG
• Thu Nov 30 11:53:44 2017
• NOTE: GroupBlock outside rolling migration privileged region
• Thu Nov 30 11:53:44 2017
• NOTE: stopping process ARB0
• Thu Nov 30 11:53:46 2017
• NOTE: rebalance interrupted for group 4/0x605884d1 (SSDDG)
• Thu Nov 30 11:53:46 2017
• NOTE: requesting all-instance membership refresh for group=4
• Thu Nov 30 11:53:46 2017
• GMON updating for reconfiguration, group 4 at 31 for pid 33, osid 86851
• Thu Nov 30 11:53:46 2017
• NOTE: cache closing disk 27 of grp 4: (not open) _DROPPED_0027_SSDDG
• Thu Nov 30 11:53:46 2017
• NOTE: group 4 PST updated.
• Thu Nov 30 11:53:46 2017
• NOTE: membership refresh pending for group 4/0x605884d1 (SSDDG)
• Thu Nov 30 11:53:46 2017
• GMON querying group 4 at 32 for pid 22, osid 21402
• Thu Nov 30 11:53:46 2017
• NOTE: cache closing disk 27 of grp 4: (not open) _DROPPED_0027_SSDDG
• Thu Nov 30 11:53:46 2017
• SUCCESS: refreshed membership for 4/0x605884d1 (SSDDG)
• Thu Nov 30 11:53:46 2017
• SUCCESS: alter diskgroup ssddg rebalance power 11
• NOTE: Attempting voting file refresh on diskgroup SSDDG
• NOTE: Refresh completed on diskgroup SSDDG. No voting file found.
• Thu Nov 30 11:53:49 2017
• NOTE: starting rebalance of group 4/0x605884d1 (SSDDG) at power 11
• Starting background process ARB0
• Thu Nov 30 11:53:49 2017
• ARB0 started with pid=37, OS id=89644
• NOTE: assigning ARB0 to group 4/0x605884d1 (SSDDG) with 11 parallel I/Os
• Thu Nov 30 11:56:56 2017
• NOTE: GroupBlock outside rolling migration privileged region
• NOTE: requesting all-instance membership refresh for group=4
• Thu Nov 30 11:56:57 2017
• GMON updating for reconfiguration, group 4 at 33 for pid 38, osid 91860
• Thu Nov 30 11:56:57 2017
• NOTE: cache closing disk 27 of grp 4: (not open) _DROPPED_0027_SSDDG
• Thu Nov 30 11:56:57 2017
• NOTE: group 4 PST updated.
• SUCCESS: grp 4 disk _DROPPED_0027_SSDDG going offline
• Thu Nov 30 11:56:57 2017
• GMON updating for reconfiguration, group 4 at 34 for pid 38, osid 91860
• Thu Nov 30 11:56:57 2017
• NOTE: cache closing disk 27 of grp 4: (not open) _DROPPED_0027_SSDDG
• Thu Nov 30 11:56:57 2017
• NOTE: group 4 PST updated.
• Thu Nov 30 11:56:57 2017
• NOTE: membership refresh pending for group 4/0x605884d1 (SSDDG)
• Thu Nov 30 11:56:57 2017
• GMON querying group 4 at 35 for pid 22, osid 21402
• GMON querying group 4 at 36 for pid 22, osid 21402
• Thu Nov 30 11:56:57 2017
• NOTE: Disk _DROPPED_0027_SSDDG in mode 0x0 marked for de-assignment
• SUCCESS: refreshed membership for 4/0x605884d1 (SSDDG)
• NOTE: Attempting voting file refresh on diskgroup SSDDG
• NOTE: Refresh completed on diskgroup SSDDG. No voting file found.
• Thu Nov 30 11:57:25 2017
• NOTE: stopping process ARB0
• Thu Nov 30 11:57:26 2017
• SUCCESS: rebalance completed for group 4/0x605884d1 (SSDDG)
• Thu Nov 30 12:16:24 2017
• SQL> alter diskgroup ssddg add failgroup ssddgsto2 disk '/dev/qdata/mpath-1s02.3262.01.P0B00S10' force
• Thu Nov 30 12:16:24 2017
• NOTE: GroupBlock outside rolling migration privileged region
• NOTE: Assigning number (4,27) to disk (/dev/qdata/mpath-1s02.3262.01.P0B00S10)
• NOTE: requesting all-instance membership refresh for group=4
• NOTE: Disk 27 in group 4 is assigned fgnum=4
• NOTE: discarding redo for group 4 disk 27
• NOTE: initializing header (replicated) on grp 4 disk SSDDG_0027
• NOTE: initializing header on grp 4 disk SSDDG_0027
• NOTE: requesting all-instance disk validation for group=4
• Thu Nov 30 12:16:25 2017
• NOTE: skipping rediscovery for group 4/0x605884d1 (SSDDG) on local instance.
• Thu Nov 30 12:16:25 2017
• NOTE: requesting all-instance disk validation for group=4
• Thu Nov 30 12:16:25 2017
• NOTE: skipping rediscovery for group 4/0x605884d1 (SSDDG) on local instance.
• Thu Nov 30 12:16:25 2017
• GMON updating for reconfiguration, group 4 at 38 for pid 33, osid 86851
• Thu Nov 30 12:16:25 2017
• NOTE: group 4 PST updated.
• Thu Nov 30 12:16:25 2017
• NOTE: membership refresh pending for group 4/0x605884d1 (SSDDG)
• Thu Nov 30 12:16:25 2017
• GMON querying group 4 at 39 for pid 22, osid 21402
• NOTE: cache opening disk 27 of grp 4: SSDDG_0027 path:/dev/qdata/mpath-1s02.3262.01.P0B00S10
• Thu Nov 30 12:16:27 2017
• NOTE: Attempting voting file refresh on diskgroup SSDDG
• NOTE: Refresh completed on diskgroup SSDDG. No voting file found.
• Thu Nov 30 12:16:27 2017
• GMON querying group 4 at 40 for pid 22, osid 21402
• Thu Nov 30 12:16:27 2017
• SUCCESS: refreshed membership for 4/0x605884d1 (SSDDG)
• Thu Nov 30 12:16:27 2017
• SUCCESS: alter diskgroup ssddg add failgroup ssddgsto2 disk '/dev/qdata/mpath-1s02.3262.01.P0B00S10' force
• Thu Nov 30 12:16:27 2017
• NOTE: starting rebalance of group 4/0x605884d1 (SSDDG) at power 1
• Starting background process ARB0
• Thu Nov 30 12:16:27 2017
• ARB0 started with pid=36, OS id=104046
• NOTE: assigning ARB0 to group 4/0x605884d1 (SSDDG) with 1 parallel I/O
• NOTE: Attempting voting file refresh on diskgroup SSDDG
• NOTE: Refresh completed on diskgroup SSDDG. No voting file found.
• Thu Nov 30 12:17:39 2017
• SQL> alter diskgroup ssddg rebalance power 11
• Thu Nov 30 12:17:39 2017
• NOTE: GroupBlock outside rolling migration privileged region
• Thu Nov 30 12:17:39 2017
• NOTE: stopping process ARB0
• Thu Nov 30 12:17:39 2017
• NOTE: rebalance interrupted for group 4/0x605884d1 (SSDDG)
• Thu Nov 30 12:17:39 2017
• NOTE: requesting all-instance membership refresh for group=4
• Thu Nov 30 12:17:39 2017
• GMON updating for reconfiguration, group 4 at 42 for pid 33, osid 86851
• Thu Nov 30 12:17:39 2017
• NOTE: group 4 PST updated.
• Thu Nov 30 12:17:39 2017
• NOTE: membership refresh pending for group 4/0x605884d1 (SSDDG)
• Thu Nov 30 12:17:39 2017
• GMON querying group 4 at 43 for pid 22, osid 21402
• Thu Nov 30 12:17:39 2017
• SUCCESS: refreshed membership for 4/0x605884d1 (SSDDG)
• Thu Nov 30 12:17:39 2017
• SUCCESS: alter diskgroup ssddg rebalance power 11
• NOTE: Attempting voting file refresh on diskgroup SSDDG
• NOTE: Refresh completed on diskgroup SSDDG. No voting file found.
• Thu Nov 30 12:17:42 2017
• NOTE: starting rebalance of group 4/0x605884d1 (SSDDG) at power 11
• Starting background process ARB0
• Thu Nov 30 12:17:42 2017
• ARB0 started with pid=36, OS id=104813
• NOTE: assigning ARB0 to group 4/0x605884d1 (SSDDG) with 11 parallel I/Os
• Thu Nov 30 12:21:55 2017
• NOTE: GroupBlock outside rolling migration privileged region
• NOTE: requesting all-instance membership refresh for group=4
• Thu Nov 30 12:21:57 2017
• GMON updating for reconfiguration, group 4 at 44 for pid 37, osid 108281
• Thu Nov 30 12:21:57 2017
• NOTE: group 4 PST updated.
• Thu Nov 30 12:21:57 2017
• NOTE: membership refresh pending for group 4/0x605884d1 (SSDDG)
• Thu Nov 30 12:21:57 2017
• GMON querying group 4 at 45 for pid 22, osid 21402
• Thu Nov 30 12:21:57 2017
• SUCCESS: refreshed membership for 4/0x605884d1 (SSDDG)
• NOTE: Attempting voting file refresh on diskgroup SSDDG
• NOTE: Refresh completed on diskgroup SSDDG. No voting file found.
•  
•  
•  
• Thu Nov 30 12:24:10 2017
• NOTE: stopping process ARB0
• Thu Nov 30 12:24:13 2017
• SUCCESS: rebalance completed for group 4/0x605884d1 (SSDDG)
•  
• ++++++++++++++++++++++++++
• 节点1上报的完整日志信息：
• Thu Nov 30 11:52:59 2017
• NOTE: updating disk modes to 0x15 from 0x7f for disk 27 (SSDDG_0027) in group 4 (SSDDG): lflags 0x4   
• NOTE: updating disk modes to 0x1 from 0x15 for disk 27 (SSDDG_0027) in group 4 (SSDDG): lflags 0x4   
• Thu Nov 30 11:52:59 2017
• NOTE: cache closing disk 27 of grp 4: SSDDG_0027
• Thu Nov 30 11:52:59 2017
• NOTE: cache closing disk 27 of grp 4: (not open) SSDDG_0027
• Thu Nov 30 11:52:59 2017
• NOTE: membership refresh pending for group 4/0x60589fce (SSDDG)
• Thu Nov 30 11:52:59 2017
• GMON querying group 4 at 34 for pid 22, osid 21900
• Thu Nov 30 11:52:59 2017
• NOTE: cache closing disk 27 of grp 4: (not open) SSDDG_0027
• Thu Nov 30 11:52:59 2017
• SUCCESS: refreshed membership for 4/0x60589fce (SSDDG)
• NOTE: Attempting voting file refresh on diskgroup SSDDG
• NOTE: Refresh completed on diskgroup SSDDG. No voting file found.
• Thu Nov 30 11:53:46 2017
• NOTE: cache closing disk 27 of grp 4: (not open) _DROPPED_0027_SSDDG
• Thu Nov 30 11:53:46 2017
• NOTE: membership refresh pending for group 4/0x60589fce (SSDDG)
• Thu Nov 30 11:53:46 2017
• GMON querying group 4 at 35 for pid 22, osid 21900
• Thu Nov 30 11:53:46 2017
• NOTE: cache closing disk 27 of grp 4: (not open) _DROPPED_0027_SSDDG
• Thu Nov 30 11:53:46 2017
• SUCCESS: refreshed membership for 4/0x60589fce (SSDDG)
• NOTE: Attempting voting file refresh on diskgroup SSDDG
• NOTE: Refresh completed on diskgroup SSDDG. No voting file found.
•  
•  
• Thu Nov 30 11:56:56 2017
• NOTE: cache closing disk 27 of grp 4: (not open) _DROPPED_0027_SSDDG
• Thu Nov 30 11:56:57 2017
• NOTE: membership refresh pending for group 4/0x60589fce (SSDDG)
• Thu Nov 30 11:56:57 2017
• GMON querying group 4 at 36 for pid 22, osid 21900
• GMON querying group 4 at 37 for pid 22, osid 21900
• Thu Nov 30 11:56:57 2017
• NOTE: Disk _DROPPED_0027_SSDDG in mode 0x0 marked for de-assignment
• SUCCESS: refreshed membership for 4/0x60589fce (SSDDG)
• NOTE: Attempting voting file refresh on diskgroup SSDDG
• NOTE: Refresh completed on diskgroup SSDDG. No voting file found.
• Thu Nov 30 12:16:24 2017
• NOTE: disk validation pending for 1 disk in group 4/0x60589fce (SSDDG)
• NOTE: completed disk validation for 4/0x60589fce (SSDDG)
• NOTE: disk validation pending for 1 disk in group 4/0x60589fce (SSDDG)
• NOTE: Assigning number (4,27) to disk (/dev/qdata/mpath-1s02.3262.01.P0B00S10)
• NOTE: completed disk validation for 4/0x60589fce (SSDDG)
• NOTE: membership refresh pending for group 4/0x60589fce (SSDDG)
• Thu Nov 30 12:16:24 2017
• GMON querying group 4 at 38 for pid 22, osid 21900
• NOTE: cache opening disk 27 of grp 4: SSDDG_0027 path:/dev/qdata/mpath-1s02.3262.01.P0B00S10
• GMON querying group 4 at 39 for pid 22, osid 21900
• Thu Nov 30 12:16:24 2017
• SUCCESS: refreshed membership for 4/0x60589fce (SSDDG)
• NOTE: Attempting voting file refresh on diskgroup SSDDG
• NOTE: Refresh completed on diskgroup SSDDG. No voting file found.
• Thu Nov 30 12:17:38 2017
• NOTE: membership refresh pending for group 4/0x60589fce (SSDDG)
• Thu Nov 30 12:17:38 2017
• GMON querying group 4 at 40 for pid 22, osid 21900
• Thu Nov 30 12:17:39 2017
• SUCCESS: refreshed membership for 4/0x60589fce (SSDDG)
• NOTE: Attempting voting file refresh on diskgroup SSDDG
• NOTE: Refresh completed on diskgroup SSDDG. No voting file found.
• Thu Nov 30 12:21:56 2017
• NOTE: membership refresh pending for group 4/0x60589fce (SSDDG)
• Thu Nov 30 12:21:56 2017
• GMON querying group 4 at 41 for pid 22, osid 21900
• Thu Nov 30 12:21:56 2017
• SUCCESS: refreshed membership for 4/0x60589fce (SSDDG)
• NOTE: Attempting voting file refresh on diskgroup SSDDG
• NOTE: Refresh completed on diskgroup SSDDG. No voting file found.
• +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++