由于存储空间不足，下线的数据库需要把存储空间腾出来，关闭集群资源，主机工程师收回lun需要（包括ocr 和 voting data 磁盘组），新的应用需要上线需要新的数据库，新的hitach存储到位需要重新安装数据库，上次删除gi 和database软件都在，这次只需要重新配置即可。参考How to Deconfigure/Reconfigure(Rebuild OCR) or Deinstall Grid Infrastructure (文档 ID 1377349.1)
last node 删除配置
[root@qdcx-db2 /]# /oracle/app/crs/crs/install/rootcrs.pl -deconfig -force -verbose
Using configuration parameter file: /oracle/app/crs/crs/install/crsconfig_params
PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'qdcx-db2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'qdcx-db2'
CRS-2673: Attempting to stop 'ora.crf' on 'qdcx-db2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'qdcx-db2'
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'qdcx-db2'
CRS-2677: Stop of 'ora.cssdmonitor' on 'qdcx-db2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'qdcx-db2' succeeded
CRS-2677: Stop of 'ora.crf' on 'qdcx-db2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'qdcx-db2'
CRS-2677: Stop of 'ora.gipcd' on 'qdcx-db2' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'qdcx-db2'
CRS-2677: Stop of 'ora.gpnpd' on 'qdcx-db2' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'qdcx-db2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'qdcx-db2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
This may take several minutes. Please wait ...
0518-307 odmdelete: 1 objects deleted.
0518-307 odmdelete: 1 objects deleted.
0518-307 odmdelete: 1 objects deleted.
Successfully deconfigured Oracle clusterware stack on this node
first node
[root@qdcx-db1 /]# /oracle/app/crs/crs/install/rootcrs.pl -deconfig -force -verbose -lastnode
Using configuration parameter file: /oracle/app/crs/crs/install/crsconfig_params
CRS-5702: Resource 'ora.cssd' is already running on 'qdcx-db1'
CRS-4000: Command Start failed, or completed with errors.
CSS startup failed with return code 1
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Delete failed, or completed with errors.
CRS-2673: Attempting to stop 'ora.ctssd' on 'qdcx-db1'
CRS-2673: Attempting to stop 'ora.asm' on 'qdcx-db1'
CRS-2677: Stop of 'ora.ctssd' on 'qdcx-db1' succeeded
CRS-2677: Stop of 'ora.asm' on 'qdcx-db1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'qdcx-db1'
CRS-2677: Stop of 'ora.cssd' on 'qdcx-db1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'qdcx-db1'
CRS-2676: Start of 'ora.cssdmonitor' on 'qdcx-db1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'qdcx-db1'
CRS-2672: Attempting to start 'ora.diskmon' on 'qdcx-db1'
CRS-2676: Start of 'ora.diskmon' on 'qdcx-db1' succeeded
CRS-2676: Start of 'ora.cssd' on 'qdcx-db1' succeeded
This ASM diskgroup does not contain voting disks to be deleted
ASM de-configuration trace file location: /tmp/asmcadc_clean2018-07-16_04-34-19-PM.log
ASM Clean Configuration START
ASM Clean Configuration END
ASM with SID +ASM1 deleted successfully. Check /tmp/asmcadc_clean2018-07-16_04-34-19-PM.log for details.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'qdcx-db1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'qdcx-db1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'qdcx-db1'
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'qdcx-db1'
CRS-2677: Stop of 'ora.ctssd' on 'qdcx-db1' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'qdcx-db1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'qdcx-db1'
CRS-2677: Stop of 'ora.mdnsd' on 'qdcx-db1' succeeded
CRS-2677: Stop of 'ora.cssd' on 'qdcx-db1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'qdcx-db1'
CRS-2677: Stop of 'ora.gipcd' on 'qdcx-db1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'qdcx-db1'
CRS-2677: Stop of 'ora.gpnpd' on 'qdcx-db1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'qdcx-db1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
This may take several minutes. Please wait ...
0518-307 odmdelete: 1 objects deleted.
0518-307 odmdelete: 1 objects deleted.
0518-307 odmdelete: 1 objects deleted.
Successfully deconfigured Oracle clusterware stack on this node
重新配置
节点1
[root@qdcx-db1 /]# /oracle/app/crs/crs/install/rootcrs.pl -deconfig -force
Using configuration parameter file: /oracle/app/crs/crs/install/crsconfig_params
PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
This may take several minutes. Please wait ...
/oracle/app/crs/install/usm/V7x/powerpc/bin//udefacfsctl[75]: /oracle/app/crs/install/usm/V7x/powerpc/bin//udefacfsctl.bin: 0403-006 Execute permission denied.
ACFS-9361: Removing device 'acfsctl' failed with error code '32256'.
Unable to uninstall ADVM/ACFS
解决
chmod -R 755 /oracle/app/crs/install/usm/V7x/powerpc/bin
/oracle/app/crs/crs/install/rootcrs.pl -deconfig -force
/oracle/app/crs/root.sh
节点2
[root@qdcx-db2 /etc/oracle]# /oracle/app/crs/crs/install/rootcrs.pl -deconfig -force
Using configuration parameter file: /oracle/app/crs/crs/install/crsconfig_params
****Unable to retrieve Oracle Clusterware home.
Start Oracle Clusterware stack and try again.
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CLSU-00100: Operating System function: opendir failed with error data: 2
CLSU-00101: Operating System error message: No such file or directory
CLSU-00103: error location: scrsearch1
CLSU-00104: additional error information: cant open scr home dir scls_scr_getval
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
This may take several minutes. Please wait ...
Either /etc/oracle/olr.loc does not exist or is not readable
Make sure the file exists and it has read and execute access
Either /etc/oracle/olr.loc does not exist or is not readable
Make sure the file exists and it has read and execute access
Successfully deconfigured Oracle clusterware stack on this node
[root@qdcx-db2 /etc/oracle]# /oracle/app/crs/root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /oracle/app/crs
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /oracle/app/crs/crs/install/crsconfig_params
User ignored Prerequisites during installation
Installing Trace File Analyzer
User grid has the required capabilities to run CSSD in realtime mode
OLR initialization - successful
Adding Clusterware entries to inittab
CRS-2672: Attempting to start 'ora.mdnsd' on 'qdcx-db2'
CRS-2676: Start of 'ora.mdnsd' on 'qdcx-db2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'qdcx-db2'
CRS-2676: Start of 'ora.gpnpd' on 'qdcx-db2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'qdcx-db2'
CRS-2672: Attempting to start 'ora.gipcd' on 'qdcx-db2'
CRS-2676: Start of 'ora.cssdmonitor' on 'qdcx-db2' succeeded
CRS-2676: Start of 'ora.gipcd' on 'qdcx-db2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'qdcx-db2'
CRS-2672: Attempting to start 'ora.diskmon' on 'qdcx-db2'
CRS-2676: Start of 'ora.diskmon' on 'qdcx-db2' succeeded
CRS-2676: Start of 'ora.cssd' on 'qdcx-db2' succeeded
Disk Group ocr creation failed with the following message:
ORA-15018: diskgroup cannot be created
ORA-15031: disk specification '/dev/rhdisk6' matches no disks
ORA-15025: could not open disk "/dev/rhdisk6"
ORA-27041: unable to open file
Configuration of ASM ... failed
see asmca logs at /oracle/app/grid/cfgtoollogs/asmca for details
Did not succssfully configure and start ASM at /oracle/app/crs/crs/install/crsconfig_lib.pm line 6912.
/oracle/app/crs/perl/bin/perl -I/oracle/app/crs/perl/lib -I/oracle/app/crs/crs/install /oracle/app/crs/crs/install/rootcrs.pl execution failed
查看hdisk6 是否同一个lun
[root@qdcx-db1 /]# lscfg -vpl hdisk6
hdisk6 U5802.001.RCH5723-P1-C2-T1-W50060E8016609092-L4000000000000 Hitachi Disk Array (Fibre)
Manufacturer................HITACHI
Machine Type and Model......OPEN-V
Part Number.................. .....
ROS Level and ID............37303036
Serial Number...............50 16090
EC Level....................
FRU Number..................
Device Specific.(Z0)........00000332EF000002
Device Specific.(Z1)........0B37 AC ....
Device Specific.(Z2)............
Device Specific.(Z3).........
Device Specific.(Z4)............
Device Specific.(Z5)........
Device Specific.(Z6)........
PLATFORM SPECIFIC
Name: disk
Node: disk
Device Type: block
[root@qdcx-db2 /]# lscfg -vpl hdisk6
hdisk6 U5802.001.RCH4330-P1-C2-T1-W50060E8016609092-L4000000000000 Hitachi Disk Array (Fibre)
Manufacturer................HITACHI
Machine Type and Model......OPEN-V
Part Number.................. .....
ROS Level and ID............37303036
Serial Number...............50 16090
EC Level....................
FRU Number..................
Device Specific.(Z0)........00000332EF000002
Device Specific.(Z1)........0B37 AC ....
Device Specific.(Z2)............
Device Specific.(Z3).........
Device Specific.(Z4)............
Device Specific.(Z5)........
Device Specific.(Z6)........
PLATFORM SPECIFIC
Name: disk
Node: disk
Device Type: block
从Device Specific.(Z1)........0B37 AC可知是同一个lun
查看hdisk6磁盘属性
[root@qdcx-db2 /]# ls -l /dev/rhdisk6
crw-rw---- 1 grid asmadmin 15, 5 Jul 16 14:39 /dev/rhdisk6
[+ASM2]@qdcx-db2[/home/grid]$lsattr -E -l hdisk6|grep reserve_policy
reserve_policy PR_exclusive Reserve Policy True
磁盘属性reserve_policy不对，系统管理员忘记修改，运行脚本修改
[root@qdcx-db2 /tmp]# cat chmodall
begin=18
end=22
i=$begin
while [ $i -le $end ];
do
chmod 660 /dev/rhdisk$i
chown grid:asmadmin /dev/rhdisk$i
chdev -l hdisk$i -a reserve_policy=no_reserve
chdev -l hdisk$i -a queue_depth=8
i=`expr $i + 1`
done
ls -l /dev/rhdisk*
继续运行root.sh 这次没报错