APPLIES TO:
Oracle Database - Enterprise Edition - Version 11.2.0.1 and later
Information in this document applies to any platform.
***Checked for relevance on 11-May-2014***
SYMPTOMS
2 node RAC - 11201 - Solaris10
ASM does not mount ASM disk groups because it finds insufficient disks. This prevents Oracle Clusterware's CRS to start as the OCR diks are stored in ASM:
~~~~~~~~~~~~~
...
Fri Oct 19 13:09:35 2012
NOTE: No asm libraries found in the system
ERROR: -5(Duplicate disk DATA:DATA_0000) <----------- Notice this entry as it tells that ASM discovers duplicate disk (1)
* instance_number obtained from CSS = 1, checking for the existence of node 0...
* node 0 does not exist. instance_number = 1
Starting ORACLE instance (normal)
...
Starting up:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options.
WARNING: using default parameter settings without any parameter file <---------------- not using a pfile/spfile (1)
...
Fri Oct 19 13:09:44 2012
SQL> ALTER DISKGROUP ALL MOUNT /* asm agent */
NOTE: Diskgroup used for Voting files is:
DATASOA
Diskgroup used for OCR is:DATA
Diskgroup used for OCR is:DATASOA
NOTE: cache registered group DATASOA number=1 incarn=0xa9eb204b
NOTE: cache registered group DATA number=2 incarn=0xaa0b204c
ERROR: no PST quorum in group: required 2, found 0 <------- ASM disk discovery problem
...
NOTE: cache ending mount (fail) of group DATASOA number=1 incarn=0xa9eb204b
...
ERROR: diskgroup DATASOA was not mounted <--------- Therefore ASM disk groups do not mount
...
NOTE: cache ending mount (fail) of group DATA number=2 incarn=0xaa0b204c
...
ERROR: diskgroup DATA was not mounted
...
WARNING: Disk Group DATASOA containing configured OCR is not mounted
WARNING: Disk Group DATASOA containing voting files is not mounted
WARNING: Disk Group DATA containing configured OCR is not mounted
ORA-15032: not all alterations performed
ORA-15017: diskgroup "DATA" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATA" <---- ASM discovers not sufficient disks
ORA-15017: diskgroup "DATASOA" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATASOA"
(1) These entries are similar to Note:1384692.1
CAUSE
However, the problem is not with ASM not finding all disks but finding duplicate disks (i,e, instead of ASM discovering only all 5 needed ASM disk, it is discovering twice that amount, or 10 disks).
When ASM discovers the same disk more than once, it ignores them (v$asm_disk.mount_status=IGNORED), so it is like it finds not sufficient disks.
Hence, the ASM alert logs shows the entry "ERROR: -5(Duplicate disk ..." at the instance starting up sequence and later gives the insufficient disk errors (ORA-15063).
.
SOLUTION
For this case (Solaris), the following has done to identify the duplicate devices:
* From previous successful attempts to mount all ASM disk groups from ASM alert log entries, the following asm disks are needed:
NOTE: cache registered group DATA number=2 incarn=0xaa0ac7b9
NOTE: Assigning number (1,0) to disk (/oracle_crs_install/shr_disk_links/ora_dbdsk1) <--- 4 disks for DATASOA
NOTE: Assigning number (1,2) to disk (/oracle_crs_install/shr_disk_links/ora_dbdsk2)
NOTE: Assigning number (1,3) to disk (/oracle_crs_install/shr_disk_links/ora_dbdsk3)
NOTE: Assigning number (1,1) to disk (/oracle_crs_install/shr_disk_links/ora_dbdsk4)
NOTE: Assigning number (2,0) to disk (/dev/rdsk/c0t60050768018105880000000000000023d0s6) <--- 1 disk for DATA
* The Grid-plug-and-play profile or <Grid Infrastructure Home>/gpnp/<hostname>/profiles/peer/profile.xml, lists the disk discovery string that will be used (along with the spfile that will be use):
* From the above, it was found that ASM was discovering It was found that
/dev/rdsk/c0t60050768018105880000000000000034d0s6 <-- dupl
/dev/rdsk/c0t60050768018105880000000000000033d0s6 <-- dupl
/dev/rdsk/c0t60050768018105880000000000000032d0s6 <-- dupl
/dev/rdsk/c0t60050768018105880000000000000038d0s6 <-- dupl
/dev/rdsk/c0t60050768018105880000000000000023d0s6
/oracle_crs_install/shr_disk_links/ora_dbdsk1
/oracle_crs_install/shr_disk_links/ora_dbdsk2
/oracle_crs_install/shr_disk_links/ora_dbdsk3
/oracle_crs_install/shr_disk_links/ora_dbdsk4
* Since the duplicate disks were identified and isolated, please change their device's owner:group or file premissions in a manner that ASM will no longer discover (access) these.
ASM can tolerate the discovery of each ASM disk only once (not more than once or it will be considered a 'duplicate disk' and hence we will likely encounter this scenario.
REFERENCES
ORA-15032: not all alterations performed
ORA-15017: diskgroup "TESTADG" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "TESTDG_1"