Failure scenario: AIX IBM storage replacement of the storage controller power module causes the storage dual controllers to lose power at the same time, and the disk lock occurs after the storage dual controllers restore power, and after IBM unlocks the disk
The RAC database cannot be started normally, and the startup error is reported: ORA-00600: internal error code, arguments: [2131], [33], [32], [], [], [], [], [], [], [ ], [], []
The recovery plan is determined: due to the huge amount of damaged database data, as much as 30T, and it is required to open the database as soon as possible, it can accept the loss of part of the data.
The database has a backup. Under negotiation, use the backup control file to restore the database and restore the inconsistency of the database.
Recovery process:
1. Pull the control file backup set from the tape library and restore it to the Oracle RAC database ASM storage
--At this time, trying to start the database will encounter an error message:
ORA-00600: internal error code, arguments: [3020], [718], [2828551], [3014338823], [], [], [], [], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 718, block# 2828551, file offset is unknown bytes)
2. Generate Oracle pfile and add the following three parameters
_allow_resetlogs_corruption=true #Skip the redo crash and allow resetlogs to open the database
undo_management=manual #Modify undo management to manual management
event="10513 trace name context forever,level 2" #Open 10513 event skip consistency verification
*.cluster_database=false
--At this time, trying to start the database will encounter an error message:
ORA-00600: internal error code, arguments: [2662], [3735], [1979553782], [3735], [1979584838], [20971664], [], [], [], [], [], []
3. After the Oracle database uses the backup control file to restore the database, it is necessary to advance the SCN of the control file
The current SCN of the database = 3735 .1979584838 = ( 3735 * 4294967296) + 1979584838 = 16043682435398
giga=16043682435398/1024/1024/1024=14941.843=14941
New SCN to be promoted=14942*1024*1024*1024=16043850334208
Oracle database 10g control file advancement can add parameters in pfile: _minimum_giga_scn;
ORACLE 11.2.0.4 SCN advance method oradebug
Steps to advance the database control file SCN
SQL> oradebug setmypid
Statement processed.
SQL> oradebug dumpvar sga kcsgscn_
kcslf kcsgscn_ [700000000019B70, 700000000019BA0) = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 07000000 00019850
SQL> oradebug poke 0x700000000019B70 8 16043850334208
BEFORE: [700000000019B70, 700000000019B78) = 00000000 00000000
AFTER: [700000000019B70, 700000000019B78) = 00000E97 80000000
SQL> oradebug dumpvar sga kcsgscn_
kcslf kcsgscn_ [700000000019B70, 700000000019BA0) = 00000E97 80000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 07000000 00019850
SQL>
--At this time, trying to start the database will encounter an error message:
ORA-00600: internal error code, arguments: [4193], [], [], [], [], [], [], [], [], [], [], []
The next processing of ORA600[4193] is the reconstruction of the undo tablespace of ORACLE rac, which needs to cooperate with the parameters: undo_management and _CORRUPTED_ROLLBACK_SEGMENTS.
After processing ORA600[4193], the database can be started and opened to the open state. At this time, it is best to perform a full database backup. If conditions permit,
Just rebuild the database and export the faulty database in a logical way to the new environment. Otherwise, you will encounter the following error messages from time to time during subsequent use:
ORA600[6122] and ORA600[KDSGRP1], the actual file is caused by the index logical bad block.