一、运行环境
业 务：某农信数据库
操作系统：HPUNIX
数据库版本：11.2.0.3
数据库配置：RAC
数据文件存储方式：asm
二、故障描述
生产环境主备库的数据文件存储方式为ASM，发现备库的归档文件路径磁盘空间不足，删除部分归档日志释放一些空间，但是经过排查发现重启备库后没有开启DG恢复模式，只是在接受主库传过来的归档日志，并没有在备端应用，导致备库删除归档日志文件脚本执行失败，造成归档空间不足。由于已经删除一部分没应用到备库的归档文件，导致灾备dg同步中断。
三、处理过程
由于归档丢失，备库恢复同步方式由如下两种：
1.通过主库全库备份来恢复备库。
2.通过基于SCN备份主库来恢复备库
第一种方式适合数据量较少的库，这里选择第二种方式，步骤如下：
- 备库确认丢失归档信息
>select * from v$archive_gap
THREAD# LOW_SEQUENCE# HIGH_SEQUENCE#
---------- ------------- --------------
2 27893 28555
>select process, client_process,sequence#,status from v$managed_standby
PROCESS CLIENT_P SEQUENCE# STATUS
--------- -------- ---------- ------------
ARCH ARCH 30775 CLOSING
ARCH ARCH 29360 CLOSING
ARCH ARCH 29361 CLOSING
ARCH ARCH 30771 CLOSING
ARCH ARCH 29357 CLOSING
ARCH ARCH 30772 CLOSING
ARCH ARCH 30773 CLOSING
ARCH ARCH 29358 CLOSING
ARCH ARCH 30774 CLOSING
ARCH ARCH 29359 CLOSING
RFS UNKNOWN 0 IDLE
RFS LGWR 29362 IDLE
RFS N/A 0 IDLE
RFS UNKNOWN 0 IDLE
MRP0 N/A 27893 WAIT_FOR_GAP
RFS UNKNOWN 0 IDLE
RFS LGWR 30776 IDLE
RFS N/A 0 IDLE
>select name, values from v$dataguard_stats
NAME VALUE
-------------------------------- ------------------------------
apply finish time +00 16:52:12.9
apply lag +94 04:21:02
estimated startup time 22
standby has been open N
transport lag +00 00:00:07
在备库上执行：
- 备库确认最小scn号
>alter database recover managed standby database cancel；--停止日志应用服务
>select current_scn from v$DATABASE;
CURRENT_SCN
----------------
39597750709
>select min(f.checkpoint_change#) from v$datafile_header f,v$datafile d
where f.file#=d.file# and d.enabled !='READ ONLY';
CURRENT_SCN
----------------
39597657793
分别查出了备库当前的scn和文件头部的最小scn，采用两者最小的scn在主库进行增量备份
>alter system set standby_file_management='AUTO' SCOPE=BOTH; --在备库端开启数据文件自动创建参数
在主库上执行：
select file#,to_char(creation_time,'yyyy-mm-dd hh24:mi:ss' ) creation_time,status,last_time,name from v$datafile where creation_change#>39597657793
FILE# CREATION_TIME STATUS LAST_TIME NAME
---------- -------------------- -------------- -------------------- ----
151 2017-07-13 online +DATA/ebsback/datafile/apps_pts_cux6_02.dbf
152 2017-08-18 online +DATA/ebsback/datafile/cuxd_05
153 2017-08-18 online +DATA/ebsback/datafile/apps_pts_cux2_03
154 2017-08-18 online +DATA/ebsback/datafile/apps_pts_cux3_03
155 2017-08-18 online +DATA/ebsback/datafile/apps_pts_cux0.03
通过查询得知在备库最下的scn之后，主库又新增了5个数据文件 151～155在备库上执行：
select file#,name from v$datafile; --备库并未查得主库新增的数据文件
- 主库通过基于scn方式rman增量备份数据，备份主库的控制文件，并且把两个备份集scp到备库
RMAN>backup as compressed backupset incremental from scn 39597657793
database format'/备份文件_U%'
SQL>alter database create standby controlfile as '/控制文件备份.ctl'
scp 两个备份文件到备库
- 备库恢复数据库
SQL>startup mount --备库首先启动至mount状态
RMAN>backup current controlfile format '/beifenwenjian.ctl' --备份当前备库的控制文件
RMAN>catalog start with ‘/主库备份文件的所在备库的路径’
RMAN>recover database noredo;
- ⭐️备库使用旧的控制文件采集数据文件路径信息
备库数据文件恢复完成后，使用当前的备库控制文件查询数据库内的数据文件个数，查询结果还是150个
- 备库恢复新的控制文件
SQL>startup nomount --启动数据库到nomount状态
RMAN>restore controlfile from '/主库控制文件的备份文件'。
SQL>alter database mount
- 备库使用catalog datafilecopy 命令扫描数据文件
为使备库新恢复的控制文件能够有原来备库上的数据文件信息（由于启动了OMF，主库的数据文件名是不一致的），所以，需要使用⭐️步骤中收集的数据文件信息来编辑下面脚本中catalog datafilecopy 后面的信息，将备库的数据文件和名称注册到新的控制文件中。
⚠️OMF(oracle managed files) 能简化管理员的管理工作，不要指定文件的名字、大小、路径。其名字，大小，路径由oracle自动分配，可以通过DB_CREATE_FILE_DEST参数来获得。当参数值为空时表示未启动omf，若是设置为文件夹的名 则是开启omf。当主备库的数据文件存储方式为asm，并且ASM管理数据文件名称的机制为omf时，会导致主备库的文件名称不一致。
rman target /
catalog datafilecopy'+DATA/ebsback/datafile/system.285.837193785';
catalog datafilecopy'+DATA/ebsback/datafile/system.331.837194971';
catalog datafilecopy'+DATA/ebsback/datafile/system.317.837194443';
catalog datafilecopy'+DATA/ebsback/datafile/system.293.837193791';
catalog datafilecopy'+DATA/ebsback/datafile/system.332.837194971';
catalog datafilecopy'+DATA/ebsback/datafile/system.318.837194445';
catalog datafilecopy'+DATA/ebsback/datafile/system.294.837193797';
catalog datafilecopy'+DATA/ebsback/datafile/system.333.837194971';
catalog datafilecopy'+DATA/ebsback/datafile/system.319.837194445';
catalog datafilecopy'+DATA/ebsback/datafile/system.295.837193797';
catalog datafilecopy'+DATA/ebsback/datafile/system.300.837193803';
catalog datafilecopy'+DATA/ebsback/datafile/apps_undots1.261.837193733';
catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_archive.304.837193805';
catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_interface.306.837193807';
catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_media.327.837194907';
catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_nologging.303.837193805';
catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_queues.307.837193807';
catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_queues.308.837193807';
catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_seed.284.837193785';
catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_seed.291.837193789';
catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_summary.305.837193807';
catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_tx_data.264.837193733';
catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_tx_data.259.837193733';
catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_tx_data.268.837193735';
catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_tx_idx.326.837194907';
catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_tx_idx.299.837193801';
catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_tx_idx.301.837193803';
catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_tx_idx.296.837193801';
catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_tx_idx.297.837193801';
catalog datafilecopy'+DATA/ebsback/datafile/ctxd.322.837194611';
catalog datafilecopy'+DATA/ebsback/datafile/odm.328.837194907';
catalog datafilecopy'+DATA/ebsback/datafile/olap.310.837193809';
catalog datafilecopy'+DATA/ebsback/datafile/owapub.313.837193811';
catalog datafilecopy'+DATA/ebsback/datafile/portal.314.837193811';
catalog datafilecopy'+DATA/ebsback/datafile/sysaux.325.837194905';
catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_tools.311.837193809';
catalog datafilecopy'+DATA/ebsback/datafile/interim.312.837193809';
catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_tx_data.265.837193735';
catalog datafilecopy'+DATA/ebsback/datafile/apps_undots1.286.837193787';
catalog datafilecopy'+DATA/ebsback/datafile/cuxd.323.837194903';
catalog datafilecopy'+DATA/ebsback/datafile/cuxi.260.837193733';
catalog datafilecopy'+DATA/ebsback/datafile/apps_pts_cux0.267.837193735';
catalog datafilecopy'+DATA/ebsback/datafile/apps_pts_cux1.263.837193733';
catalog datafilecopy'+DATA/ebsback/datafile/apps_pts_cux2.266.837193735';
catalog datafilecopy'+DATA/ebsback/datafile/apps_pts_cux3.269.837193735';
catalog datafilecopy'+DATA/ebsback/datafile/apps_pts_cux4.329.837194969';
catalog datafilecopy'+DATA/ebsback/datafile/apps_pts_cux5.315.837194441';
catalog datafilecopy'+DATA/ebsback/datafile/apps_pts_cux6.274.837193759';
catalog datafilecopy'+DATA/ebsback/datafile/apps_pts_cux7.281.837193761';
catalog datafilecopy'+DATA/ebsback/datafile/apps_undots1.292.837193789';
catalog datafilecopy'+DATA/ebsback/datafile/undotbs2.298.837193801';
catalog datafilecopy'+DATA/ebsback/datafile/apps_undots2.273.837193757';
catalog datafilecopy'+DATA/ebsback/datafile/apps_undots2.288.837193787';
catalog datafilecopy'+DATA/ebsback/datafile/apps_undots2.287.837193787';
catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_interface.334.837194973';
catalog datafilecopy'+DATA/ebsback/datafile/cuxd.278.837193759';
catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_summary.320.837194445';
catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_media.302.837193803';
catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_tx_data.321.837194607';
catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_tx_idx.309.837193807';
catalog datafilecopy'+DATA/ebsback/datafile/cuxi.282.837193763';
catalog datafilecopy'+DATA/ebsback/datafile/cuxd.324.837194905';
catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_tx_data.271.837193735';
catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_tx_idx.277.837193759';
- 备库使用set newname 命令修改数据文件路径
rman target /
run {
set newname for datafile 1 to '+DATA/ebsback/datafile/system.285.837193785';
set newname for datafile 2 to '+DATA/ebsback/datafile/system.331.837194971';
set newname for datafile 3 to '+DATA/ebsback/datafile/system.317.837194443';
set newname for datafile 4 to '+DATA/ebsback/datafile/system.293.837193791';
set newname for datafile 5 to '+DATA/ebsback/datafile/system.332.837194971';
set newname for datafile 6 to '+DATA/ebsback/datafile/system.318.837194445';
set newname for datafile 7 to '+DATA/ebsback/datafile/system.294.837193797';
set newname for datafile 8 to '+DATA/ebsback/datafile/system.333.837194971';
set newname for datafile 9 to '+DATA/ebsback/datafile/system.319.837194445';
set newname for datafile 10 to '+DATA/ebsback/datafile/system.295.837193797';
set newname for datafile 11 to '+DATA/ebsback/datafile/system.300.837193803';
set newname for datafile 12 to '+DATA/ebsback/datafile/apps_undots1.261.837193733';
set newname for datafile 13 to '+DATA/ebsback/datafile/apps_ts_archive.304.837193805';
set newname for datafile 14 to '+DATA/ebsback/datafile/apps_ts_interface.306.837193807';
set newname for datafile 15 to '+DATA/ebsback/datafile/apps_ts_media.327.837194907';
set newname for datafile 16 to '+DATA/ebsback/datafile/apps_ts_nologging.303.837193805';
set newname for datafile 17 to '+DATA/ebsback/datafile/apps_ts_queues.307.837193807';
set newname for datafile 18 to '+DATA/ebsback/datafile/apps_ts_queues.308.837193807';
set newname for datafile 19 to '+DATA/ebsback/datafile/apps_ts_seed.284.837193785';
set newname for datafile 20 to '+DATA/ebsback/datafile/apps_ts_seed.291.837193789';
set newname for datafile 21 to '+DATA/ebsback/datafile/apps_ts_summary.305.837193807';
set newname for datafile 22 to '+DATA/ebsback/datafile/apps_ts_tx_data.264.837193733';
set newname for datafile 23 to '+DATA/ebsback/datafile/apps_ts_tx_data.259.837193733';
set newname for datafile 24 to '+DATA/ebsback/datafile/apps_ts_tx_data.268.837193735';
set newname for datafile 25 to '+DATA/ebsback/datafile/apps_ts_tx_idx.326.837194907';
set newname for datafile 26 to '+DATA/ebsback/datafile/apps_ts_tx_idx.299.837193801';
set newname for datafile 27 to '+DATA/ebsback/datafile/apps_ts_tx_idx.301.837193803';
set newname for datafile 28 to '+DATA/ebsback/datafile/apps_ts_tx_idx.296.837193801';
set newname for datafile 29 to '+DATA/ebsback/datafile/apps_ts_tx_idx.297.837193801';
set newname for datafile 30 to '+DATA/ebsback/datafile/ctxd.322.837194611';
set newname for datafile 31 to '+DATA/ebsback/datafile/odm.328.837194907';
set newname for datafile 32 to '+DATA/ebsback/datafile/olap.310.837193809';
set newname for datafile 33 to '+DATA/ebsback/datafile/owapub.313.837193811';
set newname for datafile 34 to '+DATA/ebsback/datafile/portal.314.837193811';
set newname for datafile 35 to '+DATA/ebsback/datafile/sysaux.325.837194905';
set newname for datafile 36 to '+DATA/ebsback/datafile/apps_ts_tools.311.837193809';
set newname for datafile 37 to '+DATA/ebsback/datafile/interim.312.837193809';
set newname for datafile 38 to '+DATA/ebsback/datafile/apps_ts_tx_data.265.837193735';
set newname for datafile 39 to '+DATA/ebsback/datafile/apps_undots1.286.837193787';
set newname for datafile 40 to '+DATA/ebsback/datafile/cuxd.323.837194903';
set newname for datafile 41 to '+DATA/ebsback/datafile/cuxi.260.837193733';
set newname for datafile 42 to '+DATA/ebsback/datafile/apps_pts_cux0.267.837193735';
set newname for datafile 43 to '+DATA/ebsback/datafile/apps_pts_cux1.263.837193733';
set newname for datafile 44 to '+DATA/ebsback/datafile/apps_pts_cux2.266.837193735';
set newname for datafile 45 to '+DATA/ebsback/datafile/apps_pts_cux3.269.837193735';
set newname for datafile 46 to '+DATA/ebsback/datafile/apps_pts_cux4.329.837194969';
set newname for datafile 47 to '+DATA/ebsback/datafile/apps_pts_cux5.315.837194441';
set newname for datafile 48 to '+DATA/ebsback/datafile/apps_pts_cux6.274.837193759';
set newname for datafile 49 to '+DATA/ebsback/datafile/apps_pts_cux7.281.837193761';
set newname for datafile 50 to '+DATA/ebsback/datafile/apps_undots1.292.837193789';
set newname for datafile 51 to '+DATA/ebsback/datafile/undotbs2.298.837193801';
set newname for datafile 52 to '+DATA/ebsback/datafile/apps_undots2.273.837193757';
set newname for datafile 53 to '+DATA/ebsback/datafile/apps_undots2.288.837193787';
set newname for datafile 54 to '+DATA/ebsback/datafile/apps_undots2.287.837193787';
set newname for datafile 55 to '+DATA/ebsback/datafile/apps_ts_interface.334.837194973';
set newname for datafile 56 to '+DATA/ebsback/datafile/cuxd.278.837193759';
set newname for datafile 57 to '+DATA/ebsback/datafile/apps_ts_summary.320.837194445';
set newname for datafile 58 to '+DATA/ebsback/datafile/apps_ts_media.302.837193803';
set newname for datafile 59 to '+DATA/ebsback/datafile/apps_ts_tx_data.321.837194607';
set newname for datafile 60 to '+DATA/ebsback/datafile/apps_ts_tx_idx.309.837193807';
set newname for datafile 61 to '+DATA/ebsback/datafile/cuxi.282.837193763';
set newname for datafile 62 to '+DATA/ebsback/datafile/cuxd.324.837194905';
set newname for datafile 63 to '+DATA/ebsback/datafile/apps_ts_tx_data.271.837193735';
}
备库开启同步机制：
SQL>alter database recover managed database standby database disconnect from session;
RMAN>backup datafile 后接主库新加的文件号
- 主库备份缺少的数据文件
- 备库恢复新增加的数据文件
RMAN>catalog start with '/主库备份文件拷贝到备库的所在路径';
RMAN>restore datafile 文件号
alter database recover managed standby database using current logfile disconnect from session
- 开启DG同步机制