dataguard (ADG)备库移动（迁移）数据文件
最近负责维护的一套oracle rac的dataguard环境在巡检的时候发现问题，现在将当时的处理过程整理了下。
环境介绍：
| primary主库 | Standby备库 |
|---|---|
| HPUX 11.31系统 | HPUX 11.31系统 |
| ORACLE 11.2.0.4 RAC数据库 | ORACLE 11.2.0.4 RAC数据库 |
参考文章：http://blog.chinaunix.net/uid-77311-id-5818675.html
1. 问题现象：
当时巡检时发现主库归档空间用的很多，当时觉的不对劲，因为正常归档不应该保留这么多的，备份完就会删除，因为这个库是个ADG，所以开始检查ADG状态和日志，发现ADG进程异常没有正常同步到备库，且日志有异常报错。
2. 检查过程：
1） 首先检查ADG进程的状态，以及归档应用情况，发现源端（primary）和目标端（Standby）队列相差比较大源端已经到了27450,目标端还在27302，而且目标端的MRP0 进程没有运行。
2）源端查看日志应用情况，12月1日之后的归档都没有在目标端应用。
3)检查primary数据库日志，在源端数据库发现archive failed错误。
FAL[server, ARC3]: FAL archive failed, see trace file.
ARCH: FAL archive failed. Archiver continuing
ORACLE Instance ydjyorcl1 - Archival Error. Archiver continuing.
FAL[server, ARC0]: FAL archive failed, see trace file.
ARCH: FAL archive failed. Archiver continuing
ORACLE Instance ydjyorcl1 - Archival Error. Archiver continuing.
FAL[server, ARC1]: FAL archive failed, see trace file.
ARCH: FAL archive failed. Archiver continuing
ORACLE Instance ydjyorcl1 - Archival Error. Archiver continuing.
4）检查Standby数据库日志，提示ARCH的diskgroup空间不足 ，基本上找到问题关键所在就是ARCH磁盘组的空间不够造成的。
Creating archive destination file : +ARCH (1902850 blocks)
Errors in file /opt/u01/app/oracle/diag/rdbms/ydjydg/ydjydg1/trace/ydjydg1_ora_13970.trc:
ORA-19816: WARNING: Files may exist in db_recovery_file_dest that are not known to database.
ORA-17502: ksfdcre:4 Failed to create file +ARCH
ORA-15041: diskgroup "ARCH" space exhausted
5）查看磁盘组的使用情况
主库：
备库：和主库相比，ARCH和DATA磁盘组都已经没有剩于空间了。
3. 原因：
正常情况主库和备库使用的空间应该是一致的，不应该出现空间主备库相差这么大，根据检查结果主要是以下原因导致的：
1、DATA空间满是因为前一周主库增加了一个表空间，数据文件放在ARCH磁盘组，而备库standby_file_management为自动处理到DATA磁盘组。
主库：
备库：
2、ARCH磁盘组空间满主要是因为归档保留周期比较长，当前是一个月。
4. 处理：
找到原因之后，处理措施是手动在备库将新增的表空间文件迁移到ARCH磁盘组。
1、关闭adg日志应用
SQL> alter database recover managed standby database cancel;
2、关闭数据库，两个rac节点都要执行
SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
#检查状态
grid@ydjydb3[+ASM1]:/home/grid$ crsctl status resource -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ARCH.dg
ONLINE ONLINE ydjydb3
ONLINE ONLINE ydjydb4
ora.DATA.dg
ONLINE ONLINE ydjydb3
ONLINE ONLINE ydjydb4
ora.LISTENER.lsnr
ONLINE ONLINE ydjydb3
ONLINE ONLINE ydjydb4
ora.OCRVOTING.dg
ONLINE ONLINE ydjydb3
ONLINE ONLINE ydjydb4
ora.asm
ONLINE ONLINE ydjydb3 Started
ONLINE ONLINE ydjydb4 Started
ora.gsd
OFFLINE OFFLINE ydjydb3
OFFLINE OFFLINE ydjydb4
ora.net1.network
ONLINE ONLINE ydjydb3
ONLINE ONLINE ydjydb4
ora.ons
ONLINE ONLINE ydjydb3
ONLINE ONLINE ydjydb4
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE ydjydb3
ora.cvu
1 ONLINE ONLINE ydjydb3
ora.oc4j
1 OFFLINE OFFLINE
ora.scan1.vip
1 ONLINE ONLINE ydjydb3
ora.ydjydb3.vip
1 ONLINE ONLINE ydjydb3
ora.ydjydb4.vip
1 ONLINE ONLINE ydjydb4
ora.ydjydg.db
1 OFFLINE OFFLINE Instance Shutdown
2 OFFLINE OFFLINE Instance Shutdown
3、启动nomount状态。
SQL> startup nomount;
ORACLE instance started.
Total System Global Area 1.0957E+11 bytes
Fixed Size 2218568 bytes
Variable Size 5.3419E+10 bytes
Database Buffers 5.6103E+10 bytes
Redo Buffers 43307008 bytes
4、移动数据文件
ASMCMD> cp +data/ydjydg/datafile/FQCS.595.1023725157 +arch/ydjydg/datafile/FQCS.595.1023725157
copying +data/ydjydg/datafile/FQCS.595.1023725157 -> +arch/ydjydg/datafile/FQCS.595.1023725157
ASMCMD-8016: copy source '+data/ydjydg/datafile/FQCS.595.1023725157' and target '+arch/ydjydg/datafile/FQCS.595.1023725157' failed
ORA-15056: additional error message
ORA-15046: ASM file name '+arch/ydjydg/datafile/FQCS.595.1023725157' is not in single-file creation form
ORA-06512: at "SYS.X$DBMS_DISKGROUP", line 415
ORA-06512: at line 3 (DBD ERROR: OCIStmtExecute)
发现在asm里面无法移动数据文件，只需要在ASM上复制的时候不要指定文件后面的数值即可。
ASMCMD> cp +data/ydjydg/datafile/FQCS.618.1025740023 +arch/ydjydg/datafile/fqcs19.dbf
copying +data/ydjydg/datafile/FQCS.618.1025740023 -> +arch/ydjydg/datafile/fqcs19.dbf
ASMCMD> cp +data/ydjydg/datafile/FQCS.620.1025740183 +arch/ydjydg/datafile/fqcs21.dbf
copying +data/ydjydg/datafile/FQCS.620.1025740183 -> +arch/ydjydg/datafile/fqcs21.dbf
查看文件发现ASM其实并没有完全根据命令将数据文件拷贝到指定的目录，ASM只是在目标目录下存储了一个alias，真正的文件被ASM放到了其他的位置。
ASMCMD> ls -l
Type Redund Striped Time Sys Name
N fqcs01.dbf => +ARCH/ASM/DATAFILE/fqcs01.dbf.1072.1027602353
N fqcs02.dbf => +ARCH/ASM/DATAFILE/fqcs02.dbf.1859.1027602347
N fqcs03.dbf => +ARCH/ASM/DATAFILE/fqcs03.dbf.720.1027602777
N fqcs04.dbf => +ARCH/ASM/DATAFILE/fqcs04.dbf.1400.1027602837
N fqcs05.dbf => +ARCH/ASM/DATAFILE/fqcs05.dbf.2388.1027602847
N fqcs06.dbf => +ARCH/ASM/DATAFILE/fqcs06.dbf.1881.1027602885
N fqcs07.dbf => +ARCH/ASM/DATAFILE/fqcs07.dbf.3741.1027602893
N fqcs08.dbf => +ARCH/ASM/DATAFILE/fqcs08.dbf.1629.1027602923
N fqcs09.dbf => +ARCH/ASM/DATAFILE/fqcs09.dbf.3080.1027602931
N fqcs10.dbf => +ARCH/ASM/DATAFILE/fqcs10.dbf.2923.1027602817
N fqcs11.dbf => +ARCH/ASM/DATAFILE/fqcs11.dbf.3016.1027603145
N fqcs12.dbf => +ARCH/ASM/DATAFILE/fqcs12.dbf.3025.1027602857
N fqcs13.dbf => +ARCH/ASM/DATAFILE/fqcs13.dbf.3091.1027603179
N fqcs14.dbf => +ARCH/ASM/DATAFILE/fqcs14.dbf.3113.1027602891
N fqcs15.dbf => +ARCH/ASM/DATAFILE/fqcs15.dbf.3135.1027603215
N fqcs16.dbf => +ARCH/ASM/DATAFILE/fqcs16.dbf.3279.1027602927
N fqcs17.dbf => +ARCH/ASM/DATAFILE/fqcs17.dbf.3325.1027603257
N fqcs18.dbf => +ARCH/ASM/DATAFILE/fqcs18.dbf.3398.1027602969
N fqcs19.dbf => +ARCH/ASM/DATAFILE/fqcs19.dbf.947.1027603297
N fqcs20.dbf => +ARCH/ASM/DATAFILE/fqcs20.dbf.891.1027603009
N fqcs21.dbf => +ARCH/ASM/DATAFILE/fqcs21.dbf.877.1027603335
5、将数据库启动到mount状态，修改standby_file_management 参数，再修改数据文件路径。
SQL> startup mount;
ORACLE instance started.
Total System Global Area 1.0957E+11 bytes
Fixed Size 2218568 bytes
Variable Size 5.3419E+10 bytes
Database Buffers 5.6103E+10 bytes
Redo Buffers 43307008 bytes
Database mounted.
SQL> alter system set standby_file_management = MANUAL;
System altered.
SQL> alter database rename file '+data/ydjydg/datafile/FQCS.595.1023725157' to '+arch/ydjydg/datafile/fqcs01.dbf';
alter database rename file '+data/ydjydg/datafile/FQCS.596.1024163587' to '+arch/ydjydg/datafile/fqcs02.dbf';
alter database rename file '+data/ydjydg/datafile/FQCS.602.1025733207' to '+arch/ydjydg/datafile/fqcs03.dbf';
alter database rename file '+data/ydjydg/datafile/FQCS.603.1025733209' to '+arch/ydjydg/datafile/fqcs04.dbf';
alter database rename file '+data/ydjydg/datafile/FQCS.604.1025733211' to '+arch/ydjydg/datafile/fqcs05.dbf';
alter database rename file '+data/ydjydg/datafile/FQCS.605.1025733215' to '+arch/ydjydg/datafile/fqcs06.dbf';
alter database rename file '+data/ydjydg/datafile/FQCS.606.1025733217' to '+arch/ydjydg/datafile/fqcs07.dbf';
alter database rename file '+data/ydjydg/datafile/FQCS.607.1025733219' to '+arch/ydjydg/datafile/fqcs08.dbf';
alter database rename file '+data/ydjydg/datafile/FQCS.608.1025733223' to '+arch/ydjydg/datafile/fqcs09.dbf';
alter database rename file '+data/ydjydg/datafile/FQCS.609.1025733225' to '+arch/ydjydg/datafile/fqcs10.dbf';
alter database rename file '+data/ydjydg/datafile/FQCS.610.1025733227' to '+arch/ydjydg/datafile/fqcs11.dbf';
alter database rename file '+data/ydjydg/datafile/FQCS.611.1025733229' to '+arch/ydjydg/datafile/fqcs12.dbf';
alter database rename file '+data/ydjydg/datafile/FQCS.612.1025739745' to '+arch/ydjydg/datafile/fqcs13.dbf';
alter database rename file '+data/ydjydg/datafile/FQCS.613.1025739775' to '+arch/ydjydg/datafile/fqcs14.dbf';
alter database rename file '+data/ydjydg/datafile/FQCS.614.1025739797' to '+arch/ydjydg/datafile/fqcs15.dbf';
alter database rename file '+data/ydjydg/datafile/FQCS.615.1025739855' to '+arch/ydjydg/datafile/fqcs16.dbf';
alter database rename file '+data/ydjydg/datafile/FQCS.616.1025739883' to '+arch/ydjydg/datafile/fqcs17.dbf';
alter database rename file '+data/ydjydg/datafile/FQCS.617.1025739943' to '+arch/ydjydg/datafile/fqcs18.dbf';
alter database rename file '+data/ydjydg/datafile/FQCS.618.1025740023' to '+arch/ydjydg/datafile/fqcs19.dbf';
alter database rename file '+data/ydjydg/datafile/FQCS.619.1025740097' to '+arch/ydjydg/datafile/fqcs20.dbf';
alter database rename file '+data/ydjydg/datafile/FQCS.620.1025740183' to '+arch/ydjydg/datafile/fqcs21.dbf';
Database altered.
SQL>
Database altered.
SQL>
Database altered.
SQL>
Database altered.
6、打开数据库，并启动日志应用
SQL> alter database open;
SQL> alter database recover managed standby database using current logfile disconnect from session;
Database altered.
7、检查表空间数据文件信息，数据字典已经指向到迁移的文件上去了。
8、检查adg状态
#检查数据库情况
grid@ydjydb3[+ASM1]:/home/grid$ crsctl status resource -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ARCH.dg
ONLINE ONLINE ydjydb3
ONLINE ONLINE ydjydb4
ora.DATA.dg
ONLINE ONLINE ydjydb3
ONLINE ONLINE ydjydb4
ora.LISTENER.lsnr
ONLINE ONLINE ydjydb3
ONLINE ONLINE ydjydb4
ora.OCRVOTING.dg
ONLINE ONLINE ydjydb3
ONLINE ONLINE ydjydb4
ora.asm
ONLINE ONLINE ydjydb3 Started
ONLINE ONLINE ydjydb4 Started
ora.gsd
OFFLINE OFFLINE ydjydb3
OFFLINE OFFLINE ydjydb4
ora.net1.network
ONLINE ONLINE ydjydb3
ONLINE ONLINE ydjydb4
ora.ons
ONLINE ONLINE ydjydb3
ONLINE ONLINE ydjydb4
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE ydjydb3
ora.cvu
1 ONLINE ONLINE ydjydb3
ora.oc4j
1 OFFLINE OFFLINE
ora.scan1.vip
1 ONLINE ONLINE ydjydb3
ora.ydjydb3.vip
1 ONLINE ONLINE ydjydb3
ora.ydjydb4.vip
1 ONLINE ONLINE ydjydb4
ora.ydjydg.db
1 ONLINE ONLINE ydjydb3 Open,Readonly
2 ONLINE ONLINE ydjydb4 Open,Readonly
#检adg情况
SQL> select open_mode, switchover_status from
v$database;
OPEN_MODE SWITCHOVER_STATUS
-------------------- --------------------
READ ONLY NOT ALLOWED
SQL>SELECT PID,PROCESS,STATUS,CLIENT_PROCESS,CLIENT_PID, THREAD# , SEQUENCE# SEQ#,BLOCK# , BLOCKS FROM V$MANAGED_STANDBY;
PID PROCESS STATUS CLIENT_P CLIENT_PID THREAD# SEQ# BLOCK# BLOCKS
-------- --------- ------------ -------- ---------------------------------------- ------- ---------- ---------- ----------
22297 ARCH CLOSING ARCH 22297 1 27451 4096 255
22299 ARCH CONNECTED ARCH 22299 0 0 0 0
22301 ARCH CONNECTED ARCH 22301 0 0 0 0
22303 ARCH CLOSING ARCH 22303 2 33555 1033216 127
22726 RFS IDLE LGWR 5737 1 27452 30379 1
22750 RFS IDLE ARCH 23597 0 0 0 0
22722 RFS IDLE LGWR 14249 2 33557 21120 9
22728 RFS IDLE ARCH 20786 0 0 0 0
22709 RFS IDLE UNKNOWN 13249 0 0 0 0
9 rows selected.
SQL> /
PID PROCESS STATUS CLIENT_P CLIENT_PID THREAD# SEQ# BLOCK# BLOCKS
-------- --------- ------------ -------- ---------------------------------------- ------- ---------- ---------- ----------
22297 ARCH CLOSING ARCH 22297 1 27451 4096 255
22299 ARCH CONNECTED ARCH 22299 0 0 0 0
22301 ARCH CONNECTED ARCH 22301 0 0 0 0
22303 ARCH CLOSING ARCH 22303 2 33555 1033216 127
22726 RFS IDLE LGWR 5737 1 27452 38812 1
22750 RFS IDLE ARCH 23597 0 0 0 0
22722 RFS IDLE LGWR 14249 2 33557 35045 3
22728 RFS IDLE ARCH 20786 0 0 0 0
22709 RFS IDLE UNKNOWN 13249 0 0 0 0
23111 MRP0 APPLYING_LOG N/A N/A 2 32693 1027 5753
23124 RFS RECEIVING UNKNOWN 19301 1 27450 315394 1024
11 rows selected.
过个几分钟再刷新的时候可以看到mrp0已经开始在应用日志了，且接收进程也在正常运行了。
查看日志延这时间
SQL> select name,value,datum_time from v$dataguard_stats;
NAME VALUE DATUM_TIME
-------------------------------- ---------------------------------------------------------------- ------------------------------
transport lag +00 00:00:00 12/21/2019 13:44:45
apply lag +19 18:30:06 12/21/2019 13:44:45
apply finish time +00 02:16:34.324
estimated startup time 28
至此，adg运行恢复正常。
