OGG概念、安装步骤及基本的配置步骤 for ORACLE（二）

• 必要的配置说明：

1. 关闭回收站和配置附加日志

alter system set recyclebin=off;

sqlplus “/as sysdba”

ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;

ALTER SYSTEM SWITCH LOGFILE;

SELECT SUPPLEMENTAL_LOG_DATA_MIN FROM V$DATABASE;

1. initiate the logging of key values

ADD SCHEMATRANDATA:当配置DDL复制的时候应该使用，DDL不使用的时候，可以代替ADD TRANDATA，表没有主键和唯一键的时候，使用ADD TRANDATA的cols和table map的keycols来配合使用ADD SCHEMATRANDATA最好。

ADD TRANDATA：不适用DDL复制特性的时候使用，在下保证列情况下，在DDL复制环境下也可以使用ADD TRANDATA:在对表进行DDL时，一定不能有任何DML，可以有对其他不相关表的DML

1. (ADD SCHEMATRANDATA only) Apply Oracle Patch 10423000 to the source Oracle database. Oracle数据库11g以后才支持ADD/INFO/DELETE SCHEMATRANDATA命令

先sys执行exec dbms_streams_auth.grant_admin_privilege('oggadm');

2. On the source system, run GGSCI.

3. dblogin userid oggadm@source,password oggadm

4. Issue the ADD TRANDATA or ADD SCHEMATRANDATA command.

ADD TRANDATA <table> [, COLS <columns>][,NOKEY]

其中的COLS是配合extract中的TABLE参数和replicat中的MAP参数里的KEYCOLS子句来使用的，如果指定NOKEY，那就必须指定COLS，也必须指定KEYCOLS字句

ADD SCHEMATRANDATA <schema>

查看效果

INFO TRANDATA <owner/schema>.TCUST*

1. 防止由于triggers 和cascade constraints带来的数据重复修改

10.2.0.5.0以后，Replicat parameter DBOPTIONS with the SUPPRESSTRIGGERS option

9.2.0.7.0 以后，Replicat parameter DBOPTIONS with the DEFERREFCONST option

再之前的数据库版本就得手工停止trigger，或者手工忽略Replicat database user

1. Deferring constraint checking

Source上是DEFERRABLE的约束，在target里必须也是DEFERRABLE的，在Replicat parameter file文件里，添加一行SQLEXEC (“alter session set constraint deferred”)

9.2.0.7.0 以后，Replicat parameter DBOPTIONS with DEFERREFCONST

1. Limiting row changes in tables that do not have a key

如果target上的表没有主键或唯一健，那么就会有重复记录存在，那么应用过来的操作就会针对多条记录进行操作，造成source和target不一致，需要设置一个选项DBOPTIONS LIMITROWS in the Replicat parameter file.

1. 复制TIMESTAMP with TIME ZONE

Oracle GoldenGate supports the capture and replication of TIMESTAMP WITH TIME ZONE as a UTC offset (TIMESTAMP '2011-01-01 8:00:00 -8:00') but abends on TIMESTAMP WITH TIME ZONE as TZR (TIMESTAMP '2011-01-01 8:00:00 US/Pacific') by default。

要想支持TIMESTAMP WITH TIME ZONE as TZR, 得在Extract parameter 文件里使用选项TRANLOGOPTIONS 如下两个选项：

INCLUDEREGIONID 从oracle到同版本oracle或更高版本

INCLUDEREGIONIDWITHOFFSET 从oracle到低版本或者非oracle

• 配置MANAGER：

1. 配置MGR参数

Ggsci

EDIT PARAMS MGR

port 7809

AUTOSTART ER *

AUTORESTART EXTRACT *,RETRIES 3,WAITMINUTES 5,RESETMINUTES 60

DYNAMICPORTLIST 7940-8100,7930

SOURCEDB gzcrm20,USERID gguser,PASSWD gguser

PURGEOLDEXTRACTS /ggs/dirdat/aa*,USECHECKPOINTS,MINKEEPDAYS 2

LAGREPORTHOURS 1 检查lag的间隔

LAGINFOMINUTES 3 报info

LAGCRITICALMINUTES 5 报error

INFO MGR

1. 配置CHECKPOINT TABLE

创建checkpoint table有两种方法，

一种是使用chkpt_ora_create.sql脚本创建，

另一种方式是使用ggsci里的命令如下：

dblogin userid oggadm@source,password oggadm

add checkpointtable oggadm.ggs_checkpoint

通过EDIT PARAMS ./GLOBALS加一行CHECKPOINTTABLE oggadm.ggs_checkpoint来指定checkpoint table的默认位置

如果没有在GLOBALS中添加的话，需要添加replicat的时候指定

ADD REPLICAT emp_rep, EXTTRAIL C:\OGG10G\dirdat\et, CHECKPOINTTABLE oggadm.ggs_checkpoint

如果不想使用checkpoint，只是使用默认的trail文件中记录的checkpoint接受数据不一致的话，可以使用NODBCHECKPOINT

• 配置一个active-active的架构

1. SOURCE端
   1. 1. EXTRACT1

ADD EXTRACT ext_src, TRANLOG, BEGIN NOW

ADD EXTTRAIL G:\OGG\V26068-01-source\dirdat\et, EXTRACT ext_src

extract ext1

userid ggs,password ggs

exttrail g:\ogg\v26068-01-source\dirdat\et

rmthost 192.168.234.112, mgrport 7809
rmttrail c:\ggs\dirdat\r1
dynamicresolution
gettruncates

TRANLOGOPTIONS INCLUDEREGIONID,EXCLUDEUSER oggadm

DDL include all
table ggs.*;

SEQUENCE hr.employees_seq;

DDLERROR [RESTARTSKIP <1-10万>] [SKIPTRIGGERERROR <num errors>]

捕获DDL errors，Extract最多可以跳过十万个DDL 操作而不终止

DDLOPTIONS GETAPPLOPS, IGNOREREPLICATES（默认配置）

IGNOREREPLICATES 和 GETREPLICATES 控制extract是否忽略由replicat产生的ddl操作

GETAPPLOPS 和 IGNOREAPPLOPS 控制extract是否忽略由应用等其他操作产生的ddl操作

要想避免数据的looping复制，要做到两点：

第一点是合理配置IGNOREREPLICATES 和 GETREPLICATES，GETAPPLOPS 和 IGNOREAPPLOPS这两对参数；

第二点是identify local Replicat transactions，这个使用TRANLOGOPTIONS EXCLUDEUSER oggadm来实现

下面是针对DML的配置，直接写在extract文件里即可

GETAPPLOPS

IGNOREREPLICATES

下面是DDLOPTIONS里的配置

bi-directional模式里，需要配置成

Extract (primary and secondary)

DDLOPTIONS GETREPLICATES, GETAPPLOPS   

Replicat (primary and secondary)

DDLOPTIONS UPDATEMETADATA

Cascading模式

DDLOPTIONS GETREPLICATES, IGNOREAPPLOPS

一个实验过的例子：

edit params ext_src

extract ext_src

userid oggadm@source,password oggadm

EXTTRAIL G:\OGG\V26068-01-source\dirdat\et

dynamicresolution

gettruncates

TRANLOGOPTIONS INCLUDEREGIONID,EXCLUDEUSER oggadm

GETAPPLOPS

IGNOREREPLICATES

DDL include all

DDLOPTIONS GETAPPLOPS, GETREPLICATES

table source.*;

SEND EXTRACT <name> SHOWTRANS 查看extract需要的最早的log信息

1. 1. 1. Data pump1

ADD EXTRACT pump_src, EXTTRAILSOURCE G:\OGG\V26068-01-source\dirdat\et, BEGIN now

ADD RMTTRAIL G:\OGG\V26068-01-target\dirdat\rt, EXTRACT pump_src

Edit params pump_src

extract pump_src

userid oggadm@source,password oggadm

rmthost pc2011022014ptf, mgrport 7810

rmttrail g:\ogg\v26068-01-target\dirdat\rt

passthru

table source.*;

1. TARGET端
   1. 1. Replicat2

ADD REPLICAT rep_tag, EXTTRAIL G:\OGG\V26068-01-target\dirdat\rt, BEGIN now

replicat rep1

userid oggadm@target,password oggadm

assumetargetdefs

reperror (default,discard,abend)

DBOPTIONS LIMITROWS SUPPRESSTRIGGERS

DDL include all

HANDLECOLLISIONS 此参数在source active状态时做initial load才需要，initial load 之后SEND REPLICAT <Replicat group name>, NOHANDLECOLLISIONS

discardfile c:\oradata\repsz.dsc,append,megabytes 100

map ggs.*, target ggs.*;

MAP hr.employees_seq, TARGET payroll.employees_seq;

DDLOPTIONS UPDATEMETADATA

DDLERROR 捕获DDL errors

{<error> | DEFAULT} {<response>}

{INCLUDE <inclusion clause> | EXCLUDE <exclusion clause>}

[IGNOREMISSINGOBJECTS | ABENDONMISSINGOBJECTS]

例如：DDLERROR <error> IGNORE RETRYOP MAXRETRIES 3 RETRYDELAY 10 &

INCLUDE ALL OBJTYPE TABLE OBJNAME “tab*” EXCLUDE OBJNAME “tab1*”

一个实验用过的例子

Edit params rep_tag

replicat rep_src

userid oggadm@source,password oggadm

assumetargetdefs

reperror (default,discard,abend)

DBOPTIONS LIMITROWS DEFERREFCONST

DDL include all

DDLOPTIONS UPDATEMETADATA

map target.*, target source.*;

START REPLICAT

Syntax START REPLICAT <group name> ATCSN <csn> | AFTERCSN <csn>]

ATCSN <csn> 小于这个csn值的操作被跳过.

AFTERCSN <csn> 大于这个csn值的操作被跳过。

1. 1. 1. Extract2

ADD EXTRACT ext_tag, TRANLOG, BEGIN NOW

ADD EXTTRAIL G:\OGG\V26068-01-target\dirdat\et, EXTRACT ext_tag

一个实验用过的例子

Edit params ext_tag

extract ext_tag

userid oggadm@target,password oggadm

EXTTRAIL G:\OGG\V26068-01-target\dirdat\et

dynamicresolution

gettruncates

TRANLOGOPTIONS INCLUDEREGIONID

DDL include all

DDLOPTIONS GETREREPLICATES, GETAPPLOPS

table target.*;

1. 1. 1. Data pump2

ADD EXTRACT pump_tag, EXTTRAILSOURCE G:\OGG\V26068-01-target\dirdat\et, BEGIN now

ADD RMTTRAIL G:\OGG\V26068-01-source\dirdat\rt, EXTRACT pump_tag

Edit params pump_tag

extract pump_tag

userid oggadm@target,password oggadm

rmthost pc2011022014ptf, mgrport 7809

rmttrail g:\ogg\v26068-01-source\dirdat\rt

passthru

table target.*;

1. SOURCE端
   1. 1. Replicat1

ADD REPLICAT rep_src, EXTTRAIL G:\OGG\V26068-01-source\dirdat\rt, BEGIN now

Edit params rep_src

replicat rep_src

userid oggadm@source,password oggadm

assumetargetdefs

reperror (default,discard,abend)

DBOPTIONS LIMITROWS DEFERREFCONST

DDL include all  

map target.*, target source.*;

DDLOPTIONS UPDATEMETADATA

1. mapping and manipulating data

在extract中使用filter和where选行

TABLE <table spec>, FILTER ([, ON INSERT | ON UPDATE| ON DELETE][, IGNORE INSERT | IGNORE UPDATE | IGNORE DELETE], <filter clause>);

MAP <table spec>, TARGET <table spec>,

, FILTER ([, ON INSERT | ON UPDATE| ON DELETE][, IGNORE INSERT | IGNORE UPDATE | IGNORE DELETE][, RAISEERROR <error_num>], <filter clause>);

使用cols和colsexcept选列

column mapping分为表级别的和global级别的，在TABLE和MAP参数里 ，使用COLMAP选项，如果source和target的表是identical的，那么就只使用COLMAP选项就行，但是如果不是identical的，就要创建data-definitions file了

表级别的column mapping，如果使用了COLMAP，那么在TABLE中就得像MAP语句一样，增加一个TARGET选项，例如：

MAP sales.acctbl,TARGET sales.accttab,

COLMAP (

USEDEFAULTS,

name = cust_name,

transaction_date =@DATE (“YYYY-MM-DD”, “YY”,YEAR, “MM”, MONTH, “DD”, DAY),

area_code =@STREXT (phone_no, 1, 3),

phone_prefix =@STREXT (phone_no, 4, 6),

phone_number =@STREXT (phone_no, 7, 10));

GLOBAL级别的column mapping，如下：

COLMATCH NAMES customer_code = cust_code

COLMATCH NAMES customer_name = cust_name

COLMATCH NAMES customer_address = cust_addr

COLMATCH PREFIX S_

COLMATCH SUFFIX _tab

MAP sales.acct, TARGET sales.account, COLMAP (USEDEFAULTS);

MAP sales.ord, TARGET sales.order, COLMAP (USEDEFAULTS);

COLMATCH RESET

MAP sales.reg, TARGET sales.reg;

MAP sales.price, TARGET sales.price;

Tokens,传递数据用

TABLE ora.oratest,TOKENS(TK-HOST=@GETENV("GGENVIRONMENT","HOSTNAME"));

MAP

ora.oratest,TARGET ra.rpt,COLMAP(USEDEFAULTS,host=@token("TK-HOST"));

1. data-definitions file

source 和 target 的表有不同的data definitions（比如oracle到sql server，列名不同，数据类型不同，列长度不一样，表中列的顺序不一致）的时候，就要使用data-definitions file，在source上创建source-definitions file, 在target上创建target-definitions file,步骤一样：

ggsic

EDIT PARAMS DEFGEN

DEFSFILE C:\ggs\dirdef\record.def

USERID oddadm, PASSWORD oggadm

TABLE acct.cust100, DEF custdef;  此处定义了一个模板

TABLE ord.*;

TABLE hr.*;

保存退出后，运行defgen工具来生成出相应的data-definitions file

defgen paramfile dirprm/defgen.prm [reportfile dirrpt/defgen.rpt]

不指定reportfile的话，报告信息就只是输出到屏幕上

引用相应的data-definitions file的方法，是在extract参数文件里添加TARGETDEFS参数，在replicat参数文件里添加SOURCEDEFS参数

上面的例子对应的replicat的参数文件如下：

REPLICAT acctrep

SOURCEDEFS c:\ggs\dirdef\record.def

USERID ggs, PASSWORD ggs

MAP acct.cust*, TARGET acct.cust*, DEF custdef;

MAP ord.prod, TARGET ord.prod;

MAP ord.parts, TARGET ord.parts;

MAP hr.emp, TARGET hr.emp;

MAP hr.salary, TARGET hr.salary;

• 初始化数据加载

● “Loading data with a database utility”

● “Loading data from file to Replicat”

Extract writes records to an extract file and Replicat applies them to the target tables. This is the slowest initial-load method.

SOURCEISTABLE

SPECIALRUN

END RUNTIME

● “Loading data from file to database utility”

Extract writes records to extract files in external ASCII format. The files are used as data files for input into target tables by a bulk load utility. Replicat creates the run and control files.

SOURCEISTABLE

FORMATASCII, {BCP | SQLLOADER}

GENLOADFILES <template file>

● “Loading data with an Oracle GoldenGate direct load”

Extract communicates with Replicat directly across TCP/IP without using a Collector process or files. Replicat applies the data through the database engine.

ADD EXTRACT <initial-load Extract name>, SOURCEISTABLE

EXTRACT <initial-load Extract name>

ADD REPLICAT <initial-load Replicat name>, SPECIALRUN

REPLICAT <initial-load Replicat name>

• “Loading data with a direct bulk load to SQL*Loader” on page 232. Extract extracts records in external ASCII format and delivers them directly to Replicat, which delivers them to Oracle’s SQL*Loader bulk-load utility. This is the fastest method of loading Oracle data with Oracle GoldenGate.