Oracle GoldenGate realizes data synchronization (synchronization of internal and external networks in Windows environment)

Reference link: Oracle->Oracle database synchronization configuration based on Oracle GoldenGate (OGG) in Windows (super detailed)

I. Introduction

Comparison of GoldenGate and DataGuard

Basic concept of GoldenGate

Oracle Golden Gate software is a log-based structured data replication backup software. It obtains incremental changes in data by parsing the online logs or archived logs of the source database, and then applies these changes to the target database, thereby realizing the source database and the target database. Synchronize.

​ Oracle Golden Gate can realize sub-second level real-time replication of large amounts of data between heterogeneous IT infrastructures (including almost all common operating system platforms and database platforms), so that it can be used in emergency systems, online reports, and real-time data warehouses. Supply, transaction tracking, data synchronization, centralization/distribution, disaster recovery, database upgrade and migration, dual business centers and other scenarios.

​ ogg uses a modular process structure, which can flexibly extract and replicate table data, DML and DDL operations across multiple topologies (heterogeneous platforms). Because of the flexibility of ogg, it supports data filtering, transformation and custom processing features, which can support many business needs:

• Business continuity and high availability, often used for disaster recovery and replication
• Initial loading and database migration, such as online data migration without downtime
• Data integration and distribution, such as the summary of city data to the headquarters
• Decision Support and Data Warehousing

Common architectures of ogg include one-to-one, one-to-many, many-to-one, cascade, and bidirectional structures. One of the biggest features of ogg is that it can provide non-stop data synchronous replication, and supports cross-database (version, platform).

GoldenGate Basic Architecture

components	explain
Extract	The extraction process of GoldenGate software, also known as the Capture process, is generally used to extract database logs to capture data changes or transfer data in the local queue to the target end. The role of Extract can be divided into two stages: ①Initial data loading stage: In the initial data loading stage, the extract process directly extracts data from the data table at the source. ②Synchronous change capture phase: After the initialization data is completed, the extract process is responsible for capturing source data changes (DML and DDL)
Replicator	The delivery process of GoldenGate software, also known as the Delivery process, is used to convert the data changes in the queue file into SQL and apply them to the target library
Data Pump	Specifically refers to the Extract process that transfers the data in the local queue to the target end, which is different from the main Extract process that reads logs. The pump process runs on the source end of the database, and its function is to send the local trail files generated by the source end to the target end in the form of data blocks through the TCP/IP protocol. This is usually the recommended method. The essence of the Pump process is a special form of the extract process. If the trail file is not used, the extract process will directly deliver the data to the target end to generate a remote trail file after extracting the data.
Trail	GoldenGate's queue file, which stores data changes such as additions, deletions, and modifications, is stored in its proprietary format
Manager	The manger process is the control process of goldengate, which runs on the source and target respectively. Its main function is to start, monitor, and restart other goldengate processes, report errors and events, allocate data storage space, and issue threshold reports. It should be noted that there is only one manger process at the source and target.
Collector	The collector process corresponds to the pump process. It runs on the target side. You don’t need to pay too much attention to this process, because in the actual operation process, we don’t need to configure it, so it is transparent to us. Collector runs on the target side and its task is to reassemble the data delivered by extract/pump into a fortune trail file

2 Enter the introduction of this application scenario

Due to the particularity of the internal and external networks, it is impossible to directly transmit and push data through the network

​The external network cannot access the internal network, and the internal network can access the external network through a port opened on the external network

solution:

​Set the directory where the Trail file is located as a shared space through

operating environment

Different versions of GoldenGate require different environments and operations. It is recommended to operate according to the operating environment in this article. Specific operating environment requirements,

See the official instructions for details: Oracle Fusion Middleware Supported System Configurations

Three database settings

Install Oracle

Link: install Oracle in windows environment

Enable OGG

ALTER SYSTEM SET ENABLE_GOLDENGATE_REPLICATION = TRUE;

Open archive log

Note: Archived logs take up a lot of space, and you can set a scheduled task to clean them up.

1. Check to see if archive mode is turned on.

select log_mode from v$database;

2. Open the archive mode in the mount state.

-- 查看自动归档状态和归档进程
archive log list;
-- 关闭数据库
shutdown immediate;
-- 启动到mount状态
startup mount;
-- 开启归档模式
alter database archivelog;
-- 开启数据库
alter database open;

3. Check again to see if the archive mode is turned on.

Enable mandatory logging

1. Check whether force logging is enabled.

select force_logging from v$database;

2. Enable force logging.

alter database force logging;

Enable Minimal Additional Logging

1. Check whether the supplemental log is enabled.

select supplemental_log_data_min from v$database;

2. Open the supplemental log.

alter database add supplemental log data;

3. Switch the log to make the settings take effect.

alter system switch logfile;

close recycle bin

1. Check to see if the recycle bin is enabled.

show parameter recyclebin

2. Close the Recycle Bin.

alter system set recyclebin=off;

create user

Note: Oracle 12C introduces new features of CDB and PDB. The PDB mode is used here.

To create a user in the PDB, the user name does not need to start with C##, and the user is created as a local user.

To create a new user in CDB, the user name needs to start with C##, and the created is a common user (Common User)

cmd
sqlplus /nolog
conn / as sysdba

-- 查看当前连接数据库： 返回cdb或者pdb名称
show con_name;

-- 创建用户
create user c##ggs identified by 123456 default tablespace users temporary tablespace temp;
GRANT CREATE SESSION,CONNECT,RESOURCE,DBA TO c##GGS;
-- 修改任何表的权限
GRANT ALTER ANY TABLE TO c##GGS;
-- 修改系统参数的权限
GRANT ALTER SYSTEM TO c##GGS;
-- 查询任何字典表、任何事务的权限
GRANT SELECT ANY DICTIONARY,SELECT ANY TRANSACTION TO c##GGS;
-- 授予“采集模式”相关的权限，CAPTURE-采集模式，APPLY-复制模式，*-两者
EXECUTE DBMS_GOLDENGATE_AUTH.GRANT_ADMIN_PRIVILEGE('c##ggs','APPLY');

# 切换模式
alter session set container=ORCLPDB; 
# 开启
alter pluggable database open; 

-- 切换会话到指定数据库
alter session set container=cdb$root; --切换会话到CDB
alter session set container=ORCLPDB; --切换会话到PDB

-- 创建用户
create user ggs identified by 123456 default tablespace users temporary tablespace temp;
GRANT CREATE SESSION,CONNECT,RESOURCE,DBA TO GGS;
-- 修改任何表的权限
GRANT ALTER ANY TABLE TO GGS;
-- 修改系统参数的权限
GRANT ALTER SYSTEM TO GGS;
-- 查询任何字典表、任何事务的权限
GRANT SELECT ANY DICTIONARY,SELECT ANY TRANSACTION TO GGS;
-- 授予“采集模式”相关的权限，CAPTURE-采集模式，APPLY-复制模式，*-两者
EXECUTE DBMS_GOLDENGATE_AUTH.GRANT_ADMIN_PRIVILEGE('ggs','APPLY');

Configure from database

sqlplus /nolog
conn / as sysdba
-- 启用OGG
ALTER SYSTEM SET ENABLE_GOLDENGATE_REPLICATION = TRUE;
-- 关闭数据库
shutdown immediate;
-- 启动到mount状态
startup mount;
-- 开启归档模式
alter database archivelog;
-- 开启数据库
alter database open;

create user

create user c##ggs identified by 123456 default tablespace users temporary tablespace temp;
GRANT CREATE SESSION,CONNECT,RESOURCE,DBA TO c##GGS;
-- 修改任何表的权限
GRANT ALTER ANY TABLE TO c##GGS;
-- 修改系统参数的权限
GRANT ALTER SYSTEM TO c##GGS;
-- 查询任何字典表、任何事务的权限
GRANT SELECT ANY DICTIONARY,SELECT ANY TRANSACTION TO c##GGS;
-- 授予“采集模式”相关的权限，CAPTURE-采集模式，APPLY-复制模式，*-两者
EXECUTE DBMS_GOLDENGATE_AUTH.GRANT_ADMIN_PRIVILEGE('c##ggs','APPLY');

# 切换模式
alter session set container=ORCLPDB; 
# 开启
alter pluggable database open; 

-- 切换会话到指定数据库
alter session set container=cdb$root; --切换会话到CDB
alter session set container=ORCLPDB; --切换会话到PDB

-- 创建用户
create user ggs identified by 123456 default tablespace users temporary tablespace temp;
GRANT CREATE SESSION,CONNECT,RESOURCE,DBA TO GGS;
-- 修改任何表的权限
GRANT ALTER ANY TABLE TO GGS;
-- 修改系统参数的权限
GRANT ALTER SYSTEM TO GGS;
-- 查询任何字典表、任何事务的权限
GRANT SELECT ANY DICTIONARY,SELECT ANY TRANSACTION TO GGS;
-- 授予“采集模式”相关的权限，CAPTURE-采集模式，APPLY-复制模式，*-两者
EXECUTE DBMS_GOLDENGATE_AUTH.GRANT_ADMIN_PRIVILEGE('ggs','APPLY');

Four Sftp service construction

Link: Measured Summary Mount Remote Folder Scheme smb ftp sftp nfs webdav

Install freeSSHd.exe on the external network server

Link: Build sftp server on windows --freesshd

Note:
If the password is correct and you can’t connect, it’s useless to run as an administrator
. Please kill FreeSSHDService.exe in the process, and then run FreeSSHDService with administrator privileges in the installation directory.
Do not use the default 22, change it to 10022

create user

Configure SSH

Select the server default path in sftp

Authentication Set
the Password authentication column to select Required, otherwise the client will always enter the wrong password when connecting

server status 中 run ssh servers

Link: Use SSHFS under Windows to mount remote server directories through the SSH protocol

Method 1: winfsp + sshfs-win (recommended)

sshfs-win：https://github.com/billziss-gh/sshfs-win/releases

winfsp：https://github.com/billziss-gh/winfsp/releases

SSHFS-Win Manager：https://github.com/evsar3/sshfs-win-manager/releases (GUI，可选)

Just install it in sequence

Method 2: win-sshfs + Dokan (old, not recommended)

Five GoldenGate installation

Master and slave servers must be installed.

Install Visual C++

1. The Visual C++ Redistributable Package can be downloaded from the Microsoft official website . Two versions, 2010 and 2013, need to be installed here.

2. After downloading, double-click to run the installation.

Install GoldenGate

1. GoldenGate version 12 can be downloaded from Oracle China.
2. Copy the installation package to the server, and run setup.exe as an administrator after decompression.

3. Select the corresponding GoldenGate version according to the database version, here is the 11g version.

4. Specify the installation path, database home directory, and port. If you are using the client to connect to the database remotely, you don’t need to check here

Set up GoldenGate

Master server settings
Create a directory
under the ogg installation directory, run ggsci.exe as an administrator, and execute the CREATE SUBDIRS command to create an Oracle
GoldenGate working directory.

The catalog description is as follows:

dirchk：存放检查点（Checkpoint）文件
dirdat：存放Trail与Extract文件，以后详述
dirdef：通过DEFGEN工具生成的源或目标的数据定义文件
dirpcs：存放进程状态文件
dirprm：存放参数文件
dirrpt：存放进程报告文件
dirsql：存放SQL脚本文件
dirtmp：当事务所需要的内存超过已分配内存时，缺省存储于此。

Install Manager as a system service

1. Run ggsci.exe as administrator.
2. Execute the following command to create a GLOBALS file.

EDIT PARAMS ./GLOBALS

3. Add the following to the GLOBALS file. GGMGR is a custom service name.

MGRSERVNAME GGMGR

4. After saving and exiting, use the install command in the root directory of GoldenGate to install the service.

INSTALL ADDSERVICE

set tracked table

1. Log in to the GoldenGate management user in GGSCI.

dblogin userid ggs,password 123456

Client remote connection can use:

dblogin userid [email protected]:1521/orclgbk,password 123456

2. Set the table to track. (You can specify specific objects, or use wildcards, such as *)

add trandata xxx.*

Configure the Manager management process

1. Use the edit params mgr command to add the following to the popup text:

port 7500
dynamicportlist 7501-7505
autorestart extract *,waitminutes 2,retries 5

2. Enter start mgr to start the mgr process.

Configure the Extract extraction process

1. Use the edit params eora command to add the following content in the pop-up text:
   exttrail: custom extraction file storage path. The last directory must be two characters, indicating the first
   two digits of the file name.
   To connect to the database remotely, you can use userid [email protected]:1521/orclgbk, password 123456

edit params eora

extract eora
userid c##ggs,password 123456
exttrail C:\ogg\dirdat\et
SOURCECATALOG orclpdb
table ggs.demo;

Note: SOURCECATALOG orclpdb is an additional configuration added by pdb mode

2. Enter the following command to start the eora process.

-- 1.从当前时间开始抽取
add extract eora,tranlog,begin now
-- 2.最后一个目录一定要是两个字符。要不然会提示“file portion must be twocharacters.”
add exttrail C:\app\Administrator\product\ogg123\dirdat\et,extract eora 
-- 3.启动eora进程
start eora

Configure the Data Pump push process (because network transmission is not possible, this step is not done in the internal and external network interaction)

Data pumps are used to read local paths and send data to remote paths over the network. Data Pump is optional but recommended.

1. In the GGSCI of the source system, use the edit params pump_so command to create a data-pump parameter file with the
   following content:

extract pump_so
userid ggs,password 123456
rmthost xx.x.xx.xxx, mgrport 7500
// 生成trail数据的目录
rmttrail C:\app\Administrator\product\ogg123\dirdat\rt
Table xxx.*;

Among them: pump_so is the name of the data pump process.

2. Configure push directory.

-- exttrailsource为主服务器的目录
add extract pump_so,exttrailsource C:\app\Administrator\product\ogg123\dirdat\et
-- rmttrail的地址为从服务器的目录
add rmttrail C:\app\Administrator\product\ogg123\dirdat\rt,extract pump_so

3. After the slave server is configured, start the pump_so process.

start pump_so

4. You can use the info all command to view the running status of the processes.

Set from server

Bind the remote directory disk of the master server to the Z: disk through sftp

Executing the same command as the master server is simply written here

Run ggsci.exe as administrator.

1.创建Oracle GoldenGate工作目录
CREATE SUBDIRS

2.创建 GLOBALS 文件
EDIT PARAMS ./GLOBALS

3. 在 GLOBALS 文件中添加以下内容。 GGMGR 是自定义的服务名称。
MGRSERVNAME GGMGR
//添加Checkpoint Table
checkpointtable ggs.checkpoint
ALLOWOUTPUTDIR Z:\dirdat

4. 保存退出后，在GoldenGate根目录下使用 install 命令安装服务。
INSTALL ADDSERVICE

set tracked table

Log in to the GoldenGate admin user in GGSCI, and set the tables to track.

dblogin userid [email protected]:1521/orclpdb,password 123456

1.设置要跟踪的表
add trandata ggs.demo
add checkpointtable ggs.checkpoint

Configure the Manager management process

edit params mgr

1. Use the edit params mgr command to add the following to the popup text:

port 7500
dynamicportlist 7501-7505
autostart er *
autorestart extract *,waitminutes 2,retries 5
lagreporthours 1
laginfominutes 3
lagcriticalminutes 5
purgeoldextracts Z:\dirdat\et,usecheckpoints,minkeepdays 3

Start the mgr process

start mgr

Configure the Replicat process

edit params repl

1. Use the edit params repl command to add the following content in the pop-up text:
When configuring object mapping, it is recommended to specify a specific object name instead of wildcard *. Because when the object is not found, the process stops.

replicat repl
userid [email protected]:1521/orclpdb,password 123456
assumetargetdefs
reperror default,discard
discardfile ./dirrpt/repl.dsc,append,megabytes 50
// PDB模式的配置
SOURCECATALOG ORCLPDB
MAP ggs.demo, target ggs.demo;

2. Enter the following command to start the repl process.

add replicat repl,exttrail Z:\dirdat\et,checkpointtable ggs.checkpoint
start repl

Firewall settings (for internal and external networks, the external network will open a special port for the internal network)

Allow ports 7500-7505 in the firewall inbound rules.

Six GoldenGate uninstall

1. Run Run GGSCI
2. Stop all Oracle GoldenGate processes.

STOP ER *

3. Stop the Manager process

STOP MANAGER

4. Run deinstall.sh

UNIX and Linux: OGG_home/deinstall/deinstall.sh
Windows: OGG_home/deinstall/deinstall.bat

Uninstall failure prompt

windows : C:\Program Files\Oracle\Inventory\ContentsXML

Extraction of configuration DDL

DDL（data definition language）是数据定义语言：
DDL比DML要多，主要的命令有CREATE、ALTER、DROP等，
DDL主要是用在定义或改变表（TABLE）的结构，数据类型，
表之间的链接和约束等初始化工作上，他们大多在建立表时使用。
extract 里，默认是不支持 DDL 的抽取    
需要配置DDL 的抽取  这样表和字段的变更也能同步了

common problem

OGG Checkpoint table main table and auxiliary table Checkpoint_LOX

Incompatible record (logical EOF) in trail error resolution

Enter the logdump command line

Run ./logdump directly in the GoldenGate installation directory

windows ‪ logdump.exe

open dirdat/xxxx  --打开trail文件
ghdr on
pos 1775  --使用上边报错rba号
n          --查看下一条记录
sfh prev   --查看上一条记录header

显示中 Len 312 RBA 2706720 --记录此rba，接下来从这个点开始启动replicat进程

## 下面操作ggsci
GGSCI alter LR_BF9_A , extrba  2706720

info all

stopped

ABENDED force closed

RUNNING running

ogg log view

There is an error log in the root directory of GoldenGate installation, the file name is user name + err.log, such as ggserr.log, you can not only check the error message, but also see the status of data synchronization.

redesignated extraction

Link: Two ways to switch trail files manually in OGG

1) Use the alter extract ext1 etrollover command to generate a new trail file. The extract and pump processes must be stopped first, and the seqno sequence numbers of the datapump and replicat processes must be specified, otherwise the process will not automatically synchronize data from the new trail file.
2) SEND extract ext1, ROLLOVER name to generate a new trail file does not need to stop the process, and does not need to know the seqno sequence number, you can continue to synchronize data.
3) The alter extract ext1 etrollover application scenario generally regenerates the trail file when initializing data or starting to extract data at a specified time point, and specifies to extract data from the new trail file number to prevent interference from previous trail files, SEND extract ext1, ROLLOVER Generally, when the application modifies the size of the trail file, the online modification will not affect the normal operation of the business.

--前滚重新生成一个新的队列文件
alter extract eora etrollover
 
--从指定时间重新抓取（重新抓取数据前提：归档文件没有删除）
ALTER EXTRACT eora, TRANLOG, BEGIN 2022-09-05 06:00
 
--重置抽取进程，本地文件序列号从0开始生成。
alter extract eora,extseqno 0,extrba 0
 

--重置读取进程，重新从0号trial文件开始读取。
alter replicat repl,extseqno 4,extrba 0

-- 重新应用数据
start replicat repl nofilterduptransactions

pump meets ogg-01052

Link: 【ogg combat】pump meets ogg-01052

No recovery is required for target file C:\ogg\dirdat\rt000000000, at RBA 0
ext No problem, the pump is not delivered to the target, the path to create the pump is no problem, just initialize the serial number
I encountered ogg- 01052 should not find the source file. The path must be fine when creating plzsi, so it is the problem of the file name. The sequence of the file name is 00000, and the dirdat file starts from 1814. It may be that the seq of plzsi is reset to 0 No, no log, can't find out

OGG Trail analysis tool - Logdump

Link: OGG Trail analysis tool - Logdump

Running ggsci.exe prompts that OCI.dll cannot be found

Oracle client or server is not installed.

When the table does not exist, the Replicat process ends abnormally

integrated integrated mode settings

1. <=11.2.0.4 version, need to solve the problem of Patch 17030189 is required…

cd C:\app\Administrator\product\ogg123
sqlplus sys/[email protected]:1521/orclgbk as sysdba
SQL> @prvtlmpg.plb
输入创建的GoldenGate管理用户

2. Register extract process and use integrated mode

register extract eora database
add extract eora,integrated tranlog,begin now
add exttrail C:\app\Administrator\product\ogg123\dirdat\et,extract eora
start eora

3. Register the replicat process and use the integrated mode

add replicat repl,integrated,exttrail
C:\app\Administrator\product\ogg123\dirdat\rt, NODBCHECKPOINT
dblogin userid ggs,password 123456
register replicat repl database
start repl