The ogg micro-service version can configure the extraction and replication process on the interface, without entering the shell for configuration, and more information can be seen on the graphical interface.
system structure
Install ogg for oracle 19C on the source side, and
ogg for bigdata 21C
kafka 2.2
database on the target side:
all software of 19C is installed on the same server (Oracle Linux7)
Source Configuration - Extraction Process
Sample data configuration
Select the hr related table that comes with the database as an example demonstration (the related table is configured with a primary key), which is more in line with the specification.
sqlplus / as sysdba
alter session set "_ORACLE_SCRIPT"=true;
alter session set container=ORCLPDB1;
@?/demo/schema/human_resources/hr_main.sql
INSTALL OGG FOR ORACLE
#配置环境变量
# User specific aliases and functions
export ORACLE_BASE=/opt/oracle
export ORACLE_HOME=/opt/oracle/product/19c/dbhome_1
export ORACLE_SID=ORCLCDB
export PATH=$PATH:$ORACLE_HOME/bin
export JAVA_HOME=/home/oracle/jdk1.8.0_351
export PATH=$JAVA_HOME/bin:$PATH
export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/amd64/server:$LD_LIBRARY_PATH
#别名配置
alias ggsci='rlwrap ggsci'
alias sqlplus='rlwrap sqlplus'
alias rman='rlwrap rman'
alias lsnrctl='rlwrap lsnrctl'
alias asmcmd='rlwrap asmcmd'
alias adrci='rlwrap adrci'
#执行安装
sudo unzip /vagrant/$ORACLE_GG_SETUP_FILE -d /u01/ogg-installer
cp /vagrant/ora-response/ogg_install.rsp.tmpl /tmp/oggresponse.rsp
sed -i -e "s|###ORACLE_BASE###|$ORACLE_BASE|g" /tmp/oggresponse.rsp
sed -i -e "s|###ORACLE_HOME###|$ORACLE_HOME|g" /tmp/oggresponse.rsp
chown oracle:oinstall -R /u01/ogg-installer
chown oracle:oinstall -R /u01/ogg
#su -l oracle -c "/u01/ogg-installer/fbo_ggs_Linux_x64_shiphome/Disk1/runInstaller -silent -waitforcompletion -responseFile /tmp/oggresponse.rsp"
su -l oracle -c "/u01/ogg-installer/fbo_ggs_Linux_x64_Oracle_shiphome/Disk1/runInstaller -silent -waitforcompletion -responseFile /tmp/oggresponse.rsp"
echo "export LD_LIBRARY_PATH=\$ORACLE_HOME/lib:/lib:/usr/lib" >> /home/oracle/.bashrc
rm -rf /u01/ogg-installer
rm /tmp/oggresponse.rsp
echo 'INSTALLER: Oracle Golden Gate Installed'
Configuration extraction process
#1 Go to Oracle golden gate installation directory
cd /u01/ogg
#2 Open golden gate console.
./ggsci
#3 Start manager.
> start mgr
#4 Log in to the database.
> DBLOGIN USERID SYSTEM@localhost:1521/ORCLCDB PASSWORD [password copied while installation]
#5 Register Extract.
> REGISTER EXTRACT EXT1 DATABASE CONTAINER (ORCLPDB1)
#6 Enable schema-level supplemental logging for a table.
> ADD SCHEMATRANDATA ORCLPDB1.HR ALLCOLS
#7 Create an Extract group.
> ADD EXTRACT EXT1, INTEGRATED TRANLOG, BEGIN NOW
#9 Create a trail for online processing on the local system and Associate it with an Extract group.
> ADD EXTTRAIL ./dirdat/lt EXTRACT EXT1
#10 Create EXT1 parameter file and paste the content in the file.
> EDIT PARAM EXT1
EXTRACT EXT1
USERID SYSTEM@ORCLCDB, PASSWORD [password copied during installation]
EXTTRAIL ./dirdat/lt
SOURCECATALOG ORCLPDB1
TABLE HR.*;
#11 Start Extract EXT1.
> start ext1
#12 View status of manager and ext1.
> info all
Program Status Group Lag at Chkpt Time Since Chkpt
MANAGER RUNNING
EXTRACT RUNNING EXT1 00:00:00 00:00:00
target configuration - replication process
Install the microservice version
Follow the steps. First extract the compressed package to a temporary directory, then switch to the temporary directory to execute the installation process
#配置环境变量
export OGG_HOME=/home/oggbd/ogg_home1
export JAVA_HOME=/home/oracle/jdk1.8.0_351
export PATH=$JAVA_HOME/bin:$PATH
export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/amd64/server:$LD_LIBRARY_PATH
#执行安装
#OGG经典架构需要一个目录即OGG_HOME，微服务需要３个路径
#一个是OGG_HOME,软件安装路径与经典架构保持一致，简称MA；
#一个是服务管理路径，微服务架构采用服务方式管理，简称SM,可以管理多个部署.例如for oracle 11g、oracle 12c等；
#一个是部署路径，微服务采购云操作方式，所有部署与管理都是通过web其中管理，简称DEPLOY。
[oggbd@oracle-goldengate-1910-vagrant Disk1]$ pwd
/home/oggbd/temp/ggs_Linux_x64_BigData_services_shiphome/Disk1
[oggbd@oracle-goldengate-1910-vagrant Disk1]$ ./r
response/ runInstaller
[oggbd@oracle-goldengate-1910-vagrant Disk1]$ ./runInstaller
[oggbd@oracle-goldengate-1910-vagrant ~]$ export OGG_HOME=ogg_home2
[oggbd@oracle-goldengate-1910-vagrant ~]$ cd $OGG_HOME
[oggbd@oracle-goldengate-1910-vagrant ogg_home2]$ cd bin
[oggbd@oracle-goldengate-1910-vagrant bin]$ ./oggca.sh
At this place, pay attention to LD_LIBRARY_PATH, do not import repeatedly, otherwise there will be class conflicts later.
Execute the configuration copy process
Use localhost:8002 to enter, click the + sign
#custom_kafka_producer.properties 内容为:
bootstrap.servers=localhost:9092
acks=1
reconnect.backoff.ms=1000
value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
key.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
# 100KB per partition
batch.size=16384
linger.ms=0
Created successfully:
view related resource information:
view kafka
#查看创建的topic
[root@oracle-goldengate-1910-vagrant bin]# ./kafka-topics.sh --zookeeper localhost:2181 --list
REGIONS
RKAFKA.rptmySchemaTopic
__consumer_offsets
mySchemaTopic
#查看同步过来的数据
[root@oracle-goldengate-1910-vagrant bin]# ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic REGIONS --from-beginning
&ORCLPDB1.HR.REGIONSI42022-11-15 16:54:43.00000042022-11-15T17:07:12.475000(00000000000000001783REGION_IDFOO
&ORCLPDB1.HR.REGIONSI42022-11-15 17:08:46.00000042022-11-15T17:08:49.231000(00000000000000001934REGION_IDHFOO1
{
"table":"ORCLPDB1.HR.REGIONS","op_type":"I","op_ts":"2022-11-15 17:23:24.000000","current_ts":"2022-11-15T17:23:30.319000","pos":"00000000000000002081","after":{
"REGION_ID":49,"REGION_NAME":"49"}}
{
"table":"ORCLPDB1.HR.REGIONS","op_type":"I","op_ts":"2022-11-15 16:54:43.000000","current_ts":"2022-11-17 17:53:57.390000","pos":"00000000000000001783","after":{
"REGION_ID":47,"REGION_NAME":"FOO"}}
{
"table":"ORCLPDB1.HR.REGIONS","op_type":"I","op_ts":"2022-11-15 17:08:46.000000","current_ts":"2022-11-17 17:54:00.497000","pos":"00000000000000001934","after":{
"REGION_ID":48,"REGION_NAME":"FOO1"}}
{
"table":"ORCLPDB1.HR.REGIONS","op_type":"I","op_ts":"2022-11-15 17:23:24.000000","current_ts":"2022-11-17 17:54:00.516000","pos":"00000000000000002081","after":{
"REGION_ID":49,"REGION_NAME":"49"}}
{
"table":"ORCLPDB1.HR.REGIONS","op_type":"I","op_ts":"2022-11-15 16:54:43.000000","current_ts":"2022-11-17 17:54:01.712000","pos":"00000000000000001783","after":{
"REGION_ID":47,"REGION_NAME":"FOO"}}
{
"table":"ORCLPDB1.HR.REGIONS","op_type":"I","op_ts":"2022-11-15 17:08:46.000000","current_ts":"2022-11-17 17:54:01.837000","pos":"00000000000000001934","after":{
"REGION_ID":48,"REGION_NAME":"FOO1"}}
{
"table":"ORCLPDB1.HR.REGIONS","op_type":"I","op_ts":"2022-11-15 17:23:24.000000","current_ts":"2022-11-17 17:54:01.896000","pos":"00000000000000002081","after":{
"REGION_ID":49,"REGION_NAME":"49"}}
References
Official Document
The Ultimate Guide To Setup Golden Gate On Vagrant Box
OGG Microservice Deployment Configuration
[ORACLE] OGG-Microservice Architecture Preliminary Study 1 (Environment Installation)
Using Ogg to Realize the Increment from Oracle to Kafka
Use OGG Microservice to Quickly and Two-Way Synchronize RDS for MySQL Database
OGG|Oracle GoldenGate Microservice Architecture
Xu Jing Column
Kangaroo Cloud Financial Delivery Team-Use ogg to realize real-time synchronization of incremental data from oracle to kafka