GoldenGate实时投递数据到大数据平台（1）-MongoDB
mongodb安装
安装
linux下可使用apt-get install mongodb-server 或 yum install mongodb-server 进行安装。
也可以在windows上安装mongodb，只要网络连通，数据就可以从关系型数据库投递到mongodb。详细安装文档可网络搜索。
启动mongodb
先创建一个数据目录
mkdir /opt/mongodb-data
然后启动DB服务, --rest是启动restful web
mongod --dbpath /opt/mongodb-data -–reset
可以通过restful admin web访问mongodb，如下
http://192.168.89.129:28017/
基本使用
插入示例数据并查询
简单统计test db.
OGG安装测试
以下使用ogg12.3 for big data介质进行数据投递，下载地址：
http://www.oracle.com/technetwork/middleware/goldengate/downloads/index.html
ogg for bigdata需要JDK 1.8及以上，安装完成之后，需要设置LD_LIBRARY_PATH环境变量
export LD_LIBRARY_PATH=/u01/jdk1.8.0_111/jre/lib/amd64/server
OGG for bigdata解压即可安装，安装完成之后，拷贝安装目录下自带的示例到dirprm目录。
[oracle@ol73 mongodb]$ pwd
/u01/ogg4bd_12.3/AdapterExamples/big-data/mongodb
[oracle@ol73 mongodb]$ cp * /u01/ogg4bd_12.3/dirprm/
修改dirprm/mongo.props文件，主要调整mongodb jdbc jar的路径和client URI
| gg.handlerlist=mongodb gg.handler.mongodb.type=mongodb #The following handler properties are optional. #Please refer to the Oracle GoldenGate for BigData documentation #for details about the configuration. gg.handler.mongodb.clientURI=mongodb://192.168.89.129:27017/ goldengate.userexit.timestamp=utc goldengate.userexit.writers=javawriter javawriter.stats.display=TRUE javawriter.stats.full=TRUE gg.log=log4j gg.log.level=INFO gg.report.time=30sec #Path to MongoDB Java driver. gg.classpath=/u01/ogg4bd_12.3/mongo-java-driver-3.4.0.jar javawriter.bootoptions=-Xmx512m -Xms32m -Djava.class.path=.:ggjava/ggjava.jar:./dirprm |
OGG投递进程参数，可直接使用示例自带的参数文件
GGSCI (ol73) 1> view param rmongo
| REPLICAT rmongo -- Trail file for this example is located in "AdapterExamples/trail" directory -- Command to add REPLICAT -- add replicat rmongo, exttrail AdapterExamples/trail/tr TARGETDB LIBFILE libggjava.so SET property=dirprm/mongo.props REPORTCOUNT EVERY 1 MINUTES, RATE GROUPTRANSOPS 1000 MAP QASOURCE.*, TARGET QASOURCE.*; |
使用OGG自带的增量数据创建投递进程
GGSCI>add replicat rmongo, exttrail AdapterExamples/trail/tr
启动投递进程
GGSCI (ol73) 3> start rmongo
Sending START request to MANAGER ...
REPLICAT RMONGO starting
统计变更数据
GGSCI (ol73) 9> stats rmongo, total
| Sending STATS request to REPLICAT RMONGO ... Start of Statistics at 2017-12-20 17:38:34. Replicating from QASOURCE.TCUSTMER to QASOURCE.TCUSTMER: *** Total statistics since 2017-12-20 17:38:18 *** Total inserts 5.00 Total updates 1.00 Total deletes 0.00 Total discards 0.00 Total operations 6.00 Replicating from QASOURCE.TCUSTORD to QASOURCE.TCUSTORD: *** Total statistics since 2017-12-20 17:38:18 *** Total inserts 5.00 Total updates 3.00 Total deletes 2.00 Total discards 0.00 Total operations 10.00 End of Statistics. |
可以看到变更数据中有2张表，包括有insert/update/delete等操作。
在数据库端验证结果
$mongo
查看数据库，已经看到有新插入的QASOURCE
> show dbs
QASOURCE 0.078GB
admin (empty)
local 0.078GB
test 0.078GB
> use QASOURCE
switched to db QASOURCE
简单查看一下数据库的基本信息
> db.stats()
| { "db" : "QASOURCE", "collections" : 4, "objects" : 15, "avgObjSize" : 116.26666666666667, "dataSize" : 1744, "storageSize" : 32768, "numExtents" : 4, "indexes" : 2, "indexSize" : 16352, "fileSize" : 67108864, "nsSizeMB" : 16, "dataFileVersion" : { "major" : 4, "minor" : 5 }, "extentFreeList" : { "num" : 0, "totalSize" : 0 }, "ok" : 1 } |
查看一下数据库中的集合
> db.getCollectionInfos()
| [ { "name" : "TCUSTMER" }, { "name" : "TCUSTORD" }, { "name" : "system.indexes" } ] |
> db.getCollectionNames()
[ "TCUSTMER", "TCUSTORD", "system.indexes" ]
>
> db.getCollectionNames()
[ "TCUSTMER", "TCUSTORD", "system.indexes" ]
查看TCUSTMER中的数据
> db.TCUSTMER.find()
| { "_id" : "WILL", "CITY" : "SEATTLE", "STATE" : "WA", "CUST_CODE" : "WILL", "NAME" : "BG SOFTWARE CO." } { "_id" : "JANE", "CITY" : "DENVER", "STATE" : "CO", "CUST_CODE" : "JANE", "NAME" : "ROCKY FLYER INC." } { "_id" : "DAVE", "CITY" : "TALLAHASSEE", "STATE" : "FL", "CUST_CODE" : "DAVE", "NAME" : "DAVE'S PLANES INC." } { "_id" : "BILL", "CITY" : "DENVER", "STATE" : "CO", "CUST_CODE" : "BILL", "NAME" : "BILL'S USED CARS" } { "_id" : "ANN", "CITY" : "NEW YORK", "STATE" : "NY", "CUST_CODE" : "ANN", "NAME" : "ANN'S BOATS" } |
查找CITY=’DENVER’的记录
> db.TCUSTMER.find({"CITY":"DENVER"})
| { "_id" : "JANE", "CITY" : "DENVER", "STATE" : "CO", "CUST_CODE" : "JANE", "NAME" : "ROCKY FLYER INC." } { "_id" : "BILL", "CITY" : "DENVER", "STATE" : "CO", "CUST_CODE" : "BILL", "NAME" : "BILL'S USED CARS" } |
测试完成。
mongodb安装
安装
linux下可使用apt-get install mongodb-server 或 yum install mongodb-server 进行安装。
也可以在windows上安装mongodb，只要网络连通，数据就可以从关系型数据库投递到mongodb。详细安装文档可网络搜索。
启动mongodb
先创建一个数据目录
mkdir /opt/mongodb-data
然后启动DB服务, --rest是启动restful web
mongod --dbpath /opt/mongodb-data -–reset
可以通过restful admin web访问mongodb，如下
http://192.168.89.129:28017/
基本使用
插入示例数据并查询
简单统计test db.
OGG安装测试
以下使用ogg12.3 for big data介质进行数据投递，下载地址：
http://www.oracle.com/technetwork/middleware/goldengate/downloads/index.html
ogg for bigdata需要JDK 1.8及以上，安装完成之后，需要设置LD_LIBRARY_PATH环境变量
export LD_LIBRARY_PATH=/u01/jdk1.8.0_111/jre/lib/amd64/server
OGG for bigdata解压即可安装，安装完成之后，拷贝安装目录下自带的示例到dirprm目录。
[oracle@ol73 mongodb]$ pwd
/u01/ogg4bd_12.3/AdapterExamples/big-data/mongodb
[oracle@ol73 mongodb]$ cp * /u01/ogg4bd_12.3/dirprm/
修改dirprm/mongo.props文件，主要调整mongodb jdbc jar的路径和client URI
| gg.handlerlist=mongodb gg.handler.mongodb.type=mongodb #The following handler properties are optional. #Please refer to the Oracle GoldenGate for BigData documentation #for details about the configuration. gg.handler.mongodb.clientURI=mongodb://192.168.89.129:27017/ goldengate.userexit.timestamp=utc goldengate.userexit.writers=javawriter javawriter.stats.display=TRUE javawriter.stats.full=TRUE gg.log=log4j gg.log.level=INFO gg.report.time=30sec #Path to MongoDB Java driver. gg.classpath=/u01/ogg4bd_12.3/mongo-java-driver-3.4.0.jar javawriter.bootoptions=-Xmx512m -Xms32m -Djava.class.path=.:ggjava/ggjava.jar:./dirprm |
OGG投递进程参数，可直接使用示例自带的参数文件
GGSCI (ol73) 1> view param rmongo
| REPLICAT rmongo -- Trail file for this example is located in "AdapterExamples/trail" directory -- Command to add REPLICAT -- add replicat rmongo, exttrail AdapterExamples/trail/tr TARGETDB LIBFILE libggjava.so SET property=dirprm/mongo.props REPORTCOUNT EVERY 1 MINUTES, RATE GROUPTRANSOPS 1000 MAP QASOURCE.*, TARGET QASOURCE.*; |
使用OGG自带的增量数据创建投递进程
GGSCI>add replicat rmongo, exttrail AdapterExamples/trail/tr
启动投递进程
GGSCI (ol73) 3> start rmongo
Sending START request to MANAGER ...
REPLICAT RMONGO starting
统计变更数据
GGSCI (ol73) 9> stats rmongo, total
| Sending STATS request to REPLICAT RMONGO ... Start of Statistics at 2017-12-20 17:38:34. Replicating from QASOURCE.TCUSTMER to QASOURCE.TCUSTMER: *** Total statistics since 2017-12-20 17:38:18 *** Total inserts 5.00 Total updates 1.00 Total deletes 0.00 Total discards 0.00 Total operations 6.00 Replicating from QASOURCE.TCUSTORD to QASOURCE.TCUSTORD: *** Total statistics since 2017-12-20 17:38:18 *** Total inserts 5.00 Total updates 3.00 Total deletes 2.00 Total discards 0.00 Total operations 10.00 End of Statistics. |
可以看到变更数据中有2张表，包括有insert/update/delete等操作。
在数据库端验证结果
$mongo
查看数据库，已经看到有新插入的QASOURCE
> show dbs
QASOURCE 0.078GB
admin (empty)
local 0.078GB
test 0.078GB
> use QASOURCE
switched to db QASOURCE
简单查看一下数据库的基本信息
> db.stats()
| { "db" : "QASOURCE", "collections" : 4, "objects" : 15, "avgObjSize" : 116.26666666666667, "dataSize" : 1744, "storageSize" : 32768, "numExtents" : 4, "indexes" : 2, "indexSize" : 16352, "fileSize" : 67108864, "nsSizeMB" : 16, "dataFileVersion" : { "major" : 4, "minor" : 5 }, "extentFreeList" : { "num" : 0, "totalSize" : 0 }, "ok" : 1 } |
查看一下数据库中的集合
> db.getCollectionInfos()
| [ { "name" : "TCUSTMER" }, { "name" : "TCUSTORD" }, { "name" : "system.indexes" } ] |
> db.getCollectionNames()
[ "TCUSTMER", "TCUSTORD", "system.indexes" ]
>
> db.getCollectionNames()
[ "TCUSTMER", "TCUSTORD", "system.indexes" ]
查看TCUSTMER中的数据
> db.TCUSTMER.find()
| { "_id" : "WILL", "CITY" : "SEATTLE", "STATE" : "WA", "CUST_CODE" : "WILL", "NAME" : "BG SOFTWARE CO." } { "_id" : "JANE", "CITY" : "DENVER", "STATE" : "CO", "CUST_CODE" : "JANE", "NAME" : "ROCKY FLYER INC." } { "_id" : "DAVE", "CITY" : "TALLAHASSEE", "STATE" : "FL", "CUST_CODE" : "DAVE", "NAME" : "DAVE'S PLANES INC." } { "_id" : "BILL", "CITY" : "DENVER", "STATE" : "CO", "CUST_CODE" : "BILL", "NAME" : "BILL'S USED CARS" } { "_id" : "ANN", "CITY" : "NEW YORK", "STATE" : "NY", "CUST_CODE" : "ANN", "NAME" : "ANN'S BOATS" } |
查找CITY=’DENVER’的记录
> db.TCUSTMER.find({"CITY":"DENVER"})
| { "_id" : "JANE", "CITY" : "DENVER", "STATE" : "CO", "CUST_CODE" : "JANE", "NAME" : "ROCKY FLYER INC." } { "_id" : "BILL", "CITY" : "DENVER", "STATE" : "CO", "CUST_CODE" : "BILL", "NAME" : "BILL'S USED CARS" } |
测试完成。