Oracle+Kafka+Flink(CDC捕获) 部署实时同步数据

1.环境
网盘链接，有对应的组件环境以及个人手册记录：

https://pan.baidu.com/s/1dKca6VpKzegvghedv4DEtA

提取码：zdwr
1.1、Kafka：kafka_2.13-2.8.0
1.2、数据库：Oracle11G。（Oracle开启归档）
1.3、计算引擎：Flink 1.13.1
1.4、Kafka Connector：建议使用1.6可以对ddl进行捕获
debezium-connector-mysql:

debezium-connector-mysql-1.6.0.Final-plugin.tar.gz
https://repo1.maven.org/maven2/io/debezium/debezium-connector-mysql/1.6.0.Final/debezium-connector-mysql-1.6.0.Final-plugin.tar.gz

debezium-connector-postgres:

debezium-connector-postgres-1.6.0.Final-plugin.tar.gz
https://repo1.maven.org/maven2/io/debezium/debezium-connector-postgres/1.6.0.Final/debezium-connector-postgres-1.6.0.Final-plugin.tar.gz

debezium-connector-oracle:

debezium-connector-oracle-1.6.0.Final-plugin.tar.gz
https://repo1.maven.org/maven2/io/debezium/debezium-connector-oracle/1.6.0.Final/debezium-connector-oracle-1.6.0.Final-plugin.tar.gz

2.实时同步
2.1、下载debezium-connector-oracle-1.6.0.Final-plugin.tar.gz并解压，安装在自己的服务器，我的安装目录是/home/debezium/
2.2、将debezium-connector-oracle 目录下得jar包都拷贝一份到${KAFKA_HOME}/libs中
2.3、Oracle需要下载客户端并把jar包复制到
${KAFKA_HOME}/libs
下载地址：https://download.oracle.com/otn_software/linux/instantclient/211000/instantclient-basic-linux.x64-21.1.0.0.0.zip
2.4、kafka环境修改
kafka安装目录：/home/kafka/kafka_2.13-2.8.0/
单机部署修改 [connect-standalone.properties]
集群部署修改 [connect-distributed.properties]
注：小编修改单机环境启动报错，最后无奈改为集群配置
具体如下
bootstrap.servers=172.16.50.22:9092
plugin.path=/home/debezium/debezium-connector-oracle
group.id=amirSync
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter
config.storage.topic=amir-connect-configs
offset.storage.topic=amir-connect-offsets
status.storage.topic=amir-connect-statuses
config.storage.replication.factor=1
offset.storage.replication.factor=1
status.storage.replication.factor=1
cleanup.policy=compact
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
rest.host.name=172.16.50.22
rest.port=8085

2.5、启动Kafka集群，设置环境变量

export KAFKA_LOG4J_OPTS=-Dlog4j.configuration=file:/home/kafka/kafka_2.13-2.8.0/config/connect-log4j.properties
./bin/connect-distributed.sh /home/kafka/kafka_2.13-2.8.0/config/connect-distributed.properties

2.6、提交Oracle-connector，监视Oracle数据库

curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" http://172.16.50.22:8085/connectors/ -d '
{
"name": "debezium-oracle-amir",
"config": {
"connector.class" : "io.debezium.connector.oracle.OracleConnector",
"tasks.max" : "1",
"database.server.name" : "XE",
"database.hostname" : "172.16.50.239",
"database.port" : "1521",
"database.user" : "MSCDW",
"database.password" : "MSCDW2104",
"database.dbname" : "XE",
"database.schema" : "MSCDW",
"database.connection.adapter": "logminer", 
"database.tablename.case.insensitive": "true",
"table.include.list" : "MSCDW.*", 
"snapshot.mode" : "initial",
"schema.include.list" : "MSCDW",
"database.history.kafka.bootstrap.servers" : "172.16.50.22:9092",
"database.history.kafka.topic": "kafkadebeziumoracle"
}
}'

3.结果演示
3.1、查看创建的connector列表
3.2、查看创建的connector状态
3.3、查看创建的connector配置
3.4、查看topic变化
当环境搭建好之后，默认为每个表创建一个属于自己的主题，如图所示，小编这里使用的kafka Tool工具查看
3.5、查看kafka中数据
在次小编换了各schema：scott，因为之前用的是mscdw，不过是一样的效果，是后来看写的东西发现这块有点不清楚，但环境已经改为监听scott的库

3.6、在flink中创建source和sink的表

CREATE TABLE sinkMysqlConfigTable				
(
    ID  STRING,
    CRON STRING
) WITH (
    'connector.type' = 'jdbc', 
    'connector.url' = 'jdbc:mysql://IP:3306/admin', 
    'connector.table' = 'config',
    'connector.username' = 'root',
    'connector.password' = 'dhcc@2020', 
    'connector.write.flush.max-rows' = '1' 
);

CREATE TABLE createOracleConfigTable (
  id STRING,
  cron STRING
) WITH (
  'connector' = 'kafka',
  'topic' = 'XE.MSCDW.CONFIG',
  'properties.bootstrap.servers' = '172.16.50.22:9092',
  'debezium-json.schema-include' ='false',
  'properties.group.id' = 'amirdebezium',
  'scan.startup.mode' = 'earliest-offset',
  'value.format' = 'debezium-json'
);