1. What is Flink CDC
1.1 What is Flink-CDC
Flink-CDC is a CDC project on Flink, which provides CDC capabilities for various database products. Put the relevant jar packages of the project in {Flink installation directory}/lib/, and the deployment of CDC is completed. At this point, flink-job can be created through flink-sql for database CDC.
1.2 What is CDC
Change Data Capture, using CDC, we can obtain the submitted [data changes] from the database and send these [data changes] to the downstream [target database] or other data products.
2. Deployment process
- Install Flink
- Install the cdc jar package under {Flink installation directory}/lib/.
# cdc jar 包
- flink-sql-connector-mongodb-cdc-2.1.1.jar
- flink-sql-connector-mysql-cdc-2.1.1.jar
- flink-sql-connector-oracle-cdc-2.1.1.jar
- flink-sql-connector-postgres-cdc-2.1.1.jar
# jdbc 驱动包
- mysql-connector-java-5.1.29.jar
- ojdbc6.jar
- deployment complete
download link
-
Flink download address:
https://flink.apache.org/zh/downloads.html
https://dlcdn.apache.org/flink/ -
cdc jar package download address:
https://github.com/ververica/flink-cdc-connectors/releases
3. Preparations around CDC
- Enable the binlog support of the data source MySql.
- Prepare the target database.
4. How to use flink-cdc
Flink cdc provides two ways to complete database cdc. They are:
- flink-sql (recommended)
- datastream
Both methods can create [flink-job] in flink for database cdc. Compared with the latter, the former has a simple and convenient operation mode and clear semantics, but its customization ability is relatively weak.
5. Use the flink-sql method to perform flink-cdc
There is a script command ./bin/sql-client.sh in the flink installation directory. Through this command, the flink-sql console can be launched and submitted to flink-sqlcreate flink-joband flink-cdccomplete 数据库 cdc.
Official website demo address: https://ververica.github.io/flink-cdc-connectors/release-2.1/