important!
The name Flink CDC carries Flink, but pay attention! ! ! When we build a simple demo locally, we can run it locally without downloading the flink environment .
FlinkCDC is a jar package . After introducing it, write a main method to display its simple functions. (As a novice, I regarded FlinkCDC as a plug-in similar to ik word segmentation, so I downloaded Flink first...)
Preparation
- You need to know how to use maven
- ≤FlinkCDC 1.4.0 version can use MySQL5.6+, FlinkCDC must be 5.7+
Although Flink CDC officially requires MySQL 5.7+, Flink CDC version 1.4 currently has no problem running on MySQL 5.6. Simply building a demo is enough. If you build a production environment using 5.6, please test it carefully!
Enable binlog
You need to modify the MySQL configuration file my.cnf, please add or add the following content in it:
[mysqld]
log-bin=mysql-bin
server-id=1
binlog_format=ROW
MAC users need to manually create this file in the etc directory. Windows users can refer to other blogs to enable binlog (this step is very simple, so there is more content)
Check binlog
You need to restart MySQL
Then run the following statement to see if it is enabled successfully:
show variables like '%log_bin%';After running, it should show that log_bin is ON
show variables like 'binlog_format';After running, it should show that binlog_format is ROW
If the running result is correct, you can continue to the next step. If it is incorrect please correct it first.
Create a new project and test
Open the IDE and create a new blank Maven project.
Here, both my Flink CDC version 1.4.0 and Flink CDC 2.0 can be used normally. You can choose by yourself (for friends with older versions, comment out the 2.0.x coordinates and replace them with the ones in the comments below)
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.example</groupId>
<artifactId>flinkcdc-test</artifactId>
<version>1.0-SNAPSHOT</version>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>8</source>
<target>8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<classpathPrefix>lib/</classpathPrefix>
<mainClass>*.dsp.redispositive.Application</mainClass>
<useUniqueVersions>false</useUniqueVersions>
</manifest>
</archive>
</configuration>
</plugin>
</plugins>
</build>
<properties>
<flink-version>1.13.0</flink-version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-java</artifactId>
<version>${flink-version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_2.12</artifactId>
<version>${flink-version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients_2.12</artifactId>
<version>${flink-version}</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.44</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-planner-blink_2.12</artifactId>
<version>${flink-version}</version>
</dependency>
<dependency>
<groupId>com.ververica</groupId>
<artifactId>flink-connector-mysql-cdc</artifactId>
<version>2.0.2</version>
</dependency>
<!--老版本的用户打开这个注释，把上面2.0注释掉
<dependency>
<groupId>com.alibaba.ververica</groupId>
<artifactId>flink-connector-mysql-cdc</artifactId>
<version>1.4.0</version>
</dependency>-->
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.20</version>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>1.2.75</version>
</dependency>
</dependencies>
</project>We create a new main method for testing:
Note: The Flink CDC2.0 package name has been changed, and "alibaba" is not included in the import. You can paste the code to let it automatically import the package.
import com.alibaba.ververica.cdc.connectors.mysql.table.StartupOptions;
import com.alibaba.ververica.cdc.debezium.StringDebeziumDeserializationSchema;
import org.apache.commons.collections.CollectionUtils;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.source.SourceFunction;
import com.alibaba.ververica.cdc.connectors.mysql.MySQLSource;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
public class MySqlSourceExample {
public static void main(String[] args) throws Exception {
SourceFunction<String> sourceFunction = MySQLSource.<String>builder()
.hostname("localhost")
.port(3306)
.databaseList("cdc_test") //订阅的库
.username("root")
.password("root")
.deserializer(new StringDebeziumDeserializationSchema())
.build();
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.addSource(sourceFunction)
.print()
.setParallelism(1);
env.execute();
}
}carry out testing
We run the main method, the startup is slow, please wait patiently!
If the following things appear, they will not affect the running of the demo and can be ignored:
Just modify the database monitoring library and see the following binlog log information, which means success!
