Introduction:
It is no exaggeration to say that there is no developer who has not yet kicked the iron plate of "non-interoperability of application data" - in the case of different platforms, different technologies, different storage and deployment methods, and the lack of necessary interfaces, the application systems are different. Difficult to communicate. With the continuous expansion of business requirements, applications are constantly developing towards diversification and personalization. The contradiction between future business and outdated technology stacks is also becoming more and more prominent, and the number of interfaces required is also increasing.
How to get the interface development simple and fast has become a problem that we need to consider. Recently, I dug up a very fragrant open source plug-in. After carefully studying the technical documentation, I decided to give you Amway:
Open source plugin Tapdata PDK
GitHub link: github.com/tapdata/ida…
The publisher of this project is Tapdata, a domestic entrepreneurial team specializing in real-time data service platforms. According to officials, this open source component is also the stepping stone for the open source of its core products. It is backed by the team’s real-time data synchronization. A fairly mature strength.
PDK is an open source plug-in development framework abstracted from its data interface technology. Through the Source Plugin interface or the Target Plugin interface, the adaptation and compatibility of the new database as the source or target of Tapdata can be quickly realized, so that the Tapdata Cloud product and the upcoming open source can be quickly realized. Tapdata , free access to real-time data docking capabilities from various heterogeneous data sources to target databases or platforms.
According to the development specification of PDK connector, the development of data source and target can simplify the development process of data link. Through detailed development planning and built-in TDD test, the development of new data source and target can be completed simply and quickly.
Types of support include:
- Access database: MySQL, Oracle, PostgreSQL, etc.
- Access SaaS products: Salesforce, vika table, gold data form, Zoho CRM, etc.
- Access to custom data sources: can connect to private protocol data sources
Quick start target database access
At present, the PDK team has made the technical documents public, and you can go to GitHub ( github.com/tapdata/ida… for details.
Prepare the environment
- Java 8
- Maven
- Git
- IntelliJ IDEA
Download the source code and compile
git clone https://github.com/tapdata/idaas-pdk.git
cd idaas-pdk
mvn clean install
复制代码Create a Connector project for the target database
For example, the group is io.tapdata, the database name is XDB, and the version version is 0.0.1. Create a Connector project with the following command
-
When the target database does not need to create a table
./bin/tap template --type target --group io.tapdata --name XDB --version 0.0.1 --output ./connectors
复制代码Open idaas-pdk with ItelliJ IDEA, and you can see the xdb-connector project under idaas-pdk/connectors.
- Fill in configOptions in spec.json
After the configOptions is integrated into the Tapdata site, it configures the input items for the user when using the Connector, such as the database connection address, user name, password, etc.
{
...
"configOptions":{
"connection":{
"type":"object",
"properties":{
"host":{
"type": "string",
"title": "Host",
"x-decorator": "FormItem",
"x-component": "Input"
},
"port":{
"type": "number",
"title": "Port",
"x-decorator": "FormItem",
"x-component": "Input"
}
}
}
}
}
复制代码- Write the code to access the target database
@TapConnectorClass("spec.json")
public class XDBConnector extends ConnectorBase implements TapConnector {
@Override
public void discoverSchema(TapConnectionContext connectionContext, Consumer<List<TapTable>> consumer) {
//TODO Load tables from database, connection information in connectionContext#getConnectionConfig
//Sample code shows how to define tables.
consumer.accept(list(
//Define first table
table("empty-table1"),
//Define second table
table("empty-table2"))
));
}
@Override
public void connectionTest(TapConnectionContext connectionContext, Consumer<TestItem> consumer) {
//Assume below tests are successfully, below tests are recommended, but not required.
//Connection test
//TODO execute connection test here
consumer.accept(testItem(TestItem.ITEM_CONNECTION, TestItem.RESULT_SUCCESSFULLY));
//Login test
//TODO execute login test here
consumer.accept(testItem(TestItem.ITEM_LOGIN, TestItem.RESULT_SUCCESSFULLY));
//Read test
//TODO execute read test by checking role permission
consumer.accept(testItem(TestItem.ITEM_READ, TestItem.RESULT_SUCCESSFULLY));
//Write test
//TODO execute write test by checking role permission
consumer.accept(testItem(TestItem.ITEM_WRITE, TestItem.RESULT_SUCCESSFULLY));
}
private void writeRecord(TapConnectorContext connectorContext, List<TapRecordEvent> tapRecordEvents, Consumer<WriteListResult<TapRecordEvent>> writeListResultConsumer) {
//TODO write records into database
//Below is sample code to print received events which suppose to write to database.
AtomicLong inserted = new AtomicLong(0); //insert count
AtomicLong updated = new AtomicLong(0); //update count
AtomicLong deleted = new AtomicLong(0); //delete count
for(TapRecordEvent recordEvent : tapRecordEvents) {
if(recordEvent instanceof TapInsertRecordEvent) {
//TODO insert record
inserted.incrementAndGet();
} else if(recordEvent instanceof TapUpdateRecordEvent) {
//TODO update record
updated.incrementAndGet();
} else if(recordEvent instanceof TapDeleteRecordEvent) {
//TODO delete record
deleted.incrementAndGet();
}
}
//Need to tell incremental engine the write result
writeListResultConsumer.accept(writeListResult()
.insertedCount(inserted.get())
.modifiedCount(updated.get())
.removedCount(deleted.get()));
}
private void queryByFilter(TapConnectorContext connectorContext, List<TapFilter> filters, Consumer<List<FilterResult>> listConsumer){
//Filter is exactly match.
//If query by the filter, no value is in database, please still create a FitlerResult with null value in it. So that flow engine can understand the filter has no value.
}
}
复制代码-
When the target database needs to create a table
./bin/tap template --type targetNeedTable --group io.tapdata --name XDB --version 0.0.1 --output ./connectors
复制代码Open idaas-pdk with ItelliJ IDEA, and you can see the xdb-connector project under idaas-pdk/connectors.
- Fill in configOptions in spec.json
After configOptions is integrated into the Tapdata site, configure the input items for the user when using the Connector, such as the database connection address, user name, password, etc.
{
...
"configOptions":{
"connection":{
"type":"object",
"properties":{
"host":{
"type": "string",
"title": "Host",
"x-decorator": "FormItem",
"x-component": "Input"
},
"port":{
"type": "number",
"title": "Port",
"x-decorator": "FormItem",
"x-component": "Input"
}
}
}
}
}
复制代码- Fill in dataTypes (type expression) in spec.json
dataTypes 用于描述该 Connector 接入数据库的所有字段的范围，以及转换到对应的 TapType。 源端数据库也会提供相同的 dataTypes 描述， 这样当源端数据流入到 Tapdata 里时， 会结合源端 dataTypes 的字段描述信息结合源端库表的字段信息， 通过 Tapdata 的中立数据结构进入到 Tapdata 的数据流中， 当数据要流入到目标数据库之前，Tapdata 会根据这些信息， 在目标库的 dataTypes 中找到最佳的存储字段， 通过 TapField 的 originType 告知给 PDK 开发者， 用以建表。
{
...
"dataTypes":{
"boolean":{"bit":8, "unsigned":"", "to":"TapNumber"},
"tinyint":{"bit":8, "to":"TapNumber"},
"smallint":{"bit":16, "to":"TapNumber"},
"int":{"bit":32, "to":"TapNumber"},
"bigint":{"bit":64, "to":"TapNumber"},
"largeint":{"bit":128, "to":"TapNumber"},
"float":{"bit":32, "to":"TapNumber"},
"double":{"bit":64, "to":"TapNumber"},
"decimal[($precision,$scale)]":{"bit": 128, "precision": [1, 27], "defaultPrecision": 10, "scale": [0, 9], "defaultScale": 0, "to": "TapNumber"},
"date":{"byte":3, "range":["0000-01-01", "9999-12-31"], "to":"TapDate"},
"datetime":{"byte":8, "range":["0000-01-01 00:00:00","9999-12-31 23:59:59"],"to":"TapDateTime"},
"char[($byte)]":{"byte":255, "to": "TapString", "defaultByte": 1},
"varchar[($byte)]":{"byte":"65535", "to":"TapString"},
"string":{"byte":"2147483643", "to":"TapString"},
"HLL":{"byte":"16385", "to":"TapNumber", "queryOnly":true}
}
}
复制代码- 编写接入目标数据库的代码
@TapConnectorClass("spec.json")
public class XDBConnector extends ConnectorBase implements TapConnector {
@Override
public void discoverSchema(TapConnectionContext connectionContext, Consumer<List<TapTable>> consumer) {
//TODO Load schema from database, connection information in connectionContext#getConnectionConfig
//Sample code shows how to define tables with specified fields
consumer.accept(list(
//Define first table
table("empty-table1")
//Define a field named "id", origin field type, whether is primary key and primary key position
.add(field("id", "varchar").isPrimaryKey(true).partitionKeyPos(1))
.add(field("description", "string"))
.add(field("name", "varchar"))
.add(field("age", "int")))
));
}
@Override
public void connectionTest(TapConnectionContext connectionContext, Consumer<TestItem> consumer) {
//Assume below tests are successfully, below tests are recommended, but not required.
//Connection test
//TODO execute connection test here
consumer.accept(testItem(TestItem.ITEM_CONNECTION, TestItem.RESULT_SUCCESSFULLY));
//Login test
//TODO execute login test here
consumer.accept(testItem(TestItem.ITEM_LOGIN, TestItem.RESULT_SUCCESSFULLY));
//Read test
//TODO execute read test by checking role permission
consumer.accept(testItem(TestItem.ITEM_READ, TestItem.RESULT_SUCCESSFULLY));
//Write test
//TODO execute write test by checking role permission
consumer.accept(testItem(TestItem.ITEM_WRITE, TestItem.RESULT_SUCCESSFULLY));
}
@Override
public void registerCapabilities(ConnectorFunctions connectorFunctions, TapCodecRegistry codecRegistry) {
connectorFunctions.supportWriteRecord(this::writeRecord);
connectorFunctions.supportQueryByFilter(this::queryByFilter);
//If database need insert record before table created, then please implement the below two methods.
connectorFunctions.supportCreateTable(this::createTable);
connectorFunctions.supportDropTable(this::dropTable);
//If database need insert record before table created, please implement the custom codec for the TapValue that data types in spec.json didn't cover.
//TapTimeValue, TapMapValue, TapDateValue, TapArrayValue, TapYearValue, TapNumberValue, TapBooleanValue, TapDateTimeValue, TapBinaryValue, TapRawValue, TapStringValue
codecRegistry.registerFromTapValue(TapRawValue.class, "text", tapRawValue -> {
if (tapRawValue != null && tapRawValue.getValue() != null)
return toJson(tapRawValue.getValue());
return "null";
});
}
private void writeRecord(TapConnectorContext connectorContext, List<TapRecordEvent> tapRecordEvents, Consumer<WriteListResult<TapRecordEvent>> writeListResultConsumer) {
//TODO write records into database
//Below is sample code to print received events which suppose to write to database.
AtomicLong inserted = new AtomicLong(0); //insert count
AtomicLong updated = new AtomicLong(0); //update count
AtomicLong deleted = new AtomicLong(0); //delete count
for(TapRecordEvent recordEvent : tapRecordEvents) {
if(recordEvent instanceof TapInsertRecordEvent) {
//TODO insert record
inserted.incrementAndGet();
PDKLogger.info(TAG, "Record Write TapInsertRecordEvent {}", toJson(recordEvent));
} else if(recordEvent instanceof TapUpdateRecordEvent) {
//TODO update record
updated.incrementAndGet();
PDKLogger.info(TAG, "Record Write TapUpdateRecordEvent {}", toJson(recordEvent));
} else if(recordEvent instanceof TapDeleteRecordEvent) {
//TODO delete record
deleted.incrementAndGet();
PDKLogger.info(TAG, "Record Write TapDeleteRecordEvent {}", toJson(recordEvent));
}
}
//Need to tell incremental engine the write result
writeListResultConsumer.accept(writeListResult()
.insertedCount(inserted.get())
.modifiedCount(updated.get())
.removedCount(deleted.get()));
}
private void queryByFilter(TapConnectorContext connectorContext, List<TapFilter> filters, Consumer<List<FilterResult>> listConsumer){
//Filter is exactly match.
//If query by the filter, no value is in database, please still create a FitlerResult with null value in it. So that flow engine can understand the filter has no value.
}
private void dropTable(TapConnectorContext connectorContext, TapDropTableEvent dropTableEvent) {
TapTable table = connectorContext.getTable();
//TODO implement drop table
}
private void createTable(TapConnectorContext connectorContext, TapCreateTableEvent createTableEvent) {
//TODO implement create table.
TapTable table = connectorContext.getTable();
LinkedHashMap<String, TapField> nameFieldMap = table.getNameFieldMap();
for(Map.Entry<String, TapField> entry : nameFieldMap.entrySet()) {
TapField field = entry.getValue();
String originType = field.getOriginType();
//originType is the data types defined in spec.json
//TODO use the generated originType to create table.
}
}
}
复制代码开发完成之后通过 TDD 进行测试验证
提供 configOptions 里需要用户填写的内容的 json 文件， 例如上述 configOptions 里要求用户填写的是数据库的 Host 和 Port， 那么 tdd 的 xdb_tdd.json 文件内容如下：
{
"connection": {
"host": "192.168.153.132",
"port": 9030,
}
}
复制代码执行 TDD 测试命令：
./bin/tap tdd --testConfig xdb_tdd.json ./connectors/xdb-connector
复制代码当 TDD 测试没有通过， 请根据错误提示修改对应错误，直至通过 TDD测试；
当 TDD 测试通过后， PDK Connector 就处于可以提交 Pull Request 的状态。
如何提交到 PDK 开源项目
① fork idaas-pdk， 基于远程的 main 分支建立本地分支
② 根据要接入数据库名称， 在 idaas-pdk/connectors 目录下新建模块， 命名规范为 {数据库小写名称}-connector， 例如接入数据库的名称为 XDB， 模块名称为 xdb-connector
③ 开发者根据官方 API 文档完成接入数据库的开发实现
④ 通过 TDD 测试后， 提交 PR 到 idaas-pdk
⑤ 官方团队 Review 提交的 PR 之后合并代码
彩蛋
Interested students don't rush to develop it. It is understood that the official free version has successively realized real-time data connection between 30 common data sources/targets. If it already contains the database you want to access, you can Directly use Tapdata Cloud ( tapdata.net/tapdata-clo… Tapdata PDK self-development, fast access.
Data connection types currently supported by Tapdata Cloud
At present, Tapdata has opened a plug-in ecological co-construction group for developers. It can provide technical exchanges and support during the development process. Interested students can scan the code to follow and invite you to join the group: