content
Open source plugin Tapdata PDK
Quick start target database access
Download the source code and compile
Create a Connector project for the target database
Test and verify through TDD after development is complete
How to submit to the PDK open source project
Introduction:
It is no exaggeration to say that there is no developer who has not yet kicked the iron plate of "non-interoperability of application data" - in the case of different platforms, different technologies, different storage and deployment methods, and the lack of necessary interfaces, the application systems are different. Difficult to communicate. With the continuous expansion of business requirements, applications are constantly developing towards diversification and personalization. The contradiction between future business and outdated technology stacks is also becoming more and more prominent, and the number of interfaces required is also increasing.
How to get the interface development simple and fast has become a problem that we need to consider. Recently, I dug up a very fragrant open source plug-in. After carefully studying the technical documentation, I decided to give you Amway:
Open source plugin Tapdata PDK
GitHub link: https://github.com/tapdata/idaas-pdk
The publisher of this project is Tapdata, a domestic entrepreneurial team specializing in real-time data service platforms. According to officials, this open source component is also the stepping stone for the open source of its core products. It is backed by the team’s real-time data synchronization. A fairly mature strength.
PDK is an open source plug-in development framework abstracted from its data interface technology. Through the Source Plugin interface or the Target Plugin interface, the adaptation and compatibility of the new database as the source or target of Tapdata can be quickly realized, so that the Tapdata Cloud product and the upcoming open source can be quickly realized. Tapdata , free access to real-time data docking capabilities from various heterogeneous data sources to target databases or platforms.
According to the development specification of PDK connector, the development of data source and target can simplify the development process of data link. Through detailed development planning and built-in TDD test, the development of new data source and target can be completed simply and quickly.
Types of support include:
- Access database: MySQL, Oracle, PostgreSQL, etc.
- Access SaaS products: Salesforce, vika table, gold data form, Zoho CRM, etc.
- Access to custom data sources: can connect to private protocol data sources
Quick start target database access
At present, the PDK team has made the technical documents public, and you can go to GitHub ( https://github.com/tapdata/idaas-pdk) for details.
Prepare the environment
- Java 8
- Maven
- Git
- IntelliJ IDEA
Download the source code and compile
git clone https://github.com/tapdata/idaas-pdk.git
cd idaas-pdk
mvn clean install
Create a Connector project for the target database
For example, the group is io.tapdata, the database name is XDB, and the version version is 0.0.1. Create a Connector project with the following command
-
When the target database does not need to create a table
./bin/tap template --type target --group io.tapdata --name XDB --version 0.0.1 --output ./connectors
Open idaas-pdk with ItelliJ IDEA, and you can see the xdb-connector project under idaas-pdk/connectors.
- Fill in configOptions in spec.json
After the configOptions is integrated into the Tapdata site, it configures the input items for the user when using the Connector, such as the database connection address, user name, password, etc.
{
...
"configOptions":{
"connection":{
"type":"object",
"properties":{
"host":{
"type": "string",
"title": "Host",
"x-decorator": "FormItem",
"x-component": "Input"
},
"port":{
"type": "number",
"title": "Port",
"x-decorator": "FormItem",
"x-component": "Input"
}
}
}
}
}
- Write the code to access the target database
@TapConnectorClass("spec.json")
public class XDBConnector extends ConnectorBase implements TapConnector {
@Override
public void discoverSchema(TapConnectionContext connectionContext, Consumer<List<TapTable>> consumer) {
//TODO Load tables from database, connection information in connectionContext#getConnectionConfig
//Sample code shows how to define tables.
consumer.accept(list(
//Define first table
table("empty-table1"),
//Define second table
table("empty-table2"))
));
}
@Override
public void connectionTest(TapConnectionContext connectionContext, Consumer<TestItem> consumer) {
//Assume below tests are successfully, below tests are recommended, but not required.
//Connection test
//TODO execute connection test here
consumer.accept(testItem(TestItem.ITEM_CONNECTION, TestItem.RESULT_SUCCESSFULLY));
//Login test
//TODO execute login test here
consumer.accept(testItem(TestItem.ITEM_LOGIN, TestItem.RESULT_SUCCESSFULLY));
//Read test
//TODO execute read test by checking role permission
consumer.accept(testItem(TestItem.ITEM_READ, TestItem.RESULT_SUCCESSFULLY));
//Write test
//TODO execute write test by checking role permission
consumer.accept(testItem(TestItem.ITEM_WRITE, TestItem.RESULT_SUCCESSFULLY));
}
private void writeRecord(TapConnectorContext connectorContext, List<TapRecordEvent> tapRecordEvents, Consumer<WriteListResult<TapRecordEvent>> writeListResultConsumer) {
//TODO write records into database
//Below is sample code to print received events which suppose to write to database.
AtomicLong inserted = new AtomicLong(0); //insert count
AtomicLong updated = new AtomicLong(0); //update count
AtomicLong deleted = new AtomicLong(0); //delete count
for(TapRecordEvent recordEvent : tapRecordEvents) {
if(recordEvent instanceof TapInsertRecordEvent) {
//TODO insert record
inserted.incrementAndGet();
} else if(recordEvent instanceof TapUpdateRecordEvent) {
//TODO update record
updated.incrementAndGet();
} else if(recordEvent instanceof TapDeleteRecordEvent) {
//TODO delete record
deleted.incrementAndGet();
}
}
//Need to tell incremental engine the write result
writeListResultConsumer.accept(writeListResult()
.insertedCount(inserted.get())
.modifiedCount(updated.get())
.removedCount(deleted.get()));
}
private void queryByFilter(TapConnectorContext connectorContext, List<TapFilter> filters, Consumer<List<FilterResult>> listConsumer){
//Filter is exactly match.
//If query by the filter, no value is in database, please still create a FitlerResult with null value in it. So that flow engine can understand the filter has no value.
}
}
-
When the target database needs to create a table
./bin/tap template --type targetNeedTable --group io.tapdata --name XDB --version 0.0.1 --output ./connectors
Open idaas-pdk with ItelliJ IDEA, and you can see the xdb-connector project under idaas-pdk/connectors.
- Fill in configOptions in spec.json
After configOptions is integrated into the Tapdata site, configure the input items for the user when using the Connector, such as the database connection address, user name, password, etc.
{
...
"configOptions":{
"connection":{
"type":"object",
"properties":{
"host":{
"type": "string",
"title": "Host",
"x-decorator": "FormItem",
"x-component": "Input"
},
"port":{
"type": "number",
"title": "Port",
"x-decorator": "FormItem",
"x-component": "Input"
}
}
}
}
}
- Fill in dataTypes (type expression) in spec.json
dataTypes is used to describe the range of all fields that this Connector accesses the database, and converts to the corresponding TapType. The source database will also provide the same dataTypes description, so when the source data flows into Tapdata, it will combine the field description information of the source dataTypes with the field information of the source database table, and enter Tapdata through the neutral data structure of Tapdata. In the data flow, before the data flows into the target database, Tapdata will find the best storage field in the dataTypes of the target library based on this information, and inform the PDK developer through the originType of the TapField to build the table.
{
...
"dataTypes":{
"boolean":{"bit":8, "unsigned":"", "to":"TapNumber"},
"tinyint":{"bit":8, "to":"TapNumber"},
"smallint":{"bit":16, "to":"TapNumber"},
"int":{"bit":32, "to":"TapNumber"},
"bigint":{"bit":64, "to":"TapNumber"},
"largeint":{"bit":128, "to":"TapNumber"},
"float":{"bit":32, "to":"TapNumber"},
"double":{"bit":64, "to":"TapNumber"},
"decimal[($precision,$scale)]":{"bit": 128, "precision": [1, 27], "defaultPrecision": 10, "scale": [0, 9], "defaultScale": 0, "to": "TapNumber"},
"date":{"byte":3, "range":["0000-01-01", "9999-12-31"], "to":"TapDate"},
"datetime":{"byte":8, "range":["0000-01-01 00:00:00","9999-12-31 23:59:59"],"to":"TapDateTime"},
"char[($byte)]":{"byte":255, "to": "TapString", "defaultByte": 1},
"varchar[($byte)]":{"byte":"65535", "to":"TapString"},
"string":{"byte":"2147483643", "to":"TapString"},
"HLL":{"byte":"16385", "to":"TapNumber", "queryOnly":true}
}
}
- Write the code to access the target database
@TapConnectorClass("spec.json")
public class XDBConnector extends ConnectorBase implements TapConnector {
@Override
public void discoverSchema(TapConnectionContext connectionContext, Consumer<List<TapTable>> consumer) {
//TODO Load schema from database, connection information in connectionContext#getConnectionConfig
//Sample code shows how to define tables with specified fields
consumer.accept(list(
//Define first table
table("empty-table1")
//Define a field named "id", origin field type, whether is primary key and primary key position
.add(field("id", "varchar").isPrimaryKey(true).partitionKeyPos(1))
.add(field("description", "string"))
.add(field("name", "varchar"))
.add(field("age", "int")))
));
}
@Override
public void connectionTest(TapConnectionContext connectionContext, Consumer<TestItem> consumer) {
//Assume below tests are successfully, below tests are recommended, but not required.
//Connection test
//TODO execute connection test here
consumer.accept(testItem(TestItem.ITEM_CONNECTION, TestItem.RESULT_SUCCESSFULLY));
//Login test
//TODO execute login test here
consumer.accept(testItem(TestItem.ITEM_LOGIN, TestItem.RESULT_SUCCESSFULLY));
//Read test
//TODO execute read test by checking role permission
consumer.accept(testItem(TestItem.ITEM_READ, TestItem.RESULT_SUCCESSFULLY));
//Write test
//TODO execute write test by checking role permission
consumer.accept(testItem(TestItem.ITEM_WRITE, TestItem.RESULT_SUCCESSFULLY));
}
@Override
public void registerCapabilities(ConnectorFunctions connectorFunctions, TapCodecRegistry codecRegistry) {
connectorFunctions.supportWriteRecord(this::writeRecord);
connectorFunctions.supportQueryByFilter(this::queryByFilter);
//If database need insert record before table created, then please implement the below two methods.
connectorFunctions.supportCreateTable(this::createTable);
connectorFunctions.supportDropTable(this::dropTable);
//If database need insert record before table created, please implement the custom codec for the TapValue that data types in spec.json didn't cover.
//TapTimeValue, TapMapValue, TapDateValue, TapArrayValue, TapYearValue, TapNumberValue, TapBooleanValue, TapDateTimeValue, TapBinaryValue, TapRawValue, TapStringValue
codecRegistry.registerFromTapValue(TapRawValue.class, "text", tapRawValue -> {
if (tapRawValue != null && tapRawValue.getValue() != null)
return toJson(tapRawValue.getValue());
return "null";
});
}
private void writeRecord(TapConnectorContext connectorContext, List<TapRecordEvent> tapRecordEvents, Consumer<WriteListResult<TapRecordEvent>> writeListResultConsumer) {
//TODO write records into database
//Below is sample code to print received events which suppose to write to database.
AtomicLong inserted = new AtomicLong(0); //insert count
AtomicLong updated = new AtomicLong(0); //update count
AtomicLong deleted = new AtomicLong(0); //delete count
for(TapRecordEvent recordEvent : tapRecordEvents) {
if(recordEvent instanceof TapInsertRecordEvent) {
//TODO insert record
inserted.incrementAndGet();
PDKLogger.info(TAG, "Record Write TapInsertRecordEvent {}", toJson(recordEvent));
} else if(recordEvent instanceof TapUpdateRecordEvent) {
//TODO update record
updated.incrementAndGet();
PDKLogger.info(TAG, "Record Write TapUpdateRecordEvent {}", toJson(recordEvent));
} else if(recordEvent instanceof TapDeleteRecordEvent) {
//TODO delete record
deleted.incrementAndGet();
PDKLogger.info(TAG, "Record Write TapDeleteRecordEvent {}", toJson(recordEvent));
}
}
//Need to tell incremental engine the write result
writeListResultConsumer.accept(writeListResult()
.insertedCount(inserted.get())
.modifiedCount(updated.get())
.removedCount(deleted.get()));
}
private void queryByFilter(TapConnectorContext connectorContext, List<TapFilter> filters, Consumer<List<FilterResult>> listConsumer){
//Filter is exactly match.
//If query by the filter, no value is in database, please still create a FitlerResult with null value in it. So that flow engine can understand the filter has no value.
}
private void dropTable(TapConnectorContext connectorContext, TapDropTableEvent dropTableEvent) {
TapTable table = connectorContext.getTable();
//TODO implement drop table
}
private void createTable(TapConnectorContext connectorContext, TapCreateTableEvent createTableEvent) {
//TODO implement create table.
TapTable table = connectorContext.getTable();
LinkedHashMap<String, TapField> nameFieldMap = table.getNameFieldMap();
for(Map.Entry<String, TapField> entry : nameFieldMap.entrySet()) {
TapField field = entry.getValue();
String originType = field.getOriginType();
//originType is the data types defined in spec.json
//TODO use the generated originType to create table.
}
}
}
Test and verify through TDD after development is complete
Provide a json file that needs to be filled in by the user in configOptions. For example, in the above configOptions, the user is required to fill in the Host and Port of the database, then the content of the xdb_tdd.json file of tdd is as follows:
{
"connection": {
"host": "192.168.153.132",
"port": 9030,
}
}
Execute the TDD test command:
./bin/tap tdd --testConfig xdb_tdd.json ./connectors/xdb-connector
When the TDD test fails, please modify the corresponding errors according to the error prompts until the TDD test is passed;
When the TDD test passes, the PDK Connector is in a state where a Pull Request can be submitted.
How to submit to the PDK open source project
① fork idaas-pdk, create a local branch based on the remote main branch
② According to the name of the database to be accessed, create a new module in the idaas-pdk/connectors directory, and the naming convention is {database lowercase name}-connector, for example, the name of the access database is XDB, and the module name is xdb-connector
③ The developer completes the development and implementation of the access database according to the official API document
④ After passing the TDD test, submit PR to idaas-pdk
⑤ Merge the code after the PR submitted by the official team Review
Easter eggs
Interested students don't rush to develop it. It is understood that the official free version has successively realized real-time data connection between 30 common data sources/targets. If it already contains the database you want to access, you can Use Tapdata Cloud directly ( Tapdata Cloud | Free Heterogeneous Database Real-time Sync Cloud Platform - Tapdata ) for free real-time data synchronization. Of course, if your needs are not supported at this stage, you can use Tapdata PDK for self-service development and quick access.
Data connection types currently supported by Tapdata Cloud
At present, Tapdata has opened a plug-in ecological co-construction group for developers. It can provide technical exchanges and support during the development process. Interested students can scan the code to follow and invite you to join the group:
