CloudCanal Free Community Edition is a fully self-developed , visual , and automated data migration and synchronization tool launched by ClouGence . It has functions such as structure migration , data migration , data synchronization , data verification , and data correction , and supports 30+ popular relational databases. , real-time data warehouse, message middleware, cache database and search engine data interoperability, including domestic databases OceanBase , PolarDB , TiDB , StarRocks , Doris , RocketMQ , etc.
Combined with the open source project CloudCanalProcess , the following functions can be easily implemented
- wide-table : wide table data processing plug-in, mainly including fact table and single-dimensional table combination processing
- data-transform : General data conversion plug-in, such as making operation changes, adding additional fields, cleaning and backfilling data
- data-gather : Data aggregation plug-in that aggregates sub-databases, tables, vertical splits, and remote data in real time
- data-compare : Data comparison plug-in, perform business reconciliation based on changes in source data
- business-alert : Business alert plug-in, which can make corresponding alerts based on data changing trends.
CloudCanal product functions are aligned with Alibaba Cloud DTS, Amazon DMS, StreamSets, and AirByte to help developers, DBAs, or operation and maintenance students quickly build online data applications.
Release-Note
Key functions
- Supports Hana 1.x version. Currently, the supported Hana versions are 1.x and 2.x.
- Support Hana incremental task custom incremental table
- Support Hana to automatically delete triggers after modifying the subscription cancellation table
New features
- Supports MySQL <-> Tunnel <-> MySQL two-way synchronization, including add, delete, and modify operations (excluding DDL). This capability is an enhanced version of MySQL two-way synchronization. The databases at both ends do not need to disclose public network addresses.
- Support Alibaba Cloud fully managed SelectDB
- Supports the ability of StarRocks, Doris, SelectDB, Oracle, and MariaDB to clear data in full before the peer end, making it convenient for users to re-migrate and synchronize data.
- Supports MySQL, OceanBase, TiDB, StarRocks, Doris, SelectDB, Oracle, MariaDB, peer creation tasks or modification of subscriptions to rebuild peer tables
- Support MySQL -> TiDB/OceanBase link synchronization Online DDL tool statements
- Support task details > More functions > Modify task configuration, including clearing the peer data switch before full volume, rebuilding the peer table switch when creating a task, and modifying scheduled full/incremental Java Cron expressions
- Support MySQL -> You can check the DROP operation in the second step of MySQL creation task
- Support task alarm closing based on time period
- Supports the creation of task filtering tables that can be filtered in batches. That is, table1;table2;table3;...tableN can be pasted on the table selection page to search and automatically check, solving the complex problem of selecting tables when users create tasks.
- Support batch scheduling of tasks to another machine to improve manual task scheduling efficiency
- Support CloudCanal to display LicenseID to facilitate subsequent users to check the license expiration reminder information
Optimize
- Optimize Redis <-> Redis bidirectional synchronization anti-loop strategy, and support more instructions bidirectional synchronization anti-loop
- Optimize Redis -> Redis data initialization capabilities, including Set, ZSet, and List type data initialization
- Optimize Redis -> Redis instruction set Rename, Incr, Incrby, Hsetnx, Hmset, Lpush, Rpush, Lset, Sadd, Zadd, Zrem, Zremrangebyscore
- Optimize Kafka Debezium Json to support Bytes type (specific reference: Source Kafka Debezium Json usage instructions )
- Optimize the timeout processing logic of TiDB as the source-end Stub, adjust the passive reporting of timeout to active reporting, realize the task without restarting, and re-subscribe to the Region
- Optimize TiDB as a source solution for obtaining table information, greatly optimizing the speed of obtaining multi-table information.
- Optimize TiDB as a source-end strategy for maintaining multi-version table structures and proactively clean up expired table structures.
- Optimize TiDB as the source DML event sink condition to make DDL and DML consumption more orderly and reduce the risk of data inconsistency
- Optimize TiDB as the source-end Error event processing solution to reduce the consumption time of Region incremental scan
- Optimize TiDB as a source-end performance indicator chart, and add 5 new indicators (total number of CDC submission events, total number of Region exceptions within 10 seconds, total number of Region active within 10 seconds, total number of Regions in the task incremental scan phase, total number of Regions)
- Optimize MariaDB -> Oracle, Oracle -> Oracle migration using the new version structure
- Optimize PostgreSQL/MySQL -> StarRocks Json type field synchronization (optimize mapping rules)
- When optimizing and modifying subscriptions, rule mapping is only applied to newly added tables.
- Optimize and modify the subscription page to prevent repeated clicks
- Optimize that CloudCanal cannot display console monitoring information when it is not activated, so that users can view the current console status when it is not activated.
Problem fixes
- Fixed the problem of too many fields in the Hana source table, causing an error when creating a trigger
- Fixed the problem of error reporting in Hana table creation task without primary key
- Fixed the problem of Hana lowercase Schema/Table/Column creation task reporting error
- Fix the problem of repeated data insertion in Hana incremental synchronization without primary key table
- Fixed an issue that caused triggers of multiple subscription tables to be accidentally deleted when deleting tasks from the Hana source.
- Fixed the problem of parsing error caused by special characters in Hana source incremental phase data.
- Fixed the issue where the data verification task page displays missing, inconsistent and inconsistent results.
- Fixed the synchronization interruption problem caused by modification of LSN site sorting in SQL Server incremental synchronization
- Fixed the timeout problem caused by multiple Region subscriptions to TiDB as the source of large tables, and supports setting the timeout period.
- Fixed the data inconsistency problem caused by TiDB as the source task location being smaller than the GC safe point.
- Fix the parsing exception problem caused by asynchronous business DDL such as TiDB as the source end Create Database...
- Fixed the problem of incremental GC safe point detection error caused by time precision in TiDB source v6.5.0 and above.
- Fixed the issue where MySQL -> Doris structure migration comments were not synchronized to the peer
- Fixed the problem of missing default value of new column in MySQL -> StarRocks DDL
- Fixed the problem of MySQL -> StarRocks DDL modifying column properties with Not Null causing synchronization failure
- Fixed the problem of synchronization failure of new Text type field in MySQL -> StarRocks DDL
- Fix the problem of invalid MySQL -> Oracle table mapping
- Fixed the problem of PostgreSQL -> MySQL modifying subscription to perform multiple addition/subtraction operations on the same table, resulting in task error reporting
- Fixed the issue where the incremental position was not reset after the mission was rerun.
- Fixed the issue where the task cannot be deleted when the Oracle peer only has an external host.
- Fixed the issue where the target Schema is set to Public by default when modifying a subscription or creating a similar task when Oracle -> Oracle Schema does not exist.
- Fixed the issue where the message structure display in the Kafka/MQ -> MySQL creation task page is inconsistent with the actual selection
- Fixed the issue where the original mapping relationship between PostgreSQL -> StarRocks modification and subscription is invalid.
- Fixed the problem of error reporting when creating similar tasks in structure migration tasks
- Fixed the issue where the time point changed incorrectly when resetting the file point.
Free to download and use
- Get started quickly: https://www.clougence.com/cc-doc/quick/quick_start