BDS positioning
Aiming at the current synchronization migration pain points of open source HBase, BDS has independently developed a set of data migration platform for seamless migration of HBase clusters, active and standby disaster recovery, remote multiple activities, online and offline business separation, HBase data archiving, and RDS docking. Real-time incremental data and so on. The purpose is to help customers on Alibaba Cloud solve the data import and export of self-built HBase, EMR HBase, and Apsara HBase, so as to facilitate cloud customers to build highly available and flexible business systems around HBase.
scenes to be used
BDS currently has the following usage scenarios on the cloud:
Seamless migration between new and old clusters
Separation of online and offline business
Primary and standby disaster recovery
RDS real-time data synchronization to HBase, Phoenix
Seamless migration between new and old clusters
Scope of use
HBase major version upgrade, 1.x upgrade 2.x
Cluster configuration upgrade
Switch of cluster network, classic switch VPC
Relocation across computer rooms
Business split
Business Process
Separation of online and offline business
Business architecture
Through BDS, online business data is synchronized to offline clusters in real time, and offline data analysis is performed with big data components such as Spark and MR, so that online business queries are not affected.
Primary and standby disaster recovery
Business Process
Through BDS, real-time data synchronization between the active and standby clusters is realized. When the main cluster has a problem, you can switch to the standby cluster to minimize the impact on the business. When the main cluster service is restored, the BDS synchronization service is used to make up for the missing of the main cluster. The data.
RDS real-time data synchronization to HBase, Phoenix
Business architecture
Real-time synchronization of RDS data to HBase cluster through BDS
Combining with big data components such as Spark and MR to perform offline data analysis, so as not to affect online business queries
Use the function of HBase to support dynamic columns to build multiple tables in mysql into a large HBase table, avoiding join queries in RDS
Make RDS history library
BDS advantage
Feature
Support cross-version data migration and synchronization, HBase <-> HBase, Phoenix <-> Phoenix
For the migration of homogeneous data sources, support for synchronous table creation statements to ensure consistent partitions
Support real-time data synchronization of heterogeneous data sources, RDS -> HBase, Phoenix
Migration synchronization does not perceive the business side
Non-stop service data migration, while getting historical data migration and real-time incremental data synchronization
No intrusion into business-side application code
Migration synchronization has little impact on business
During the migration process, there is basically no interaction with the HBase of the cluster, only the HDFS of the cluster is read and written
Can dynamically adjust the speed of migration and concurrent reading
Correctness
Ensure that data migration is not lost
Data sampling and verification
stability
There is a complete failover mechanism that can automatically handle common errors
Real-time system monitoring, convenient to locate upstream and downstream problems
Alarm mechanism, alarm for some abnormalities that require human intervention
Error retry supports resumable transmission
High efficiency
The migration speed of a single node can reach 100MB/s, and the number of nodes supports horizontal expansion, which can meet the requirements of supporting TB and PB level data migration
to sum up
BDS has been launched on Alibaba Cloud to facilitate data migration and real-time data synchronization among self-built HBase, EMR HBase, and Apsara HBase.
It has served more than 40 customers on the cloud, helping customers achieve seamless migration between clusters, cluster upgrades, primary and backup disaster recovery, and data migration synchronization scale of more than 400T (single copy).
Customers who have migration needs on the cloud can view "BDS Service Introduction", or contact Side Tian:
https://help.aliyun.com/document_detail/120883.html
Friends who are more interested in the principles and solutions of BDS implementation can view "Design and Practice of BDS-HBase Data Synchronization Solution":
https://yq.aliyun.com/articles/704977
