As one of the most widely used open source relational databases, MySQL has the characteristics of high performance, stability, and ease of use. It is the main data storage for many websites, applications, and commercial products. In some scenarios, if there are hundreds of millions of rows in a single table, development and DBA may be required to optimize the large table: table splitting, archiving, or expansion operations, all of which involve data migration. Therefore, it is very necessary to provide efficient, accurate, and stable large table migration capabilities.
1. Traditional Migration Solution
At present, data migration is mainly divided into logical migration and physical migration. Logical migration mainly includes mysqldump, mysqlpump and load data, and physical migration mainly includes XtraBackup. For this type of traditional migration scheme of importing, exporting and copying files, there are some problems in large table migration:
-
Downtime is required, and online services cannot be guaranteed for physical migration.
-
Poor performance, for logical migration, single-threaded export table.
-
The reliability is poor. After exporting an exception, breakpoints are not supported and need to be re-exported.
-
The operation and maintenance performance is poor. During the migration period, operations such as pausing and limiting the import and export cannot be performed.
2. Leading Migration Solution
The data replication provided by NineData includes both data migration and data synchronization capabilities, and provides efficient, stable, and maintainable large table migration capabilities without affecting business. Migration of large tables (227G single table, 1 billion rows): NineData completes the migration in about 30 minutes, with an average speed of 124.2 MB/s and an average of 534188 RPS.
Through the performance test of MySQL large table migration, compared with traditional migration, the advantages of NineData data migration are:
-
Easy to use: task configuration can be completed in one minute, and task migration can be fully automated.
-
Strong performance: Perfect core technologies such as intelligent sharding, row-level concurrency, and dynamic batch accumulation can effectively guarantee migration performance.
-
High reliability: Combined with new breakpoints, abnormal diagnosis and rich repair methods, it provides perfect disaster recovery capabilities for software and hardware failures that may occur during the migration process, greatly improving the success rate of large table migrations.
Through the above advantages, NineData's leading position in the large table migration scenario is guaranteed. In addition, NineData also provides a comparison function, including data and structure comparison, as well as full, fast and inconsistent re-inspection comparison methods, and also supports different comparison frequencies. After migration or replication, data consistency can be effectively guaranteed through data and structure comparison.
While providing powerful migration capabilities, NineData also ensures the simplicity of use. It only takes 1 minute to complete the configuration of the migration task, realizing a fully automated data migration process. Let's take a look at the configuration process of the entire task:
Migration link configuration
-
Configure the task name, select the source and target instances to migrate.
-
Select the type of copy, select structure and full copy (data migration) for data migration.
-
Select an appropriate conflict handling strategy as required.
-
Click Next to select the migration object.
Select migration object
Select migration objects: Select migration objects for different granularities: instances, libraries, tables, views, etc.
Configure mapping and filtering rules: The target instance can customize the migrated library name, table name, and perform partial data migration according to the filter conditions, and can also perform mapping and partial column migration for the column names of the table.
pre-check
Through rich check items, the stability of the migration task and the consistency with the data are guaranteed. So far, we have completed the configuration of a high-performance migration task.
At the same time, in order to provide a better migration experience, NineData provides comprehensive observation and intervention capabilities for the migration process. It not only provides detailed status, progress, and details of object migration, but also reveals the internal execution of background threads through monitoring and logs, helping users track migration progress in all directions. At the same time, it also provides basic diagnosis capabilities for abnormal situations that may occur during operation, as well as multiple repair strategies such as repair, skip, and removal, and migration and current limiting capabilities, allowing users to independently and quickly diagnose and repair links to ensure migration stability.
3. Summary
The efficient, fast and stable MySQL large table migration capability provided by NineData well complements the deficiencies of traditional migration solutions. Currently, NineData already supports the migration and replication of dozens of common databases. At the same time, in addition to the SAAS mode, it also provides an enterprise-specific cluster mode to meet the highest data security compliance requirements of enterprises. At present, NineData has completed large-scale application practice in many industries such as operators, finance, manufacturing, real estate, and e-commerce. If you are interested, you can log on to the official website https://www.ninedata.cloud/dbmigration and start using it for free immediately.