In the previous article " Redis efficient and safe non-stop data migration solution ", the performance and advantages of NineData in the Redis migration scenario were introduced. Because data migrates between active and backup, multi-cloud, and multi-region environments, data consistency problems inevitably arise, and inconsistency between structure and data is often one of the causes of failures. Therefore, how to quickly and accurately complete Redis data comparison has become an important issue.
Now, NineData has implemented a data comparison service that is simple to configure, efficient, and secure while supporting uninterrupted business, which effectively protects the problem of data inconsistency caused by data migration in various scenarios. After actual testing, NineData can complete the data comparison of 8 million keys (2GB, involving all types) in 1 minute, with an average comparison rate of 130,000/second. It can also find inconsistent keys and generate detailed comparison reports to help you quickly Locate and fix inconsistent data. Address: https://www.ninedata.cloud/compare
At present, the main methods of data comparison include: parsing and traversing RDB binary files, SCAN traversal instances and some open source tools for data comparison. There will be some problems with the comparison of these methods:
Poor compatibility. Different versions of Redis and RDB file formats change. Comparison of ways to parse RDB files requires development and adaptation.
- The usability is poor. After the comparison is completed, the repair operation cannot be automatically generated.
- The reliability is poor. For open source tools, breakpoints cannot be broken after comparing exceptions.
- The operation and maintenance is poor, and operations such as progress viewing, pause, and alarm cannot be performed.
NineData’s efficient data comparison solution
NineData not only provides "data replication" and "visual management", but also further expands Redis's comparison capabilities, which well meets Redis's requirements for data consistency. Compared with traditional comparison, NineData's Redis data comparison capability has the following advantages:
- Simple and easy to use: task configuration can be completed in one minute, and comparison tasks are fully automated.
- Strong performance: Through core technologies such as multi-threading, queue optimization, and streaming memory management, the comparison performance reaches 130,000/second, effectively ensuring comparison efficiency.
- Complete functions: Supports consistency comparison of Redis data, and provides one-time comparison, periodic comparison, quick comparison and other comparison functions suitable for a variety of usage scenarios, and also provides the ability to repair inconsistent data.
- Visual interface: It has a simple and intuitive user interface that allows data comparison without complicated settings, and the comparison results are presented in intuitive charts and reports, allowing you to easily understand data differences.
- Safe and reliable: The NineData platform has passed the Level 3 network security level protection certification of the Ministry of Public Security, providing high-level protection for enterprise information security.
Through the above advantages, NineData's leadership in Redis comparison scenarios is guaranteed. In addition, NineData also provides comparison functions, including full, fast and inconsistent recheck comparison methods, and also supports different comparison frequencies. After the migration or copy is completed, the quality of the data can be effectively guaranteed through comparison.
Compare link configurations
1. Configure the task name and select the source data source and target data source to be compared.
2. Comparison frequency, you can choose one-time comparison or regular comparison as needed.
3. Comparison method, you can choose full data comparison or customized sampling ratio comparison.
Select comparison object
Select comparison objects: You can select different DBs for comparison.
Configure mapping object
Configuration mapping: You can map multiple databases of the source instance to one or more databases specified by the target instance.
pre-check
Through complete check items, the stability of the comparison task is ensured. At this point, we have completed the configuration of a fast and efficient Redis comparison task. After completing the configuration and starting the task, NineData will implement fully automated data comparison.
At the same time, in order to provide a better comparison experience, NineData provides complete observation and intervention capabilities for the comparison process. It not only provides detailed status, progress, and details of object comparison, but also reveals the internal execution of background threads through monitoring and logs, helping users track the progress of comparison in an all-round way. At the same time, it also provides alarms and comparison pause capabilities for abnormal situations that may occur during operation, allowing users to independently and quickly diagnose and repair abnormalities to ensure the stability of comparison. Information during comparison:
You can see the compared RPS in the monitoring and understand the compared performance in real time. In addition to monitoring, if data inconsistency is encountered, a "View Details" button will appear in the operation column. Go in and you can see the inconsistent information about the specific key:
Finally, you can see the "SQL" button in the operation column, which provides operations to repair inconsistent data:
Summarize
NineData data comparison is a cloud-native data comparison product with the efficient ability to process 1 million records per second. It provides one-stop support and is suitable for self-built IDC, self-built cloud hosts and cloud databases. It also supports a variety of databases, including Redis, MySQL, SQL Server, PostgreSQL, MongoDB, ClickHouse, Doris, etc., allowing structural comparison and data comparison between databases.
In addition, NineData also supports data consistency verification between homogeneous and heterogeneous data sources. Users can choose multiple comparison methods such as structural comparison, full comparison, sampling comparison, and periodic comparison. NineData can dynamically monitor the database load and adaptively adjust the comparison speed to ensure the stability of the database. Rich data source support enables NineData to meet the needs of different customers. In addition, NineData has industry-leading coverage in terms of structural comparison, and is No. 1 in the industry in object type and attribute coverage.
https://www.ninedata.cloud/compare