Open source big data comparison platform design and practice—dataCompare
The previous article introduced the design and practice of the open source big data comparison platform, and recently improved and developed the overall business process.
1. The current version implements the following functions:
(1) Low-code and simple configuration to complete the core function of data comparison
(2) Data magnitude comparison and data consistency comparison
2. Demonstration of system functions
System home page (system function and technology stack introduction)
Database configuration information page (currently, MySQL and Hive have been implemented)
Job configuration information page (select data source and table as well as primary key and comparison field)
Comparison results page display (magnitude comparison, consistency comparison)
3. Subsequent development plan:
(1) Difference case discovery
(2) Data pointer detection - enumeration value detection, range detection, value detection, primary key modulo detection
(3) Timing and automatic scheduling of data comparison tasks
(4) Automatically send an email report for the comparison result
4. Specific code reference
https://github.com/zhugezifang/dataCompare
https://gitee.com/ZhuGeZiFang/data-compare