In the era of cloud computing, big data and the Internet of Things, digital transformation is quietly taking place in all walks of life. However, as the most basic element, how to carry out multi-source and heterogeneous mass data exchange? How to ensure the timeliness and accuracy of data? How to make massive heterogeneous data realize the collaboration between cloud and cloud? It is the focus of our attention for a long time. The ZMP data migration platform is dedicated to data exchange scenarios between clouds, supports different data structures, and can help government and enterprise customers to quickly build a high-performance, high-security, high-reliability, and high-stability data exchange platform.
1. Function
The ZMP data migration platform not only has basic system, user and topic management, but also server management and message monitoring for visual operation and maintenance. More importantly, it integrates middleware such as message service, data conversion and synchronization, and adds migration evaluation and application. smart transformation.
-
Server management: Provides functions such as server, message service, and synchronization component installation, start-stop, configuration, and monitoring.
-
Topic management: Provides the creation, deletion, editing and monitoring of message service topics. In data exchange, users do not need to create and delete topics, and are managed by the data exchange task in a unified manner.
-
Data migration evaluation: Provide data collection, portrait and compatibility evaluation of source database (Oracle9i, Oracle11g, Oracle12c, SQLserver) database and destination database as Yunxi database or MySQL.
-
Data collection: Data collection tools are built for multiple Oracle versions and SQL Server.
-
Source database portrait: According to the collected data, automatically generate an intelligent five-sided graph (scale, session, risk, load, complexity), summary and detailed information of the database - including database performance and runtime performance, capacity, Oracle features , object details, panoramic search, etc. Users can grasp the above information intuitively and view specific information on the details page. At the same time, in the dependencies of different objects, jumps can also be implemented to help understand the original structure of the database.
-
Compatibility evaluation: parse all the objects in the source library and SQL statements, determine the compatibility between them and the target library, and output the statistical results. In the evaluation details, you can see the compatibility and migration risks of each object.
-
Data exchange: Provide functions for creating, deleting, editing, monitoring and statistical reporting of jobs, tasks, data sources and synchronization tables.
-
Support structured data exchange: Yunxi database, Oracle, SQLserver, DB2, Sybase, MySQL, PostgreSQL, Greenplum, Redis, Cache, Kudu, etc.
-
Support unstructured data exchange: Linux/Windows files, FTP, SFTP, OSS, HDFS various styles, data exchange of various sizes.
-
Support semi-structured data exchange: JSON, CSV and other semi-structured data exchange with database.
-
Support multi-protocol data collection: MQTT, COAP, MODBUS, XPH, DICOM, HTTP, RTSP, RTMP, etc.
-
Intelligent transformation: Load the source code (project) folder, sniff the files in it, find the SQL statements in it, and perform statistics, analysis, transformation, and statistical analysis of the transformation situation, and provide related functions such as manual viewing and transformation .
-
User management: Provides functions such as user creation, deletion, editing, permission assignment and auditing.
-
System configuration: Provides functions such as console logs and system alarms.
-
Version management: Provides functions such as synchronizing component installation package updates and synchronizing component upgrades.
2. Advantages
On the basis of supporting multiple data sources, the ZMP data migration platform provides a visual drag-and-drop method for process orchestration. It not only greatly reduces the impact on the source end, but also improves the synchronization efficiency. At the same time support large files and various network data transmission.
-
High synchronization efficiency: more than 100,000 records of synchronization per second.
-
Small source impact: Supports incremental synchronization of Oracle, SQLserver, MySQL, MariaBD and Dameng logs without relying on triggers, CDC, and timestamps.
-
Multi-field timestamp: supports multi-timestamp, incremental field, supports timestamp and incremental field offset forward.
-
DDL change capture: supports database level and table field level
-
Data management and monitoring: Visual drag-and-drop method is used for process choreography, visual management and monitoring of data flow, data statistics, data reports and related warnings.
-
Support structured data exchange: Yunxi database, Oracle, SQLserver, DB2, Sybase, MySQL, PostgreSQL, Greenplum, Redis, Cache, Kudu, etc.
-
Large file and network data transmission: support TB-level files in complex environments, support relay routing, file gatekeeper, limited bandwidth and other network data transmission, and support breakpoint resuming.
-
Localized environment and middleware: Supports CPU architectures such as mips, x86, and arm, as well as Kingdee, Zhongchuang, etc.
3. Architecture
Functional Architecture:
Technology Architecture:
Database Exchange Architecture:
File Exchange Architecture:
4. Application
Various types of data, big data platforms and industry applications can achieve efficient data exchange through the ZMP data migration platform. Structured, unstructured, file and other data generated in medical communication and other industries are extracted, converted, encrypted, transmitted and stored through the ZMP platform to maximize the value of the data. At the same time, data collaboration on and off the cloud.
Deploying platforms across networks, firewalls, relay routes, and gatekeepers can use a variety of different application scenarios. Heterogeneous collection has little impact on data sources, realizes efficient data collection, reliable transmission, and supports mass message storage; data exchange supports processing of millions of messages per second on a single node, and the cluster supports a throughput of 100 million messages per second, and the message capacity can reach PB level.
