Methods, steps and experience of data migration
In the project, the problem of historical data migration after the system is completely replaced is often encountered to show respect for the customer's historical work, not to mention that many data still need to be retained.
How to do historical data migration?
System Analysis :
1. Analyze the original business system
Accurate to the approximate system function modules and the approximate processing flow
2. Analyze the existing business system
Accurate to the approximate system function modules and the approximate processing flow
3. Analyze the differences and differences between the two
A rough analysis of the differences between the two business systems can help determine workload and progress
4. Analyze user needs for legacy data
Analyze the demand for old data, so as not to blindly migrate completely
5. Analyze the user's processing rules for old data
The old data processing rules are generally divided into the following categories:
1. Basic data, usually this type is easy to migrate, and the data format is simple, but it will affect all related business data. The focus is on the way of the primary key and unique key of the data.
2. The import of pure historical data is for reference only, this type of data is easy to import
2.1 Pure historical data
This type of data is easier to process, and it can be imported at one time, and then incremental data import will be used later.
2.2 Process data
This type of data can only be terminated after the record is completely closed. Incremental import and data update are required. At the same time, the development of the relevant query interface is required to ensure that the old data can be queried in the new system.
3. Historical data with large changes in the structure of the new and old system tables
The workload of this type of data is the heaviest, and it is necessary to carefully study the data structure of the new and old business systems.
3.1. Try to collect all relevant design documents of the original system through Party A as much as possible. This is very helpful for data analysis. After all, it is not reliable to analyze through human feeling and observation of data.
3.2. Observing relevant data on the original system to understand the relationship between data changes and data table data (very helpful for difficult-to-understand related fields)
3.3. Compare the differences between the old and new system data. If it is really unreliable, it is recommended to deal with it according to 2.2.
system design:
1. After completing the system analysis, classify the relevant data, basic data, pure historical data, and historical data with large changes
2. Start simple and give yourself some confidence
3. Compare the data dictionary of related tables in the excel table, outline the corresponding fields, conversion logic, dependencies, and if necessary, make corresponding redundancy on the new system table, and then clear it after the data migration is completed.
key point:
The problem of matching the field types of different databases, such as the text of SQL Server, should correspond to clob in Oracle, but it is relatively easy to convert it into several varchar2 from the perspective of implementation.
Regarding the primary key issue, try to maintain the current state for consistent data types, and try to use oracle sequence or sqlserver identity int for inconsistent data types.
System implementation:
1. PL/SQL or T-SQL, DTS, SSIS or PowerBuilder pipelines as data migration methods
2. Migration of related business logic, rewriting of stored procedures, triggers, functions related to data integrity and business, and even rewriting of SQL
3. Compilation of business query pages
Afterword:
1. There is no special or advanced approach to data migration
2. The analysis of business data is the most critical, and it is necessary to constantly try and test on the new system
3. Data migration technology is mainly implemented through SQL, stored procedures, and even cursors, and the priorities are as above
There is also a kind of data migration that is only the migration of the database or the migration of heterogeneous databases
Database migration, that is, it needs to be migrated from one server to another server for performance expansion, which can be processed with the export and import of the database or the backup and recovery tool. Of course, the initial values of some sequence fields after the migration should also be considered.
Heterogeneous data migration, that is, migration from one database platform to another database platform, can be implemented with ETL tools or SQL, but pay attention to the migration of business logic, that is, stored procedures, functions, triggers, etc.
refer to: