The full amount of data is mainly for the historical data of multiple systems, and there are about tens of millions of data, which only needs to be initialized once.
The incremental data is the data of subsequent changes in the system.
This requirement is actually not simple, at least there are the following difficulties:
-
No direct access to third-party databases.
-
Historical data cannot be exported to excel, and there is a risk of data leakage.
-
How to quickly synchronize historical data?
-
How to deal with incremental data?
-
Does the interface need to be limited?
-
How to check the consistency of data for incremental data?
With these questions in mind, start today's article journey.
1. How to quickly synchronize historical data?
If you want to quickly synchronize historical data, the first thing that comes to mind may be to directly synchronize the data in the database.
However, in consideration of data security, multiple third-party systems cannot directly tell you their database access addresses and related account passwords.
Even if they tell you, there are many systems, and it is very troublesome for you to connect to the database to check the data one by one.
Some friends may say: This is easy to handle, let the third-party system export their historical data to excel, we can write a program to parse these excel, and then the data can be quickly imported into our database.
This is a good way, but it ignores one point: these data are sensitive data and cannot be exposed to the outside world.
So the solution of exporting to excel won't work.
So, how to quickly synchronize historical data?
Answer: use SFTP.
I don’t know if you have ever connected with a bank, but SFTP is often used in banking business.
So, how to synchronize data with SFTP?