table of Contents
An old library table writing and closing
Three business impact is minimal
Four incremental data write back
Seven before, during and after the event, what should we do
This article mainly explains the problems encountered in the process of data migration from the old database to the new database, and the solutions given to these problems, hoping to give you a little inspiration or help.
An old library table writing and closing
Because multiple services of multiple business ends share an old database, and multiple services share a set of passwords for accessing the old database.
How to make sure that only we write to the old database table?
In order to exclude other business-side services directly connected to the old database and write the table of the old database, we add a field to the table, such as A, to identify that it is our client's write to the table. If it is not our end to write to the table, this field should be a null value.
Why is it necessary to be sure to write and close ?
Suppose there is a business-side service that writes to the old database table, and then we perform data migration, the data of the new and old database tables will be inconsistent, which is unbearable in the business.
2. Smooth migration of data
There are 4 tables on our side, with a total of 100 million data, which need to be migrated. The core points of this section are: how to solve the time-consuming problem of migrating data, and how to reduce the impact of table downtime on business measurement.
At the beginning, we decided to stop writing RD. DBA executes commands through scripts to directly migrate historical data from the old database to the new database. The evaluation of the whole process is expected to take 1 hour, which means that writing needs to be stopped for 1 hour, and then this The result is synchronized to the business side, and the business side cannot tolerate this result.
How to solve the time-consuming problem of data migration?
We use ALI's DTS to migrate data. DTS can synchronize historical data and incremental data.
The historical data is migrated, so how to smooth it?
We use the new database cluster as the slave database of the old database for data synchronization. When our business is stopped, and the data of the new database and the old database are consistent, the DBA will use a machine in the new database cluster Upgrade to the main library, and mount other machines to this main library as the slave library of this machine.
Three business impact is minimal
In the process of data migration, it is best if you can be indifferent to the business. So how to do it to minimize the impact on the business?
First, we analyze business data to find the trough of the business. Performing data migration operations in the low valley has little impact on the business.
Secondly, we stop writing at the minute level. When we stop writing, we are always readable. This is tolerable in business.
Four incremental data write back
Why do we need to write back data?
In the process of data migration, due to some team scheduling issues, the service reads the old database. At this time, in order to ensure that other teams can read the incremental data and ensure that the business is not affected, we use Databus to perform data processing. To synchronize, write back to the old library.
Of course, our expectation is to only maintain the new library, so we require other business ends to schedule the reading and closing. After 2 months, other business ends will use our interface to realize the reading and closing.
Five migration rollback plan
The rollback scheme is not necessarily needed in the data migration process, but it is still needed in terms of the integrity of one thing. For this piece of data migration, we need to consider service rollback and database rollback .
Service rollback?
It means that the service is switched from reading the new library to reading the old library. This section can be used as a data source dynamic switch, which can flexibly switch access to the new and old data sources.
Database rollback?
If it is incremental data, we need to synchronize them from the new database to the old database to ensure data integrity.
Liu Lao Library offline
Is it over after data migration? No, we still need to offline the old library.
From the perspective of company assets, with the new library, in order to reduce costs, we need to take the old library offline.
From the perspective of the completeness of one thing, we have only completed 90% of the data migration, and we need to offline the old database for the remaining 10%.
Seven before, during and after the event, what should we do
Beforehand: We should send emails to the company's R&D and business, the subject of the email, the scope of the email, the affected functions, the time of data migration, the person in charge, the migration plan, and the DingTalk communication group.
In the process: Our research and development should confirm the feasibility of the plan with each business side, establish close communication with DBA and QA, and synchronize the plan progress, development progress, and migration progress at any time.
After the event: Our R&D should send an email to inform the R&D and business, BI, big data and other teams that the database migration has been completed, and the migration results should be synchronized in time. The DingTalk group also needs to be synchronized.