Sub-library smooth expansion sub-table

For our common sub-library sub-table program, there are a lot of advantages, the expansion of sub-library sub-table is a headache, if support for db layer made consistent hash, or central parity, its too costly, if not the case, can only be down for maintenance treatment, it will have an impact on high availability.

Is there a program that can not only rapid expansion without reducing availability? This one, we talk about expansion program sub-library sub-table for everyone to explore.

First, the horizontal component library extensions issues

In order to increase concurrency db, a common scenario is that the data sharding, which is often said that the sub-library sub-table, the need to have a data plan is expected early in order to pre-allocate sufficient library to handle.

For example, currently planned three databases, based on uid be taken over fragmentation, then the rules on the division of each library are as follows:

As we can see, the data can be evenly distributed to three inside the database.

However, if the speed of the subsequent business development rapidly, the amount of data a large number of users increases, the current capacity is insufficient to support, how should I do?

The need for horizontal expansion of the database, and then add a new library to decompose. After the addition of the new library, the original data sharding to three libraries, you can sharding to four inside the library

However, this time due to fragmentation rules were changed (uid% 3 becomes uid% 4), most of the data can not hit, you need to re-allocate in the original database, large amounts of data need to migrate.

Before UID1 example by dispensing on uid1% 3 A library, after the addition of the new library D, the algorithm to uid1% 4, then there is probably above the partitioned library B. If you have seen before, "Principles and Practice of consistent hashing", you will find a new node, there will be about 90% of the data needs to be migrated, the pressure on the DB students is still quite large, then how to deal with?

There are usually several ways.

Second, stop taking migration

Stop taking the migration is the most common type of program, the following general procedure:

1. Estimated stop taking time to stop taking the announcement release

2. Stop taking, through the data migration tool well in advance, according to the new rules of fragmentation, migration

3. Modify fragmentation rules

4. Start Service

We see this way is safer, no data is written, can guarantee the normal migration after stop taking, there is no consistency problems. The only problem is to stop taking the time and the pressure.

1. Stop taking and hurt the user experience, while also reducing server availability

2. Must complete the migration within the developing time, if it fails, another day times need be. While increasing the pressure on developers, prone to major accident

3. When the huge amount of data, migration requires a lot of time

That there is no other way to improve the look, we look at the following two options.

Third, the upgrade from the library

Online database, in order to maintain its availability, generally each host with a library from the library, reading and writing in the master library, then the library to the master-slave synchronization. As follows, A, B is the primary library, A0 and B0 from the library.

At this time, when capacity is needed, we put A0 and B0 upgraded to the new main library node, so the two become four and warehouses and warehouses. While the upper slice configuration, good mapping rules are as follows:

uid% 4 = 0 and uid% 4 = 2 respectively, and point A A0, i.e. the data points before uid% 2 = 0, the split uid% 4 = 0 and uid% 4 = 2

uid% 4 = 1 and uid% 4 = 3 in the point B and B0, i.e. point data uid% 2 = 1 before, split uid% 4 = 1 and uid% 4 = 3

Because the same data repository A and A0, B0 and B the same data, so in this case without the need for data migration can be. Only need to change it to slice configuration, updated by the distribution center, without having to restart.

Since the data before uid is partitioned between 2% 2 pools which, when dispersed into four banks, because the old data exist (uid% 4 0, and half data uid =% = 2. 4), it is necessary to redundant data to do a clean-up.

And this clean-up will not affect the consistency of data online, but anywhere, anytime.

After processing is complete, in order to ensure high availability, and demand for the next expansion. A library can be allocated from the existing main library again.

To sum up this scheme as follows:

1. Modify slice configuration, do the mapping of the new library and the old library.

2. Synchronization configuration, upgrade the main library from the library

3. Lift the master-slave relationship

4. Redundant data clean-up

5. Build a new library for the new data from the node

Fourth, double the migration

Double the program is more for online database migration to use, of course, for the sub-library is also extended to migrate data, therefore, can also help the problem of sub-library expansion.

Principle the same as above, do split expansion, but the data synchronization method is different.

1. Add the new library to write link

Double the core principle is the need for expansion of the database, adding new libraries, and increased write-link the existing fragmentation, while two write data.

Because the data for the new library is empty, so CRUD data has no effect, in the upper layer of logic, or in the old library-based data.

2. New and old library data migration

By tools to migrate data to the new library old library inside, then you can select the data (1/2) after the split sync to synchronize, it can fully synchronous, full synchronization is generally recommended, the final checksum data to do a good deal of time .

3. Data checksum

According to the ideal environmental conditions, after data migration, because it is a dual-write operations, so both sides of the data is consistent, in particular, insert and update, very high consistency of the situation. But the real network environment, there will be delays, etc., to delete the situation is not very satisfactory, such as:

A library when deleting a data, a data migration is not yet written to the C library, then delete the C library has been executed, the C library will be more of a data.

At this point you need to do the data checksum, data checksum can do a few times, until the data is almost the same, as far as possible subject to the data from the old library.

4. Fragmentation Configuration Review

Data synchronization is complete, you can put the new library fragment mapping reprocessed, or be split according to the old library the way,

Before u uid% 2 = 0, becomes uid% 4 = 0 and the uid% 4 = 2

uid% 2 = 1, becomes uid% 4 = 1 and the uid% 4 = 3.