Architecture components: based on Shard-Jdbc sub-database sub-table, database expansion plan

1. Database expansion

1. Business scenario

In Internet projects, there are many business scenarios that "the amount of data is large, the business complexity is high, and the database and tables need to be divided."
Insert picture description here
Such a layered architecture
(1) the upper layer is the business layer biz, which implements business logic encapsulation;
(2) the middle is the service layer service, which encapsulates data access;
(3) the lower layer is the data layer db, which stores business data;

2. Expansion scenarios and problems

When the amount of data continues to increase, facing such some requirements, the two databases cannot accommodate, and the database needs to be expanded. Here, choose the mode of 2-expansion to 3, as shown in the figure below:
Insert picture description here
The problem of such expansion
(1) The problem of sub-database and sub-table The strategy leads to a large amount of data migration;
(2) It affects the continuous serviceability of the data;
(3) The specified time is completed, and the technical pressure is high, which may easily lead to unexpected errors;

How to smoothly migrate data without stopping the machine and ensure the continuous service of the system is the question to be discussed in this article.

Second, the expansion solution

1. Diagram of the expansion plan

Insert picture description here
(1) Sub-database and sub-table are based on MySQL database, using shard-jdbc middleware
(2) The whole idea of this solution is based on SpringCloud microservice architecture

2. Solve the problem of expansion

(1) There is no need to suspend the service in the case of capacity expansion;
(2) The pressure of data migration is small and no specified time is required;

3. Data access layer logic

Insert picture description here
Solution description
Based on two databases sub-database sub-table, abbreviation: service two
Based on three database sub-database sub-table, abbreviation: service three
(1) provide two sets of services, service two and service three
(2) after the database expansion, if you visit Service 3 obtains the data directly, and the process ends.
(3) If access to service three cannot obtain the data, access service two to obtain the data.
(4) During the period of the beginning of the migration, the access pressure will still be on the second service.
(5) In this way, the data access service will not be downtime.
(6) This access mode is easy to do based on SpringCloud.

4. Data migration layer logic

Insert picture description here
Program description
(1) Close the data storage process based on two libraries
(2) Open the data storage process based on three libraries, so that the newly received data can be directly accessed by Service 3.
(3) Develop data migration middleware and scan the data of the original two libraries.
(4) The scanned data is judged whether it needs to be migrated according to the three-database strategy.
(5) If the data needs to be migrated, call the data storage interface of service three.
(6) After the data migration is completed, delete the data in the original location.
(7) This migration mode is easy to do based on SpringCloud.

5. The advantages of this program migration

(1) The whole process is to continuously provide services online;
(2) The development complexity of data migration middleware is low;
(3) It can be migrated slowly at a limited speed without time pressure;

Three, write at the end

The next article will update the program based on the SpringCloud code implementation.
Insert picture description here