Mysql super practical sub-library sub-table strategy, one hundred million data easily solve the problem
First, the sub-library sub-table background
In the era of data explosion, single-table data reaches millions of levels, volume and even billions of dollars, it is very common scenario. This time to operate the database is a very difficult thing, select all get a half-day data, this time the business has been difficult to sustain. A last resort, the sub-library sub-table on the agenda, our goal is simple, to reduce the pressure on the database, shorten the operation time table.
Second, how data segmentation
Data segmentation (Sharding), simple terms, is through certain conditions, the same data will be stored in a database to store a plurality of split database (host) in the case of dispersion so as to achieve a single machine load that sub-library sub-table. Depending on the data segmentation rules, there are two modes,
Vertical segmentation (sectioned longitudinally) , is different tables (or Schema) be segmented, stored on different databases (host).
Segmentation level (horizontal segmentation) , the data is the same table were segmented, stored on different databases (host). The logic rules are data in the table, according to some split conditions.
Vertical segmentation
Vertical segmentation, emphasis is split service . A database composed of a plurality of tables, each corresponding to a different service, then we may refer to the table are classified according to the different services, and distribute it to different databases, such data will be allocated to the different libraries above, do special purposes.
For example, the original list of goods database, transaction table, the Orders table, we can make vertical segmentation according to different business, the merchandise table, the transaction table, the Orders table are split into commodities library, library transaction, order the library go with.
Vertical Split advantages:
Clear split rule, after the split service definition; integration becomes easy to extend or between systems; becomes easy data maintenance; in accordance with the cost, the application level, application type, etc. placed on the table of the different machines, easy to manage . A vertical split disadvantages:
Not part of the business association table (the Join), can only be resolved by way of an interface, increases the complexity of the system; different limits by each service, there is a single database performance bottlenecks, easy to expand and improve the performance data; Distributed Transaction Processing Complex .
Level segmentation (focus)
The level of segmentation, emphasized that the resolution of the technical level. She is to the data table according to a certain logic rules across multiple libraries, each comprising a portion of the data in the table, all the tables add up to the total amount of data. In simple terms, we will have a level of data segmentation understood to be segmented according to the data line is to cut some rows in the table assigned to a database table, while the other line cut assigned to other database tables.
For example, the original database with a transaction table, the amount of data is very large, there are regions where the table fields, after in-depth research eligible split level. We split horizontally according to this field, according to different regions (Beijing, Shanghai, Jiangsu, Zhejiang, Guangdong and other) split into 10 libraries.
While peak periods 1,000,000 request, if single library, database request will withstand a pressure of 100 million times, split into 100 Table 10 were placed in the library, each table 10,000 requests, each database requests will withstand a pressure of 10 million times, so the pressure is reduced a lot, and is exponentially reduced.
Split level advantages:
Split good abstract rules, join the database can do basic operation; a single database performance bottlenecks large data, high concurrency does not exist; the application side reform less; improved stability with load capacity of the system. Split level disadvantages:
Split rule not abstract; fragmented transactional consistency is difficult to solve; data repeatedly extended and difficult; poor performance of cross-database join.
Third, the data points cut some of the problems caused by
Above we have talked about the advantages and disadvantages of the two data segmented patterns, but some of them common drawback,
The problem of distributed transactions; across nodes Join the problem; paging inter-node merge sort of problem; multi-source data management problems. In general, there is a complex scenario is difficult to join segmentation is often easy to separate business segmentation on business. How segmentation, we follow the following principles,
Segmentation can not try not to segmentation; if you want to be sure to choose the appropriate segmentation segmentation rule, planned in advance; segmentation data as much as possible to reduce the possibility of cross-database data through redundancy or Join group tables; because the database middleware Join the pros and cons achieve data is difficult to grasp, but extremely difficult to achieve high performance, business read as little as possible to use multi-table Join.
Fourth, data source management issues
After the sub-library sub-table, the management of data sources is the key to system implementation.
System-level system-level application code, there are mainly two ideas,
Client mode is configured in each application module to manage their needs one (or more) data sources, each directly access the database, to complete the integration of the data in the module. For example, you can rely on spring annotation achieve. Intermediate proxy mode, unified management of all data sources, the back-end database cluster transparent to the front-end application. Considering the complexity and scalability of the system, the proposed second intermediate proxy mode. While the cost of short-term need to pay may be relatively larger, but the expansion of the whole system, it is very practical.
2. middleware level
The above system-level code required to achieve more complex, is to add a layer of intermediate agent before the data cluster, such Cobar, Mycat other database middleware. Practical database middleware, the realization of the code level is a great liberation.
Fifth, the Distributed Transaction Solutions
The most prominent problems caused by sub-library sub-table is distributed transaction processing. Distributed Transaction section we distributed under the technical interview points if interested can refer to the previous article Internet architecture overview, followed by if you have time, you can explore the individual and then the next.
First, the sub-library sub-table background
In the era of data explosion, single-table data reaches millions of levels, volume and even billions of dollars, it is very common scenario. This time to operate the database is a very difficult thing, select all get a half-day data, this time the business has been difficult to sustain. A last resort, the sub-library sub-table on the agenda, our goal is simple, to reduce the pressure on the database, shorten the operation time table.
Second, how data segmentation
Data segmentation (Sharding), simple terms, is through certain conditions, the same data will be stored in a database to store a plurality of split database (host) in the case of dispersion so as to achieve a single machine load that sub-library sub-table. Depending on the data segmentation rules, there are two modes,
Vertical segmentation (sectioned longitudinally) , is different tables (or Schema) be segmented, stored on different databases (host).
Segmentation level (horizontal segmentation) , the data is the same table were segmented, stored on different databases (host). The logic rules are data in the table, according to some split conditions.
Vertical segmentation
Vertical segmentation, emphasis is split service . A database composed of a plurality of tables, each corresponding to a different service, then we may refer to the table are classified according to the different services, and distribute it to different databases, such data will be allocated to the different libraries above, do special purposes.
For example, the original list of goods database, transaction table, the Orders table, we can make vertical segmentation according to different business, the merchandise table, the transaction table, the Orders table are split into commodities library, library transaction, order the library go with.
Vertical Split advantages:
Clear split rule, after the split service definition; integration becomes easy to extend or between systems; becomes easy data maintenance; in accordance with the cost, the application level, application type, etc. placed on the table of the different machines, easy to manage . A vertical split disadvantages:
Not part of the business association table (the Join), can only be resolved by way of an interface, increases the complexity of the system; different limits by each service, there is a single database performance bottlenecks, easy to expand and improve the performance data; Distributed Transaction Processing Complex .
Level segmentation (focus)
The level of segmentation, emphasized that the resolution of the technical level. She is to the data table according to a certain logic rules across multiple libraries, each comprising a portion of the data in the table, all the tables add up to the total amount of data. In simple terms, we will have a level of data segmentation understood to be segmented according to the data line is to cut some rows in the table assigned to a database table, while the other line cut assigned to other database tables.
For example, the original database with a transaction table, the amount of data is very large, there are regions where the table fields, after in-depth research eligible split level. We split horizontally according to this field, according to different regions (Beijing, Shanghai, Jiangsu, Zhejiang, Guangdong and other) split into 10 libraries.
While peak periods 1,000,000 request, if single library, database request will withstand a pressure of 100 million times, split into 100 Table 10 were placed in the library, each table 10,000 requests, each database requests will withstand a pressure of 10 million times, so the pressure is reduced a lot, and is exponentially reduced.
Split level advantages:
Split good abstract rules, join the database can do basic operation; a single database performance bottlenecks large data, high concurrency does not exist; the application side reform less; improved stability with load capacity of the system. Split level disadvantages:
Split rule not abstract; fragmented transactional consistency is difficult to solve; data repeatedly extended and difficult; poor performance of cross-database join.
Third, the data points cut some of the problems caused by
Above we have talked about the advantages and disadvantages of the two data segmented patterns, but some of them common drawback,
The problem of distributed transactions; across nodes Join the problem; paging inter-node merge sort of problem; multi-source data management problems. In general, there is a complex scenario is difficult to join segmentation is often easy to separate business segmentation on business. How segmentation, we follow the following principles,
Segmentation can not try not to segmentation; if you want to be sure to choose the appropriate segmentation segmentation rule, planned in advance; segmentation data as much as possible to reduce the possibility of cross-database data through redundancy or Join group tables; because the database middleware Join the pros and cons achieve data is difficult to grasp, but extremely difficult to achieve high performance, business read as little as possible to use multi-table Join.
Fourth, data source management issues
After the sub-library sub-table, the management of data sources is the key to system implementation.
System-level system-level application code, there are mainly two ideas,
Client mode is configured in each application module to manage their needs one (or more) data sources, each directly access the database, to complete the integration of the data in the module. For example, you can rely on spring annotation achieve. Intermediate proxy mode, unified management of all data sources, the back-end database cluster transparent to the front-end application. Considering the complexity and scalability of the system, the proposed second intermediate proxy mode. While the cost of short-term need to pay may be relatively larger, but the expansion of the whole system, it is very practical.
2. middleware level
The above system-level code required to achieve more complex, is to add a layer of intermediate agent before the data cluster, such Cobar, Mycat other database middleware. Practical database middleware, the realization of the code level is a great liberation.
Fifth, the Distributed Transaction Solutions
The most prominent problems caused by sub-library sub-table is distributed transaction processing. Distributed Transaction section we distributed under the technical interview points if interested can refer to the previous article Internet architecture overview, followed by if you have time, you can explore the individual and then the next.