First, the main reason for database performance bottlenecks
1, the database connection
MySQL database default connection is 100, we can be tuned by configuring initialSize, minIdle, maxActive, etc., but due to limitations of hardware resources, increasing the database connection can not be unlimited, there may be the largest single application connected to a large single-instance database the number of cases can not meet the actual demand, then the system will be traffic congestion.
2, the amount of data table (storage space problem)
It is commonly believed that database read performance bottlenecks single table data volume over 10 million. From the point of view of the index, if the index is not hit, the database system will scan the entire table, the greater the amount of data, a full table scan time will be longer; even if the index is hit, since the B + TREE index is located in the hard disk , the greater the amount of data B + TREE deeper level, the more the number of IO.
3, the hardware resource constraints
Hardware resources directly affects the QPS QPS / TPS transactions per second.
Second, the performance data optimization
Common Data Performance Optimization: SQL optimization, caching, create the index, separate read and write, sub-library sub-lists.
Large amount of data to solve performance optimization, a truly effective solution is to use distributed data storage, i.e. above the separate read and write sub-library and the sub-table.
1, separate read and write
Separation based on the master copy from the reader, using the difference between reading and writing of multiple data sources way of storing and loading data. Storing data (CRUD) specify the data sources to write, read data query specifies read data source.
Separating the master copy of the same data source (a master from multiple), multiple data sources can be deployed on different hosts, so that data can be solved in connection hardware resource constraints and bottlenecks by reading and writing.
2, sub-library sub-table
Library table data split, the management data slice manner.
A vertical split a single database repository database split fields, each field better portability database, function more clearly divided. But also to solve the database connection, the hardware resource constraints. Any plan to solve the problem at the same time, will bring new problems, sub-library sub-table is no exception, such as relational query becomes complex, distributed transaction issues.
Split level is a large table split into several small table according to certain rules, such as the split 30 million data amount of a single-user Table 3 small table user01, user02, user03. Mainly to solve the problem single-table large amount of data, which would resolve database performance bottlenecks caused by memory space.
The next article will focus on the points table MyCat its application in terms of sub-library of more detailed sub-library sub-table implementation.