Sub-library and sub-table
Single library single table
Single database single table is the most common database design. For example, there is a user (user) table in the database db, and all users can be found in the user table in the db database.
Single library with multiple tables
As the number of users increases, the amount of data in the user table will increase. When the amount of data reaches a certain level, the query on the user table will gradually slow down, thus affecting the performance of the entire DB. If you use mysql, there is a more serious problem that when you need to add a column, mysql will lock the table, and all read and write operations can only wait.
Users can be divided horizontally in some way to generate two tables with exactly the same structure as user_0000, user_0001, etc. The data of user_0000 + user_0001 + ... is just a complete data.
Multi-cudo table
As the amount of data increases, the storage space of a single DB may not be enough, and as the amount of queries increases, a single database server can no longer support it. At this time, the database can be divided horizontally.
Sub-library and sub-table rules
When designing a table, it is necessary to determine what rules the table is divided into. For example, when there is a new user, the program has to determine which table to add the user information to; similarly, when logging in, we have to find the corresponding record in the database through the user's account, all of which need to follow a certain rule conduct.
routing
The process of finding the corresponding table and library through the sub-library and sub-table rules. For example, the rule of sub-database and sub-table is the method of user_id mod 4. When a user newly registers an account, the account id is 123. We can determine that this account should be saved in the User_0003 table by the method of id mod 4. When user 123 logs in, we confirm that it is recorded in User_0003 after passing 123 mod 4.
Problems arising from sub-database and sub-table, and matters needing attention
1. The problem of sub-database and sub-table dimensions
If a user purchases a product, the transaction record needs to be saved and retrieved. If the table is divided according to the user's latitude, each user's transaction record is stored in the same table, so it is very fast and convenient to find the purchase of a user. However, when a certain commodity is purchased, it is likely to be distributed in multiple tables, which is more troublesome to find. On the contrary, according to the product dimension, it is easy to find the purchase status of this product, but it is more troublesome to find the transaction records of the buyer.
Therefore, the common solutions are:
a. Solve by scanning the table, this method is basically impossible, and the efficiency is too low.
b. Record two pieces of data, one is divided according to user latitude, and the other is divided according to product dimension.
c. It is solved by search engine, but if real-time requirements are very high, it must be related to real-time search.
2. The problem of joint query
Union queries are basically impossible because the associated tables may not be in the same database.
3. Avoid cross-database transactions
Avoid modifying the table in db1 when modifying the table in db0 in one transaction. One is that the operation is more complicated and the efficiency will also have a certain impact.
4. Try to put the same set of data on the same DB server
For example, put the commodities and transaction information of seller a into db0. When db1 is hung up, things related to seller a can be used normally. That is to say, avoid data in a database to depend on data in another database.
One master and multiple backups
In practical applications, in most cases, reading is far greater than writing. Mysql provides a read-write separation mechanism. All write operations must correspond to the Master. Read operations can be performed on the Master and Slave machines. The structure of Slave and Master is exactly the same. You can hang Slave, which can effectively improve the QPS of the DB cluster.
All write operations are performed on the Master first, and then synchronously updated to the Slave. Therefore, there is a certain delay in synchronizing from the Master to the Slave machine. When the system is very The latency problem is exacerbated when it is busy, and the increase in the number of slave machines will also make the problem worse.
In addition, it can be seen that the Master is the bottleneck of the cluster. When there are too many write operations, the stability of the Master will be seriously affected. If the Master hangs up, the entire cluster will not work properly.
Therefore, 1. When the reading pressure is very high, you can consider adding the fractional solution of the slave machine, but when the number of slave machines reaches a certain number, you have to consider the sub-library. 2. When the writing pressure is very high, the sub-library operation must be performed.
----------------------------------------------
Why should MySQL use points? Database sharding
can be used to say where MySQL is used. As long as the amount of data is large, there will be a problem immediately. It is necessary to shard the database and shard the table.
Here is a question. Why do you need to shard the database and shard the table? MySQL cannot handle large tables ?
In fact, it is a large table that can be handled. In the project I have experienced, the physical file size of a single table is more than 80G, and the number of records in a single table is more than 500 million, and this table
belongs to a very core table: friend relationship table.
But This method can be said to be not the best method. Because there are many problems with the file system such as the Ext3 file system in handling large files.
This layer can be replaced with the xfs file system. However, there is a problem when the MySQL single table is too large. It is not easy to solve: the operation related to table structure adjustment is
basically impossible. Therefore, the application of sub-database and sub-table will be monitored in the use of major items.
From Innodb itself, there are only two locks on the Btree of the data file, and the leaf node locks It can be imagined that when a page is split or a
new leaf is added, data cannot be written to the table.
Therefore, the sub-database and sub-table are still a better choice.
Then the sub-database and sub-table are more appropriate . What?
After testing 10 million records in a single table, the performance of writing and reading is better. In this way, if you leave some buffers, then the single table is all data fonts and keeps below
8 million records, and there are characters in the single table Tables are kept below 5 million.
If you plan according to 100 databases and 100 tables, such as user business:
5 million*100*100 = 50000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000uuuuuuuuuuuuuuuh
, it is relatively easy to plan according to the business.
Reprinted from: http://wentao365.iteye.com/blog/1740874
Sub-library and sub-table
Single library single table
Single database single table is the most common database design. For example, there is a user (user) table in the database db, and all users can be found in the user table in the db database.
Single library with multiple tables
As the number of users increases, the amount of data in the user table will increase. When the amount of data reaches a certain level, the query on the user table will gradually slow down, thus affecting the performance of the entire DB. If you use mysql, there is a more serious problem that when you need to add a column, mysql will lock the table, and all read and write operations can only wait.
Users can be divided horizontally in some way to generate two tables with exactly the same structure as user_0000, user_0001, etc. The data of user_0000 + user_0001 + ... is just a complete data.
Multi-cudo table
As the amount of data increases, the storage space of a single DB may not be enough, and as the amount of queries increases, a single database server can no longer support it. At this time, the database can be divided horizontally.
Sub-library and sub-table rules
When designing a table, it is necessary to determine what rules the table is divided into. For example, when there is a new user, the program has to determine which table to add the user information to; similarly, when logging in, we have to find the corresponding record in the database through the user's account, all of which need to follow a certain rule conduct.
routing
The process of finding the corresponding table and library through the sub-library and sub-table rules. For example, the rule of sub-database and sub-table is the method of user_id mod 4. When a user newly registers an account, the account id is 123. We can determine that this account should be saved in the User_0003 table by the method of id mod 4. When user 123 logs in, we confirm that it is recorded in User_0003 after passing 123 mod 4.
Problems arising from sub-database and sub-table, and matters needing attention
1. The problem of sub-database and sub-table dimensions
If a user purchases a product, the transaction record needs to be saved and retrieved. If the table is divided according to the user's latitude, each user's transaction record is stored in the same table, so it is very fast and convenient to find the purchase of a user. However, when a certain commodity is purchased, it is likely to be distributed in multiple tables, which is more troublesome to find. On the contrary, according to the product dimension, it is easy to find the purchase status of this product, but it is more troublesome to find the transaction records of the buyer.
Therefore, the common solutions are:
a. Solve by scanning the table, this method is basically impossible, and the efficiency is too low.
b. Record two pieces of data, one is divided according to user latitude, and the other is divided according to product dimension.
c. It is solved by search engine, but if real-time requirements are very high, it must be related to real-time search.
2. The problem of joint query
Union queries are basically impossible because the associated tables may not be in the same database.
3. Avoid cross-database transactions
Avoid modifying the table in db1 when modifying the table in db0 in one transaction. One is that the operation is more complicated and the efficiency will also have a certain impact.
4. Try to put the same set of data on the same DB server
For example, put the commodities and transaction information of seller a into db0. When db1 is hung up, things related to seller a can be used normally. That is to say, avoid data in a database to depend on data in another database.
One master and multiple backups
In practical applications, in most cases, reading is far greater than writing. Mysql provides a read-write separation mechanism. All write operations must correspond to the Master. Read operations can be performed on the Master and Slave machines. The structure of Slave and Master is exactly the same. You can hang Slave, which can effectively improve the QPS of the DB cluster.
All write operations are performed on the Master first, and then synchronously updated to the Slave. Therefore, there is a certain delay in synchronizing from the Master to the Slave machine. When the system is very The latency problem is exacerbated when it is busy, and the increase in the number of slave machines will also make the problem worse.
In addition, it can be seen that the Master is the bottleneck of the cluster. When there are too many write operations, the stability of the Master will be seriously affected. If the Master hangs up, the entire cluster will not work properly.
Therefore, 1. When the reading pressure is very high, you can consider adding the fractional solution of the slave machine, but when the number of slave machines reaches a certain number, you have to consider the sub-library. 2. When the writing pressure is very high, the sub-library operation must be performed.
----------------------------------------------
Why should MySQL use points? Database sharding
can be used to say where MySQL is used. As long as the amount of data is large, there will be a problem immediately. It is necessary to shard the database and shard the table.
Here is a question. Why do you need to shard the database and shard the table? MySQL cannot handle large tables ?
In fact, it is a large table that can be handled. In the project I have experienced, the physical file size of a single table is more than 80G, and the number of records in a single table is more than 500 million, and this table
belongs to a very core table: friend relationship table.
But This method can be said to be not the best method. Because there are many problems with the file system such as the Ext3 file system in handling large files.
This layer can be replaced with the xfs file system. However, there is a problem when the MySQL single table is too large. It is not easy to solve: the operation related to table structure adjustment is
basically impossible. Therefore, the application of sub-database and sub-table will be monitored in the use of major items.
From Innodb itself, there are only two locks on the Btree of the data file, and the leaf node locks It can be imagined that when a page is split or a
new leaf is added, data cannot be written to the table.
Therefore, the sub-database and sub-table are still a better choice.
Then the sub-database and sub-table are more appropriate . What?
After testing 10 million records in a single table, the performance of writing and reading is better. In this way, if you leave some buffers, then the single table is all data fonts and keeps below
8 million records, and there are characters in the single table Tables are kept below 5 million.
If you plan according to 100 databases and 100 tables, such as user business:
5 million*100*100 = 50000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000uuuuuuuuuuuuuuuh
, it is relatively easy to plan according to the business.
Reprinted from: http://wentao365.iteye.com/blog/1740874