Why did you and partition table?
Our growing database data, followed by too much data in a single table. So that the query chance to read slow, and because the locking mechanism is also operating table cause the application to search a serious impact on the emergence of database performance bottlenecks.
mysql there is a table lock mechanism and a locking row, in order to ensure data integrity. You can not lock the table represents operate on this table, so I have to finish the job on the operating table. Line lock, too, other sql I have to wait for data manipulation finished this article, in order to operate these pieces of data. When this happens, we can consider the points table or partition.
1, sub-table
What are the points table?
Sub-table is broken down into a large table having a plurality of independent entity table storage according to certain rules, each table corresponding to the three documents, MYD data file, .MYI index file, .frm file table structure. These tables can be distributed on the same disk, or on a different machine. app give the corresponding write time table in accordance with pre-defined rules, and then to operate it.
When a single database table split, split into a plurality of data tables, and accessed by the user, according to certain algorithms (e.g., a hash manner, can also be used a remainder (modulo) manner), allows users to access different table, so that the data dispersed to a plurality of data tables, reducing the access pressure a single data table. Improved database access performance. The purpose of the score sheet in this, reduce the burden of the database, shorten the query time.
Mysql sub-table is divided into vertical and horizontal segmentation segmentation
Vertical segmentation means splitting the data tabulated, to split a column of a table is more multiple tables
Usually we split vertically according to the following principles :
The field is not used on a single table ;
The text, BLOB ( binary Large Object, binary large object) and other large fields split out on Schedule ;
Column often combined query on a table ;
Vertical resolution step more time should be executed in the data table design at the beginning, then the query time with jion key up to.
Refers horizontal split split data table row, a data table split into multiple tables to store.
Split level principles
Generally, we use the hash, modulus, etc. to split a table
For example, there is a user table 400W of users, in order to improve its query efficiency we divided into four tables users1, , users2, users3, users4
By using the method of the modulo ID dispersed into four data table Id% 4 = [0,1,2,3]
Then query , update , delete, query is the method by modulo
Part of the business logic may also be resolved by archiving region, year and other fields ;
Table after the split, then we will constraint the user query behavior. For example, we are by years of split , this time in the page design is constrained user must first select years before to search.
Sub-table in several ways:
1) MySQL Cluster
It is not the score sheet, but the points table and played the same role. Clusters can share the database of the number of operations, will share tasks across multiple databases. Clusters may separate read and write, read and write reduced pressure. So as to enhance database performance.
2) estimated in advance and the large amount of data occur frequently accessed table, divided into several tables
The certain algorithm (e.g., a hash manner, can also be used a remainder (modulo) manner) to allow different users to access the table.
For example forum inside the posting of the table, this table a long time must be very large, hundreds of thousands, millions are likely. Chat inside information table, a chat with dozens of people a night, a long time, the data for this table must be very large. Like many such cases. Therefore, this data can be projected out of the large scale, we will advance the separation of a N tables, that N is the number, according to the actual situation. In the chat information table, for example: we had to build 100 such tables, message_00, message_01, message_02 .......... message_98, then the user's message_99. Determines this information into which the user ID of a chat tables inside, you can use hash way to get, it can also be used to obtain the remainder way, many ways.
Or it may be designed to accommodate the amount of data for each table is the N, then how to determine whether the data a particular table is full of it? To be added to the table data, prior to insertion of the number of recording statistics do first operation block, when <N pieces of data, direct insertion, when the threshold has been reached, can create a new database table in the block (or has been previously created), then insert operation).
3) using the merge storage engine to achieve sub-table
If you want to separate the existing large scale data more painful, the most painful thing is to change the code, because the program inside the sql statement has been written, with the merge storage engine to achieve the points table , this method is more suitable.
merge sub-table, the sub-divided into a main table and the table, the master table is similar to a shell that encapsulates the logical sub-table, the data is actually stored in the child table.
We can insert the main table, and query the data, if the sub-table clearly rule may directly operate the child table.
Let's implement a simple to use merge storage engine to achieve the sub-table presentation:
Create a complete list of all of the member information storage (table named member)
mysql> drop database IF EXISTS test;
mysql> use test;
create table member(
id bigint auto_increment primary key,
name varchar(20),
sex tinyint not null default '0'
)engine=myisam default charset=utf8 auto_increment=1;
Join point data:
mysql> insert into member(name,sex) values('tom1',1);
mysql> insert into member(name,sex) select name,sex from member;
The second statement execute it several times there is a lot of data
mysql> select * from member;
+----+------+-----+
| id | name | sex |
+----+------+-----+
| 1 | tom1 | 1 |
| 2 | tom1 | 1 |
| 3 | tom1 | 1 |
| 4 | tom1 | 1 |
| 5 | tom1 | 1 |
| 6 | tom1 | 1 |
| 7 | tom1 | 1 |
| 8 | tom1 | 1 |
| 9 | tom1 | 1 |
| 10 | tom1 | 1 |
| 11 | tom1 | 1 |
| 12 | tom1 | 1 |
| 13 | tom1 | 1 |
| 14 | tom1 | 1 |
| 15 | tom1 | 1 |
| 16 | tom1 | 1 |
+----+------+-----+
Let's divide the table, where we put member in two tables tb_member1, tb_member2
mysql> use test;
DROP table IF EXISTS tb_member1;
create table tb_member1(
id bigint primary key ,
name varchar(20),
sex tinyint not null default '0'
) ENGINE=MyISAM DEFAULT CHARSET=utf8 ;
DROP table IF EXISTS tb_member2;
create table tb_member2(
id bigint primary key,
name varchar(20),
sex tinyint not null default '0'
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
// Create tb_member2 can also use the following statement to create table tb_member2 like tb_member1;
Create a master table tb_member
DROP table IF EXISTS tb_member;
create table tb_member(
id bigint primary key ,
name varchar(20),
sex tinyint not null default '0'
) ENGINE=MERGE UNION=(tb_member1,tb_member2) INSERT_METHOD=LAST CHARSET=utf8 ;
Note: the INSERT_METHOD, this parameter INSERT_METHOD = NO indicates that the table can not do any writing operation only used as a query , INSERT_METHOD = LAST denotes the last inserted inside a table. INSERT_METHOD = first inserted into the first table showing the inside.
Look at the structure tb_member table :
mysql> desc tb_member;
mysql> desc tb_member;
+-------+-------------+------+-----+---------+-----------------------------------------+
| Field | Type | Null | Key | Default | Extra |
+-------+-------------+------+-----+---------+-----------------------------------------+
| id | bigint(20) | NO | PRI | NULL | auto_increment |
| name | varchar(20) | YES | | NULL | |
| sex | tinyint(4) | NO | | 0 | |
+-------+-------------+------+-----+---------+------------------------------------------+
3 rows in set (0.00 sec)
Note: Check the child table and main table field definitions to be consistent
Next, we divide the data into two tables to go:
mysql> insert into tb_member1(id,name,sex) select id,name,sex from member where id%2=0;
mysql> insert into tb_member2(id,name,sex) select id,name,sex from member where id%2=1;
View data from two sub-tables:
mysql> select * from tb_member1;
+----+------+-----+
| id | name | sex |
+----+------+-----+
| 16 | tom1 | 1 |
| 14 | tom1 | 1 |
| 12 | tom1 | 1 |
| 10 | tom1 | 1 |
| 8 | tom1 | 1 |
| 6 | tom1 | 1 |
| 4 | tom1 | 1 |
| 2 | tom1 | 1 |
+----+------+-----+
8 rows in set (0.00 sec)
mysql> select * from tb_member2;
+----+------+-----+
| id | name | sex |
+----+------+-----+
| 3 | tom1 | 1 |
| 1 | tom1 | 1 |
| 5 | tom1 | 1 |
| 7 | tom1 | 1 |
| 9 | tom1 | 1 |
| 11 | tom1 | 1 |
| 13 | tom1 | 1 |
| 15 | tom1 | 1 |
+----+------+-----+
8 rows in set (0.00 sec)
Look at the data in the main table:
mysql> select * from tb_member;
+----+------+-----+
| id | name | sex |
+----+------+-----+
| 16 | tom1 | 1 |
| 14 | tom1 | 1 |
| 12 | tom1 | 1 |
| 10 | tom1 | 1 |
| 8 | tom1 | 1 |
| 6 | tom1 | 1 |
| 4 | tom1 | 1 |
| 2 | tom1 | 1 |
| 15 | tom1 | 1 |
| 13 | tom1 | 1 |
| 11 | tom1 | 1 |
| 9 | tom1 | 1 |
| 7 | tom1 | 1 |
| 5 | tom1 | 1 |
| 3 | tom1 | 1 |
| 1 | tom1 | 1 |
+----+------+-----+
16 rows in set (0.00 sec)
mysql> select * from tb_member where id=3;
+----+------+-----+
| id | name | sex |
+----+------+-----+
| 3 | tom1 | 1 |
+----+------+-----+
1 row in set (0.00 sec)
Note: The total tables are a housing, accessing data in a place in which a sub-table.
Note: Each table has its own separate sub-table-related documents, while the main table is only a shell, and there is no complete list of related documents
[root@localhost ~]# ls -l /usr/local/mysql/data/test/tb_member*
-rw-r-----. 1 mysql mysql 8614 Sep 15 21:49 /usr/local/mysql/data/test/tb_member1.frm
-rw-r-----. 1 mysql mysql 320 Sep 16 00:02 /usr/local/mysql/data/test/tb_member1.MYD
-rw-r-----. 1 mysql mysql 2048 Sep 16 00:43 /usr/local/mysql/data/test/tb_member1.MYI
-rw-r-----. 1 mysql mysql 8614 Sep 15 21:50 /usr/local/mysql/data/test/tb_member2.frm
-rw-r-----. 1 mysql mysql 180 Sep 16 00:02 /usr/local/mysql/data/test/tb_member2.MYD
-rw-r-----. 1 mysql mysql 2048 Sep 16 00:43 /usr/local/mysql/data/test/tb_member2.MYI
-rw-r-----. 1 mysql mysql 8614 Sep 16 21:12 /usr/local/mysql/data/test/tb_member3.frm
-rw-r-----. 1 mysql mysql 0 Sep 16 21:12 /usr/local/mysql/data/test/tb_member3.MYD
-rw-r-----. 1 mysql mysql 1024 Sep 16 21:12 /usr/local/mysql/data/test/tb_member3.MYI
-rw-r-----. 1 mysql mysql 8614 Sep 16 21:14 /usr/local/mysql/data/test/tb_member.frm
-rw-r-----. 1 mysql mysql 53 Sep 16 21:14 /usr/local/mysql/data/test/tb_member.MRG
Sub-tables and partitions (ie, the difference between sub-library)
1 , the implementation
A) MySQL sub-sub-table is a real table, a table is divided into many tables after every small tables are complete a table, corresponds to three files, one .MYD data file, .MYI index files . frm file table structure.
b) partition is not the same, after a large table partition, he was a table, the table does not become two, but he blocks stored data becomes more
2 , data processing
After a) sub-table, the data points are stored in the table, a summary table just housing, accessing data in a place in which one part table.
b) partition it, there is no concept of partition table, partition, but the files are stored data is divided into many small pieces, after the partition table, or is it a table, or data processing done by themselves.
3 , improve performance
After a) sub-table, table concurrency improved single disk I / O performance is improved. Why concurrency to improve it, because it takes time to search for a shortened, if high concurrency appear, the total table according to different queries, concurrent pressure into different small tables inside.
b) mysql proposed the concept of partition, mainly trying to break through the disk I / O bottlenecks, want to improve literacy disk to increase mysql performance.
At this point, the focus of measuring the partition table and points of different points table focused on accessing data, how to improve the mysql concurrent capacity; and partition it, how to break the disk reading and writing skills, so as to improve the purpose mysql performance.
4 , the degree of ease of implementation
There are many ways a) of the table, with the merge to sub-table, it is the simplest way. In this way the degree of difficulty with a partition similar to the program code and it can be done transparent. If you are using other sub-table way trouble the score zone.
b) partitioning implementation is relatively simple, the establishment of the partition table, root build the usual table makes no difference, and end off the code is transparent.
mysql score sheet, and partitioning What is the connection?
1. Can improve high-mysql, there is a good state at high concurrent performance.
2. Sub-partition table and not contradictory, can cooperate with each other for the big number of visits, and more table data table, we can take the form of a combination of sub-partition table and little traffic, but a lot of table data table, we It may take the form of partition and so on.
3. The technical sub-table is too much trouble, you need to manually create the child table, App server to read and write when you need to calculate the child table name. Using merge better, but also created between the child table and the child table to configure union relations.
4. The sub-table with respect to the partition table, convenient operation, no need to create sub-table.
Remember mysql characteristics : There are table and row locks, using the primary key update is update row lock will not affect other data to improve the performance of concurrent use of non-primary key update will cause table locks, affect concurrent performance. .