Why split tables and partitions?
In daily development, we often encounter large tables. The so-called large tables refer to tables that store millions or even tens of millions of records. Such a large table will cause the database to take too long to query and insert, and the performance will be low. If the joint query is involved, the performance will be even worse. The purpose of table partitioning and table partitioning is to reduce the burden on the database and improve the efficiency of the database. Generally speaking, it is to improve the efficiency of adding, deleting, modifying and querying tables.
What is a sub-table?
Sub-table is to decompose a large table into multiple entity tables with independent storage space according to certain rules, we can call it sub-table, each table corresponds to three files, MYD data file, .MYI index file, .frm table structure file. These subtables can be distributed on the same disk or on different machines. When the app reads and writes, it gets the corresponding subtable name according to the pre-defined rules, and then operates it.
What is a partition?
Partitions are similar to sub-tables in that they decompose tables according to rules. The difference is that the partition table decomposes the large table into several independent entity tables, while the partition divides the data into multiple locations for storage, which can be the same disk or different machines. After partitioning, it still appears to be a table, but the data is hashed to multiple locations. When the app reads and writes, it still operates the name of the large table, and the db automatically organizes the partitioned data.
What is the relationship between mysql table and partition?
1. Both can improve the high performance of mysql, and have a good performance in high concurrency state.
2. Table subdivision and partition are not contradictory and can cooperate with each other. For those tables with a large number of visits and a large amount of table data, we can adopt a combination of table subdivision and partitioning (if the table subdivision method of merge cannot be combined with partitioning) If you cooperate, you can use other sub-tables to try), the number of visits is not large, but the table with a lot of table data, we can adopt the partition method.
3. The table-splitting technology is more troublesome, you need to manually create the sub-table, and the app server needs to calculate the sub-table name when reading and writing. It is better to use merge, but also create a union relationship between sub-tables and configure sub-tables.
4. Compared with sub-tables, table partitioning is easy to operate and does not need to create sub-tables.
There are several ways to divide the table:
1. mysql cluster
It is not a sub-table, but it serves the same function as a sub-table. The cluster can share the number of operations on the database and share the tasks among multiple databases. Clusters can separate read and write, reducing read and write pressure. Thereby improving database performance.
2. Custom rules sub-table
A large table can be decomposed into multiple sub-tables according to business rules. Usually there are the following types, or you can define your own rules.
Range – This mode allows data to be divided into different ranges. For example, a table can be divided into several partitions by year.
Hash (Hash) - This mode allows the calculation of the Hash Key of one or more columns of the table, and finally partitions the data area corresponding to different values of the Hash code. For example, you can create a table that partitions the table's primary key.
Key (key value) - an extension of the above Hash mode, where the Hash Key is generated by the MySQL system.
List (Predefined List) – This mode allows the system to divide data by a predefined list of values.
Composite (composite mode) - a combination of the above modes.
The table partitioning rules are the same as the partitioning rules, which are described in detail in the partitioning module.
The following is a brief introduction to how to divide the table (according to the year table) with Range.
Suppose the table structure has 4 fields: auto-increment id, name, deposit amount, deposit date
Use the deposit date as a rule table and create several tables respectively
2011: account_2011
2012: account_2012
……
2015: account_2015
When the application reads and writes, it looks up the corresponding table name according to the date, which needs to be determined manually.
var getTableName = function() {
var data = {
name: 'tom',
money: 2800.00,
date: '201410013059'
};
var tablename = 'account_';
var year = parseInt(data.date.substring(0, 4));
if (year < 2012) {
tablename += 2011; // account_2011
} else if (year < 2013) {
tablename += 2012; // account_2012
} else if (year < 2014) {
tablename += 2013; // account_2013
} else if (year < 2015) {
tablename += 2014; // account_2014
} else {
tablename += 2015; // account_2015
}
return tablename;
}3. Use the merge storage engine to implement sub-tables
The merge sub-table is divided into a main table and a sub-table. The main table is similar to a shell, and logically encapsulates the sub-table. In fact, the data is stored in the sub-table.
We can insert and query data through the main table, and if we know the rules of sub-tables, we can also directly operate the sub-tables.
Sublist 2011
CREATE TABLE `account_2011` (
`id` int(11) NOT NULL AUTO_INCREMENT ,
`name` varchar(50) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL ,
`money` float NOT NULL ,
`tradeDate` datetime NOT NULL
PRIMARY KEY (`id`)
)
ENGINE=MyISAM
DEFAULT CHARACTER SET=utf8 COLLATE=utf8_general_ci
AUTO_INCREMENT=2
CHECKSUM=0
ROW_FORMAT=DYNAMIC
DELAY_KEY_WRITE=0
;
子表2012年
CREATE TABLE `account_2012` (
`id` int(11) NOT NULL AUTO_INCREMENT ,
`name` varchar(50) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL ,
`money` float NOT NULL ,
`tradeDate` datetime NOT NULL
PRIMARY KEY (`id`)
)
ENGINE=MyISAM
DEFAULT CHARACTER SET=utf8 COLLATE=utf8_general_ci
AUTO_INCREMENT=2
CHECKSUM=0
ROW_FORMAT=DYNAMIC
DELAY_KEY_WRITE=0
;
主表，所有年
CREATE TABLE `account_all` (
`id` int(11) NOT NULL AUTO_INCREMENT ,
`name` varchar(50) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL ,
`money` float NOT NULL ,
`tradeDate` datetime NOT NULL
PRIMARY KEY (`id`)
)
ENGINE=MRG_MYISAM
DEFAULT CHARACTER SET=utf8 COLLATE=utf8_general_ci
UNION=(`account_2011`,`account_2012`)
INSERT_METHOD=LAST
ROW_FORMAT=DYNAMIC
;When creating the main table, there is an INSERT_METHOD, which indicates the insertion method. The value can be: 0 does not allow insertion; FIRST is inserted into the first table in UNION; LAST is inserted into the last table in UNION.
When querying through the main table, it is equivalent to querying all sub-tables together. This does not reflect the advantages of sub-tables. It is recommended to query sub-tables.
Several ways to partition
Range：
create table range(
id int(11),
money int(11) unsigned not null,
date datetime
)partition by range(year(date))(
partition p2007 values less than (2008),
partition p2008 values less than (2009),
partition p2009 values less than (2010)
partition p2010 values less than maxvalue
)；
List：
create table list(
a int(11),
b int(11)
)(partition by list (b)
partition p0 values in (1,3,5,7,9),
partition p1 values in (2,4,6,8,0)
);
Hash：
create table hash(
a int(11),
b datetime
)partition by hash (YEAR(b)
partitions 4;
Key：
create table t_key(
a int(11),
b datetime)
partition by key (b)
partitions 4;
分区管理
新增分区
ALTER TABLE sale_data
ADD PARTITION (PARTITION p201010 VALUES LESS THAN (201011));
删除分区
--当删除了一个分区，也同时删除了该分区中所有的数据。
ALTER TABLE sale_data DROP PARTITION p201010;
分区的合并
下面的SQL，将p201001 - p201009 合并为3个分区p2010Q1 - p2010Q3
ALTER TABLE sale_data
REORGANIZE PARTITION p201001,p201002,p201003,
p201004,p201005,p201006,
p201007,p201008,p201009 INTO
(
PARTITION p2010Q1 VALUES LESS THAN (201004),
PARTITION p2010Q2 VALUES LESS THAN (201007),
PARTITION p2010Q3 VALUES LESS THAN (201010)
);