Practical background
projects and some table space is too big, and too many lines, it was decided to table a number of sub-library sub-table. Further Study Selection program, we noticed some common sub-library sub-table solutions for business more code changes, it was decided to adopt MySQL partitioning scheme.
In fact, in my personal opinion, the partition table is MySQL help us to achieve the underlying sub-library sub-table, need not involve modification of business codes, distributed transaction need not be concerned. Because in terms of access to the database, logic, or only a table, but in fact does have multiple physical partitions objects, queries the specific partitions based on specific zoning rules.
Introduce the practice table, table space 172G, 1 120 million records.
Database version: RDS MySQL 5.6
tool: Ali cloud DTS
First, why partition?
Advantages:
for expired or no saved data, you can quickly delete data by deleting data related to these partitions, it is more efficient than DELETE high
when the partition containing the conditions necessary to scan only one in the where clause or a plurality of partitions to improve query efficiency
, for example, the following statement:
the sELECT * the pARTITION the FROM T (p0, p1) WHERE C <. 5 selects only the records that match the criteria WHERE partition p0 and p1 are
directed to the aggregate function SUM (), COUNT () of when the query will be processed in parallel on each partition
partition originally a data table is stored on multiple physical disks to achieve higher IOPS
Cons:
can not use foreign keys, do not support full-text indexing (now no company should also with a foreign key, right)
partition key design is not flexible, if you do not take the partition key, it is prone to the whole table lock
developers to write a SQL, mysql is not clear how to play
Two, RANGE partitioning
currently supports MySQL range partitioning (RANGE), list partitioning (LIST), hash partitioning (HASH) and KEY partitions four.
This article is based on range partitioning (RANGE) partition of time, so I will briefly explain RANGE partitions. See more partition type partition type of official documents MySQL 5.6
on a given column value continuation interval, according to the interval assigned partition. The most common is based on the time field. In fact, based on the partition of the integer column it is best if you can use the date type function converted to an integer. MySQL 5.6 Supported partition function
used in this example TO_DAYS function of
the CREATE TABLE Members (
ID VARCHAR (25) the NOT NULL,
FirstName VARCHAR (25) the NOT NULL,
LastName VARCHAR (25) the NOT NULL,
username VARCHAR (16) the NOT NULL,
In Email VARCHAR (35),
joindate DATETIME the NOT NULL the DEFAULT '0000-00-00 00:00:00',
a PRIMARY KEY (ID, joindate) the USING BTREE,
KEY idx_joindate (joindate) the USING BTREE
) ENGINE = the InnoDB the DEFAULT the CHARSET = UTF8 the ROW_FORMAT the COMPACT =
PARTITION BY RANGE (TO_DAYS (joindate) ) (
P0 the VALUES LESS THAN the PARTITION (TO_DAYS ( '1960-01-01')),
the PARTITION P1 the VALUES LESS THAN (TO_DAYS ( '1970-01-01')),
the PARTITION P2 the VALUES LESS THAN (TO_DAYS ( '1980-01-01 ')),
the pARTITION P3 the VALUES LESS THAN (TO_DAYS (' 1990-01-01 ')),
the pARTITION the VALUES LESS THAN MAXVALUE P4
);
the PS: If you have a primary key or a unique index, you have to put your partitions like the examples also add key, which is joindate partitioning key, or create will fail!
PS: as above plus LESS THAN MAXVALUE, you can not later add a new partition! ! !
Example: The
following query will fall on the index partition p2 defined. Therefore, when the query key will bring down your partition corresponding to partition the data query, if your condition across a plurality of partitions aggregate function SUM (), COUNT () queries, it will be processed in parallel on each partition . If you did not take full table partition key query will query.
explain partitions select * from members WHERE joindate BETWEEN '1970-02-03' AND '1970-02-04';
I found a particularly interesting phenomenon at the time of completion of the migration of data query, if the time interval of the partitioning key is not the same as a SQL, it will be less to go a different range of indices based on the number of rows rows. As it is how to achieve the underlying I did not go to study the
Third, partition management
A brief introduction to the scope of partitions, and then say something about the partition of common operations.
Partition Manager includes an increase for the partition, delete, and query. More detailed partition management MySQL official documentation
1. Increase the partition
to RANGE and LIST partitions:
the ALTER the Table table_name the Add Partition (Partition P0 values ... (exp))
.. // Copyright 2011 All Rights Reserved at The Go the Authors
// the Use Source code governed the this IS of A by BSD-style
// License CAN BE found that in at The LICENSE File.
// Package errors implements functions to manipulate errors.
package errors
// New returns an error that formats as the given text.
func New(text string) error {
return &errorString{text}
}
// errorString is a trivial implementation of error.
type errorString struct {
s string
}
FUNC (E * errorString) Error () {String
return ES
}
The above code is not complicated, with reference to the above, we write a custom error-handling.
Custom error handling
define a alarm.go, for processing alarms.
Ado, a direct look at the code.
package alarm
import (www.chaoyuepint.com
"encoding/json"
"fmt"
"ginDemo/common/function"
"path/filepath"
"runtime"
"strings"
)
type errorString struct {
s string
}
type errorInfo struct {
Time string `json:"time"`
Alarm string `json:"alarm"`
Message string `json:"message"`
Filename string `json:www.zbzxyL12.com"filename"`
Line int `json:www.yisheng3yuL.com"line"`
Funcname string `json:www.seocelve.com"funcname"`
}
func (and errorString *) Error (www.chenghylpt.com) {string
return es
}
func New (text string) error {
alarm("INFO", text)
return &errorString{text}
}
// 发邮件
func Email (text string) error {
alarm("EMAIL", text)
return &errorString{text}
}
// 发短信
func Sms (text string) error {
alarm("SMS", text)
return &errorString{text}
}
// 发微信
func WeChat (text string)www.renheyL.com error {
alarm("WX", text)
return &errorString{text}
}
// alarming method
FUNC Alarm (Level String, String STR) {
// current time
currentTime: = function.GetTimeStr ()
// definition file name, line number, method name
fileName, line, functionName: =, 0, "?" "?"
pc, fileName, line, ok www.feironggw.cn:= runtime.Caller(2)
if ok {
functionName = runtime.FuncForPC(pc).Name()
functionName = filepath.Ext(www.ztylegw.cn functionName)
functionName = strings.TrimPrefix(functionName, ".")
}
var msg = errorInfo {
Time : currentTime,
Alarm : level,
Message : str,
Filename : fileName,
Line : line,
Funcname : functionName,
}
jsons, errs := json.Marshal(msg)
if errs != nil {
fmt.Println("json marshal error:", errs)
}
errorJsonInfo := string(jsons)
fmt.Println(errorJsonInfo)
Level == IF "EMAIL" {
// implementation of e-mail
IF the else == Level} "the SMS" {
// perform texting
IF the else == Level} "WX" {
// perform micro channel
Level the else == IF} "INFO" {
// implementation of the diary
}
}
look at how call:
package v1
import (
"fmt"
"ginDemo/common/alarm"
"ginDemo/entity"
"github.com/gin-gonic/gin"
"net/http"
)
The AddProduct FUNC (C * gin.Context) {
// Get Get Parameter
name: = c.Query ( "name" )
var res = entity.Result{}
STR, ERR: = Hello (www.hengxyul.com name)
! IF ERR = nil {
res.SetCode (entity.CODE_ERROR)
res.SetMessage (err.Error ())
c.JSON (http.StatusOK, RES)
C. ABORT (www.chengmingdl.com)
return
the ALTER TABLE Members the ADD the pARTITION (TO_DAYS ( '2021-03-01'));
2. delete partition
deleted partition, will also delete all the data in the partition. If you delete a partition lead to a partition can not cover all the value, then insert the data when error.
Partition P0 the Table table_name drop the ALTER;
3. Discover how many partitions
SELECT * FROM INFORMATION_SCHEMA.PARTITIONS WHERE TABLE_NAME = ' members';
Four, Data Migration
said in front of so many concepts, I talk about this large table to migrate data to a partition the process table.
DTS Why did you choose it? Because it can be non-stop migration of data to support the full amount of migration and incremental migration, it has little effect on the original table.
The migration process is as follows:
First, in the same instance of RDS built inside with a partition table structure
to create a migration task using DTS, do not choose when to migrate structure Qian, select only the full amount + incremental migration
Then also need to edit the next target database table name, which is done under the migration mapping A-> B from
turning off write data tasks, when the task queue is empty, wait a few minutes to pause and end of the migration task
last modified table name, data migration and switching (my test environment to modify the partition table for some time, but RDS modify the table name is changed in seconds)
refer to the official document: MySQL 5.6 partitions
than purely personal point of view, if not please correct me.