The whole network the most comprehensive MySQL performance optimization practice, battle programmer essential phrases! ! !

A Inscription

Recently the company projects to add new features on the line after a long time find a list of some query functions. The reason is a new feature to use the old function interface, and the SQL query interfaces associated with these old tables 5,6 and writing is not standardized, resulting in failure of the index MySQL in the implementation of SQL statements, a full table scan. Originally responsible for optimizing colleagues something to leave to go home, thus optimizing the query data problem falls in the hands of the author. The author after online information access SQL optimization successfully solved the problem, at this from a global point of view, record and summarize relevant MySQL query optimization techniques.

Second, the optimization ideas

Data query slow, does not mean that SQL statement written in question. First, we need to find the source of the problem, "the right medicine." Use a flow chart showing MySQL optimization ideas:

 

No further words can be clearly seen from the figure, resulting in slow data query a variety of reasons, such as: cache invalidation, within this period of time due to the high concurrent access leads to a MySQL server to crash; problem writing SQL statements; MySQL server parameters of the problem; hardware configuration limitations MySQL service performance issues.

Third, view the status of the MySQL server is running value

If the number of concurrent requests of the system is not high, and the query is slow, this step can be omitted SQL statements directly tuning step.

Excuting an order:

show status

Since the return too many results, where the results are not posted. Among them, the results back, we focus on the value of "Queries", "Threadsconnected" and "Threadsrunning", namely the number of inquiries, thread connections and the number of threads running.

We can execute the following script to monitor MySQL server is running state value

#!/bin/bash
while true
do
mysqladmin -uroot -p"密码" ext | awk '/Queries/{q=$4}/Threads_connected/{c=$4}/Threads_running/{r=$4}END{printf("%d %d %d\n",q,c,r)}' >> status.txt
sleep 1
done

Execute the script for 24 hours to obtain status.txt in the content, again the number of requests per second MySQL services == == calculated by awk

awk '{q=$1-last;last=$1}{printf("%d %d %d\n",q,$2,$3)}' status.txt

The calculated content copy to Excel graph generated periodically observed data.

If the cyclical changes observed data, as in FIG interpretation need to modify the cache invalidation policy.

E.g:

In [3,6,9] wherein the random number acquired by the interval value as a time of a cache miss, the cache expiration so dispersed, thus saving a portion of memory consumption.

When accessing the peak of the split part of the request to a cache failure, the other part is a MySQL database access, thus reducing the pressure MySQL server.

Fourth, the need to obtain optimized SQL statements

4.1 Method 1: View the running thread

Excuting an order:

show processlist

Return result:

mysql> show processlist;
+----+------+-----------+------+---------+------+----------+------------------+
| Id | User | Host      | db   | Command | Time | State    | Info             |
+----+------+-----------+------+---------+------+----------+------------------+
|  9 | root | localhost | test | Query   |    0 | starting | show processlist |
+----+------+-----------+------+---------+------+----------+------------------+
1 row in set (0.00 sec)

From the returned results, we can understand what the thread executing the command / SQL statements and execution time. Practical application, the query returns a result there will be N records.

Among them, the value of the State's return is the key to judge good or bad performance , its value appears following, the rows need to optimize the SQL statement:

Converting HEAP to MyISAM # 查询结果太大时，把结果放到磁盘，严重
Create tmp table #创建临时表，严重
Copying to tmp table on disk  #把内存临时表复制到磁盘，严重
locked #被其他查询锁住，严重
loggin slow query #记录慢查询
Sorting result #排序

4.2 way: turn on slow query log

In the configuration file my.cnf in the [mysqld] add the following line of two parameters:

slow_query_log = 1
slow_query_log_file=/var/lib/mysql/slow-query.log
long_query_time = 2

log_queries_not_using_indexes = 1

Wherein, slowquerylog = 1 indicates turning slow query; slowquerylogfile represents slow query log storage location; longquerytime = 2 indicates that the query> = 2 seconds before the log; logqueriesnotusing_indexes = 1 recording does not use the index SQL statement.

Note: The path slowquerylog_file can not just write, otherwise it might not have permission to MySQL server log files written to the specified directory. Recommended path copied directly above.

After modify save the file, restart the MySQL service. It creates a slow-query.log log files in / var / lib / mysql / directory. Connecting to a MySQL server can execute the following command to view the configuration.

show variables like 'slow_query%';

show variables like 'long_query_time';

Test slow query log:

mysql> select sleep(2);
+----------+
| sleep(2) |
+----------+
|        0 |
+----------+
1 row in set (2.00 sec)

Open the slow query log file

[root@localhost mysql]# vim /var/lib/mysql/slow-query.log
/usr/sbin/mysqld, Version: 5.7.19-log (MySQL Community Server (GPL)). started with:
Tcp port: 0  Unix socket: /var/lib/mysql/mysql.sock
Time                 Id Command    Argument
# Time: 2017-10-05T04:39:11.408964Z
# User@Host: root[root] @ localhost []  Id:     3
# Query_time: 2.001395  Lock_time: 0.000000 Rows_sent: 1  Rows_examined: 0
use test;
SET timestamp=1507178351;
select sleep(2);

We can see just two seconds to perform a SQL statement to be recorded.

Although the recording slow SQL query information in the slow query log, but the log records of intensive and difficult to view. Therefore, we need to filter out through SQL tool.

MySQL provides mysqldumpslow  tool for log analysis . We can use mysqldumpslow --help command to view the relevant usage.

Common parameters are as follows:

    -s：排序方式，后边接着如下参数
        c：访问次数
        l：锁定时间
        r：返回记录
        t：查询时间
    al：平均锁定时间
    ar：平均返回记录书
    at：平均查询时间
    -t：返回前面多少条的数据
    -g：翻遍搭配一个正则表达式，大小写不敏感

Case:

获取返回记录集最多的10个sql
mysqldumpslow -s r -t 10 /var/lib/mysql/slow-query.log

获取访问次数最多的10个sql
mysqldumpslow -s c -t 10 /var/lib/mysql/slow-query.log

获取按照时间排序的前10条里面含有左连接的查询语句
mysqldumpslow -s t -t 10 -g "left join" /var/lib/mysql/slow-query.log

 

V. Analysis of SQL statements

5.1 Method 1: explain

Screened problematic SQL, we can use the supplied explain MySQL View SQL execution plan case (association table, table query sequence, index usage, etc.).

usage:

explain select * from category;

Return result:

mysql> explain select * from category;
+----+-------------+----------+------------+------+---------------+------+---------+------+------+----------+-------+
| id | select_type | table    | partitions | type | possible_keys | key  | key_len | ref  | rows | filtered | Extra |
+----+-------------+----------+------------+------+---------------+------+---------+------+------+----------+-------+
|  1 | SIMPLE      | category | NULL       | ALL  | NULL          | NULL | NULL    | NULL |    1 |   100.00 | NULL  |
+----+-------------+----------+------------+------+---------------+------+---------+------+------+----------+-------+
1 row in set, 1 warning (0.00 sec)

Fields explained: 1) id: select the query sequence number. The same id, execution order from top to bottom; id different, the larger the value id higher priority, the first to be executed

2) select_type: query data type of operation, which values ​​are as follows:

• simple: a simple query, does not contain sub-queries or union
• primary: contain complex sub-queries, the outermost query marks for value
• subquery: or where the select comprising subqueries, this value is labeled
• derived: sub-query included in the list are marked from this value, MySQL will recursively these sub-queries, the results in a temporary table
• union: if the second select appears after the union, were marked for that value. If the union included in the sub query from the outer layer is labeled select derived
• union result: get results from the union table select

3) table: displaying the line data on which tables

4) partitions: a partition match

5) type: the type of connection table, its value, the following properties in the end of a high order:

• system: the table has only a row, corresponding to the system tables
• const: once through the index to find only one row of data matching
• eq_ref: unique index scan, for each index key, only one record in the table match. Commonly used in the primary key or unique index scan
• ref: non-unique index scanning, returns all rows matching a separate value. A =, <or> operator indexed columns
• range: only retrieves a row of a given range, using an index to select the rows. Generally used between,>, <situation
• index: only the index tree traversal
• ALL: full table scan, the worst performance

Note: The first five cases are ideal situation index usage. Usually optimized at least to the level range, it is best to optimize the ref

6) possible_keys: MySQL indicate which index to use in this table to find the rows. If the value is NULL, you do not use the index description can be indexed to improve performance

7) key: Displays the index MySQL actually used. If NULL, then do not use the index query

8) key_len: number of bytes used in the index, said index used in the query is calculated by the column length. Without loss in accuracy, the shorter the length, the better the display is the maximum length of the index field, not the actual length

9) ref: displays the index fields associated with the table which fields of which tables

10) rows: The selection table statistics and case, a rough estimate of the number of lines needed to locate the desired recording or reading, the smaller the value, the better

11) filtered: the percentage of the number of rows returned results representing the number of read rows, the better the value

12) extra: contain inappropriate displayed in the other columns but additional information is very important, common values ​​are as follows:

• using filesort: MySQL will be described using an external data ordering index, instead of reading the index order in the table. The emergence of value, should be optimized SQL
• using temporary: the use of a temporary table to hold intermediate results, MySQL use temporary tables when sorting query results. Common in order by sorting and grouping query group by. The emergence of value, should be optimized SQL
• using index: represented by a corresponding select operations using the covered index, to avoid data access table rows, good efficiency
• using where: where clause restricts which line
• using join buffer: Use connection caching
• distinct: the first find a match, stop searching for the current row combination of more lines

Note: before the advent of two values, SQL statements must be optimized.

5.2 Second way: profiling

Use profiling command to learn more information about SQL statements are consuming resources (cost of each step of execution).

5.2.1 Viewing profile case open

select @@profiling;

Return result:

mysql> select @@profiling;
+-------------+
| @@profiling |
+-------------+
|           0 |
+-------------+
1 row in set, 1 warning (0.00 sec)

0 represents off, turn 1 represents

5.2.2 Enable profile

set profiling = 1;  

Return result:

mysql> set profiling = 1;  
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> select @@profiling;
+-------------+
| @@profiling |
+-------------+
|           1 |
+-------------+
1 row in set, 1 warning (0.00 sec)

After the connection is closed, profiling state is automatically set to the closed state.

View a list of executed SQL 5.2.3

show profiles;

Return result:

mysql> show profiles;
+----------+------------+------------------------------+
| Query_ID | Duration   | Query                        |
+----------+------------+------------------------------+
|        1 | 0.00062925 | select @@profiling           |
|        2 | 0.00094150 | show tables                  |
|        3 | 0.00119125 | show databases               |
|        4 | 0.00029750 | SELECT DATABASE()            |
|        5 | 0.00025975 | show databases               |
|        6 | 0.00023050 | show tables                  |
|        7 | 0.00042000 | show tables                  |
|        8 | 0.00260675 | desc role                    |
|        9 | 0.00074900 | select name,is_key from role |
+----------+------------+------------------------------+
9 rows in set, 1 warning (0.00 sec)

Before the command is executed, SQL statements only need to perform additional recording.

5.2.4 Queries ID specified execution details

show profile for query Query_ID;

Return result:

mysql> show profile for query 9;
+----------------------+----------+
| Status               | Duration |
+----------------------+----------+
| starting             | 0.000207 |
| checking permissions | 0.000010 |
| Opening tables       | 0.000042 |
| init                 | 0.000050 |
| System lock          | 0.000012 |
| optimizing           | 0.000003 |
| statistics           | 0.000011 |
| preparing            | 0.000011 |
| executing            | 0.000002 |
| Sending data         | 0.000362 |
| end                  | 0.000006 |
| query end            | 0.000006 |
| closing tables       | 0.000006 |
| freeing items        | 0.000011 |
| cleaning up          | 0.000013 |
+----------------------+----------+
15 rows in set, 1 warning (0.00 sec)

Each line is a state change process as well as their duration. Status of this column and show processlist of State is the same. Therefore, attention needs to be optimized in the same points as described above.

5.2.5 obtain CPU, Block IO and other information

show profile block io,cpu for query Query_ID;

show profile cpu,block io,memory,swaps,context switches,source for query Query_ID;

show profile all for query Query_ID;

Sixth, optimization methods

Mainly in query optimization, indexing and table structure using design to explain.

6.1 Query Optimization

1) Avoid SELECT *, what data is needed, it queries the corresponding field.

2) small table-driven large table, i.e. small data set to drive large data sets. Such as: the A, B two table, for example, by associating two table id field.

When B is less than A set of table data table, Optimization in exist; used in, two tables is performed to check the sequence of Table B, Table A and then check

select * from A where id in (select id from B)

When A is less than B the table data set table, Optimization exist in; using EXISTS, two tables is performed to check the sequence of Table A, Table B and then check

select * from A where exists (select 1 from B where B.id = A.id)

3) In some cases, the connection can be used instead of the sub-queries, since the use of join, MySQL will not create a temporary table in memory.

4) suitably added a redundant field, reduce the associated table.

5) the rational use of the index (described below). Such as: to sort, group, index fields, to avoid the filesort. More: MySQL index to a data structure and optimize inventory

6.2 Use Index

6.2.1 for use of the index scene

1) primary key automatically creates a unique index

2) field frequently as the query conditions

Field 3) the query associated with other tables

4) sort the query field

5) statistical query or group field

6.2.2 scenes not suitable for use index

1) frequently updated fields

2) with field conditions where less than

3) table records too

4) regular table CRUD

5) the difference field value or little high repeatability

6.2.3 index creation and use principles

1) single-table query: Which columns for the query, create an index on the column

2) multi-table queries: When you left join, the index added to the right-table related field; when the right join, the index added to the left-field correlation table

3) Do not perform any operation (calculation, type conversion) column index

4) Do not use the index column! =, <> Non-equal

5) the column index is not empty, and not use is null or is not null Analyzing

6) The index field is a string type, the query condition to increase the value of 'single quotation marks, type the bottom to avoid automatic conversion

Contrary to the above principles may result in failure of the index, the specific circumstances need to explain command to view it

6.2.4 Failure index case

In addition to breach of the principles of index creation and use, the following circumstances can lead to failure of the index:

1) When the fuzzy inquiry, beginning with%

2) when using or, such as: field 1 (non-indexed) or field 2 (index) will cause failure index.

3) When using a composite index, without using a first index column.

index (a, b, c), to field a, b, c as a composite index, for example:

 

6.3.1 Select the appropriate data type database table structure 6.3 Design

1) Use is possible to store data of a minimum data type

2) the use of simple data types. int varchar type in mysql process than simple

3) make use tinyint, smallint, mediumint rather than as integral type int

4) define the field as much as possible to use not null, because null occupy 4 bytes of space

5) minimize the use of the text type, it is not advisable to consider when using a non-sub-table

6) Try to use instead of datetime timestamp

7) Do not have too many single table field, within the recommended 20

Split Table 6.3.2

When the data in the database is very large, query optimization plan will not solve the problem of slow queries, we can consider splitting tables, each table so that the data becomes small, thus improving query efficiency.

1) Vertical Split: a plurality of columns in the table are separated into different tables. For example, some of the fields in the user table is accessed frequently, these fields in a table, a number of additional fields not used in another table. When insert data, using transactions to ensure data consistency of two tables.

2) Horizontal Split: split in rows. User table e.g., user ID, user ID to access more than 10, the even distribution of the user data to the user 10 in Table 0-9. When Find also query data in accordance with this rule.

6.3.3 separate read and write

Under normal circumstances for the database are "read-write less." In other words, the pressure of the database mostly because a large number of operations to read data caused. We can use the database cluster solutions, use the main library as a library, is responsible for writing data; other library from the library, is responsible for reading data. This will ease the pressure on access to the database.

Seven, server tuning parameters

7.1 memory-related

sortbuffersize sort buffer memory size

buffer size using the connection joinbuffersize

When the full table scan allocated buffer size readbuffersize

7.2 IO related

Innodblogfile_size transaction log size

Innodblogfilesingroup number of transaction logs

Innodblogbuffer_size transaction log buffer size

Innodbflushlogattrx_commit transaction log refresh strategy, its values ​​are as follows:

0: write every second log cache, and flush log to disk

1: presented to the log write cache in each transaction, and flush log to disk

2: Every time a transaction is committed, execution log data written to the cache, execute once per second flush log to disk

7.3 Security-related

expirelogsdays specify automatic cleanup of the number of days binlog

maxallowedpacket MySQL can control the size of the received packet

skipnameresolve Disable DNS lookup

read_only prohibit non-super user permission write permission

skipslavestart level you use slave automatic recovery

7.4 Other

Maximum number of connections allowed by the control max_connections

tmptablesize temporary table size

maxheaptable_size maximum memory size table

I did not use these parameters to the MySQL server tuning, performance results and introduce the specific details, please refer to the information at the end of the article or otherwise Baidu.

Eight, hardware purchase and parameter optimization

Performance of the hardware directly determine the performance of the MySQL database. Hardware performance bottlenecks, directly determines the efficiency of operating data and MySQL database.

As a software developer programmer, we focus on content optimization software to optimize the hardware as you can understand

8.1 memory-related

IO memory faster than a lot of the hard drive, you can increase the buffer capacity of the system, the data stay longer in memory to reduce disk IO

8.2 Disk I / O-related

1) using SSD or SSD PCle device, obtain at least several hundred or even thousands of times lifting IOPS

2) simultaneously with the acquisition card array and CACHE BBU module, can significantly improve IOPS

3) as far as possible selection of RAID-10, RAID-5 instead

8.3 Configuration related CUP

In the BIOS setup server, adjusted as follows:

1) Select the Performance Per Watt Optimized (DAPC) mode, play the maximum CPU performance

2) Close C1E C States and other options, CPU enhance efficiency

3) Memory Frequency (Frequency Memory) selected Maximum Performance