1. Optimization steps of database server
When we encounter database tuning problems, how should we think? Here the thinking process is organized into the picture below.
The whole process is divided into 观察（Show status）two 行动（Action）parts. The part of the letter S represents observation (the corresponding analysis tools will be used), and the part of the letter A represents action (corresponding to the actions that can be taken by the analysis).
We can understand the overall running status of the database through observation. Performance analysis tools can let us know what SQL is executing slowly, check the specific SQL execution plan, and even the cost of each step in SQL execution, so that we can locate the problem. Find the problem, and then take appropriate action.
Explain this picture in detail:
First, in the S1 part, we need to observe whether there are periodic fluctuations in the status of the server. If so 存在周期性波动, it may be due to cyclical nodes, such as Double Eleven, promotions, etc. In this case, we can solve it through step A1, that is, add cache, or change the cache invalidation policy.
If the caching strategy is not resolved, or is not the cause of the periodic fluctuations, we need to go further 分析查询延迟和卡顿的原因. Next, we need to enter
S2 开启慢查询. Slow query can help us locate SQL statements that execute slowly. We can long_query_timedefine the "slow" threshold by setting parameters. If the SQL execution time exceeds long_query_time, it will be considered a slow query. After collecting these slow queries, we can analyze the slow query logs through analysis tools.
In the S3 step, we know the SQL that executes slowly, so that we can EXPLAINview the execution plan of the corresponding SQL statement in a targeted manner, or view show profilethe time cost of each step in the SQL. In this way, we can understand whether the SQL query is slow because of long execution time or long waiting time.
If the SQL waiting time is long, we enter step A2. In this step, we can 调优服务器的参数, for example, increase the database buffer pool appropriately. If the SQL execution takes a long time, go to step A3. In this step, we need to consider whether it is a problem with the index design? Or are there too many data tables associated with the query? Or is this phenomenon caused by field design problems in the data tables. Then make corresponding adjustments in these dimensions.
If neither A2 nor A3 can solve the problem, we need to consider whether the SQL query performance of the database itself has reached the bottleneck. If it is confirmed that the performance bottleneck has not been reached, we need to recheck and repeat the above steps. If you have reached 性能瓶颈the A4 stage, you need to consider 增加服务器adopting a read-write separation architecture, or consider database sharding 分库分表, such as vertical sharding, vertical sharding, and horizontal sharding.
The above is the process idea of database tuning. If we find that there are irregular delays or freezes when executing SQL, we can use analysis tools to help us locate the problematic SQL. You can understand these three analysis tools as the three steps of SQL tuning: 慢查询、 EXPLAINand SHOWPROFILING.
summary:
2. View system performance parameters
In MySQL, you can use statements SHOW STATUSto query some MySQL database servers 性能参数.执行频率
The syntax of the SHOW STATUS statement is as follows:
SHOW [GLOBAL|SESSION] STATUS LIKE'参数';
Some commonly used performance parameters are as follows:
-
Connections: The number of times to connect to the MySQL server.
-
Uptime: The online time of the MySQL server.
-
Slow_queries: The number of slow queries.
- Default is more than ten seconds
-
Innodb_rows_read: Number of rows returned by Select query
-
Innodb_rows_inserted: Number of rows inserted by performing INSERT operation
-
Innodb_rows_updated: Number of rows updated by UPDATE operation
-
Innodb_rows_deleted: Number of rows deleted by DELETE operation
-
Com_select: The number of query operations.
-
Com_insert: The number of insert operations. For batch insert INSERT operations, only one accumulation is performed.
-
Com_update: The number of update operations.
-
Com_delete: The number of delete operations.
For example:
# 慢查询次数
show status like 'Slow_queries';
表结构优化The slow query count parameter can be combined with the slow query log to find out the slow query statement, and then perform OR on the slow query statement 查询语句优化. For another example, the following instructions can be used to view related instructions:
show status like 'Innodb_rows_%';
3. Statistics of SQL query cost: last_query_cost
A SQL query statement needs to determine the query execution plan before executing it. If there are multiple execution plans, MySQL will calculate the cost of each execution plan and select one of them as the final execution 成本最小plan.
last_query_costIf we want to check the query cost of a certain SQL statement, we can get the cost of the current query by checking the variable values in the current session after executing the SQL statement . It's often 评价一个查询的执行效率a common metric for us as well. This query cost corresponds to SQL语句所需要读取的页的数量.
We still use the student_info table in Chapter 8 as an example:
CREATE TABLE `student_info` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`student_id` INT NOT NULL ,
`name` VARCHAR(20) DEFAULT NULL, `course_id` INT NOT NULL ,
`class_id` INT(11) DEFAULT NULL,
`create_time` DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`)
) ENGINE=INNODB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8;
If we want to query the record with id=900001, and then look at the query cost, we can directly search on the clustered index:
SELECT student_id, class_id, NAME, create_time FROM student_info WHERE id = 900001 ;
Running results (1 record, running time 0.042s)
Then look at the cost of the query optimizer. In fact, we only need to retrieve one page:
mysql> SHOW STATUS LIKE 'last_query_cost';
+-----------------+----------+
| Variable_name | Value |
+-----------------+----------+
| Last_query_cost | 1.000000 |
+-----------------+----------+
What if we want to query student records with IDs between 900001 and 9000100?
SELECT student_id, class_id, NAME, create_time FROM student_info
WHERE id BETWEEN 900001 AND 900100;
Running results (100 records, running time 0.046s)
Then look at the cost of the query optimizer. At this time we need to query about 20 pages.
mysql> SHOW STATUS LIKE 'last_query_cost';
+-----------------+-----------+
| Variable_name | Value |
+-----------------+-----------+
| Last_query_cost | 21.134453 |
+-----------------+-----------+
You can see that the number of pages is 20 times that of before, but the efficiency of the query has not changed significantly. In fact, the time of the two SQL queries is basically the same, because the page is loaded in one go using sequential reading. into the buffer pool and then search again. Although 页数量（last_query_cost）增加了不少, through the buffer pool mechanism, 并没有增加多少查询时间.
**Usage scenarios:** It is very useful for comparing costs, especially when we have several query methods to choose from.
SQL query is a dynamic process. From the perspective of page loading, we can draw the following two conclusions:
位置决定效率. If the page is in the database缓冲池, then the efficiency is the highest, otherwise it needs磁盘to be
read from it. Of course, for reading a single page, if the page exists in the memory, it will be much more efficient than reading from the disk.
批量决定效率. If we randomly read a single page from the disk, the efficiency is very low (almost 10ms). If we use
sequential reading to read pages in batches, the average reading efficiency of one page will be improved a lot. , even faster than a random read of a single page in memory.Therefore, there is no need to worry when encountering /O. If you find the right method, the efficiency is still very high. We must first consider the location of data storage. If it is frequently used data, we should try to put it in the middle
缓冲池. Secondly, we can make full use of the throughput capacity of the disk and read the data in batches at one time, so that the reading efficiency of a single page is also improved. promote.
4. Locate SQL that executes slowly: slow query log
MySQL's slow query log is used to record 响应时间超过阀值statements in MySQL. Specifically, long_query_timeSQL whose running time exceeds the value will be recorded in the slow query log. The default value of long_query_time is 10, which means that statements that run for more than 10 seconds (excluding 10 seconds) are considered to exceed our maximum tolerance time value. |
Its main function is to help us discover those SQL queries that take particularly long execution times and optimize them in a targeted manner, thereby improving the overall efficiency of the system. When our database server is blocked and runs slowly, checking the slow query log and finding those slow queries is very helpful in solving the problem. For example, if a sq| executes for more than 5 seconds, we will consider it a slow SQL. We hope to collect the sql that exceeds 5 seconds and conduct a comprehensive analysis based on the explain.
By default, the MySQL database 没有开启慢查询日志requires us to set this parameter manually. If it is not required for tuning, it is generally not recommended to enable this parameter , because enabling slow query logs will have a more or less certain performance impact.
The slow query log supports writing log records to a file.
4.1 Enable slow query log parameters
1. Turn on slow_query_log
mysql > show variables like '%slow_query_log%';
mysql > set global slow_query_log='ON';
Then let's check whether the slow query log is turned on and the location of the slow query log file:
You can see that the slow query analysis has been turned on and the file is saved in /var/lib/mysql/atguigu02-slow.logFile
.
2. Modify long_query_time threshold
Next, let’s look at the time threshold setting for slow queries, using the following command:
mysql > show variables like '%long_query_time%';
# 测试发现：设置global的方式对当前session的long_query_time失效。对新连接的客户端有效。所以可以一并执行下述语句
mysql> set global long_query_time = 1 ;
mysql> show global variables like ' %long-query_time% ';
# 即更改global 也更改了session变量
mysql> set long_query_time=1;
mysql> show variables like '%long_query_time%';
Supplement: Set parameters in the configuration file together
Compared with the previous command line method, the following method can be regarded as a permanent setting method.
Modify my.cnfthe file, [mysqld]下add or modify parameters long_query_time, slow_query_logand slow_query_log_filethen restart the MySQL server. |
[mysqld]
slow_query_log=ON #开启慢查询日志的开关
slow_query_log_file=/var/lib/mysql/my-slow.log #慢查询日志的目录和文件名信息
long_query_time=3 #设置慢查询的阈值为3秒，超出此设定值的SQL即被记录到慢查询日志
log_output=FILE
If you do not specify a storage path, the slow query log will be stored in the data folder of the MySQL database by default. If you do not specify a file name, the default file name is hostname-slow.log.
4.2 Check the number of slow queries
Query how many slow query records there are in the current system
SHOW GLOBAL STATUS LIKE '%Slow_queries%';
4.3 Case demonstration
Step 1. Create table
CREATE TABLE `student` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`stuno` INT NOT NULL ,
`name` VARCHAR(20) DEFAULT NULL,
`age` INT(3) DEFAULT NULL,
`classId` INT(11) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=INNODB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8;
Step 2: Set parameter log_bin_trust_function_creators
Create a function, if an error is reported:
This function has none of DETERMINISTIC......
Command on: Allow creation of function settings:
set global log_bin_trust_function_creators=1; # 不加global只是当前窗口有效。
Step 3: Create the function
Randomly generate strings: (same as previous chapter)
DELIMITER //
CREATE FUNCTION rand_string(n INT)
RETURNS VARCHAR(255) #该函数会返回一个字符串
BEGIN
DECLARE chars_str VARCHAR(100) DEFAULT
'abcdefghijklmnopqrstuvwxyzABCDEFJHIJKLMNOPQRSTUVWXYZ';
DECLARE return_str VARCHAR(255) DEFAULT '';
DECLARE i INT DEFAULT 0;
WHILE i < n DO
SET return_str =CONCAT(return_str,SUBSTRING(chars_str,FLOOR(1+RAND()*52),1));
SET i = i + 1;
END WHILE;
RETURN return_str;
END //
DELIMITER ;
Generate random values: (same as previous chapter)
#测试
SELECT rand_string(10);
DELIMITER //
CREATE FUNCTION rand_num (from_num INT ,to_num INT) RETURNS INT(11)
BEGIN
DECLARE i INT DEFAULT 0;
SET i = FLOOR(from_num +RAND()*(to_num - from_num+1)) ;
RETURN i;
END //
DELIMITER ;
#测试：
SELECT rand_num(10,100);
Step 4: Create stored procedure
DELIMITER //
CREATE PROCEDURE insert_stu1( START INT , max_num INT )
BEGIN
DECLARE i INT DEFAULT 0;
SET autocommit = 0; #设置手动提交事务
REPEAT #循环
SET i = i + 1; #赋值
INSERT INTO student (stuno, NAME ,age ,classId ) VALUES
((START+i),rand_string(6),rand_num(10,100),rand_num(10,1000));
UNTIL i = max_num
END REPEAT;
COMMIT; #提交事务
END //
DELIMITER ;
Step 5: Call the stored procedure
#调用刚刚写好的函数, 4000000条记录,从100001号开始
CALL insert_stu1(100001,4000000);
4.4 Testing and Analysis
1. Test
mysql> SELECT * FROM student WHERE stuno = 3455655;
+---------+---------+--------+------+---------+
| id | stuno | name | age | classId |
+---------+---------+--------+------+---------+
| 3523633 | 3455655 | oQmLUr | 19 | 39 |
+---------+---------+--------+------+---------+
1 row in set (2.09 sec)
mysql> SELECT * FROM student WHERE name = 'oQmLUr';
+---------+---------+--------+------+---------+
| id | stuno | name | age | classId |
+---------+---------+--------+------+---------+
| 1154002 | 1243200 | OQMlUR | 266 | 28 |
| 1405708 | 1437740 | OQMlUR | 245 | 439 |
| 1748070 | 1680092 | OQMlUR | 240 | 414 |
| 2119892 | 2051914 | oQmLUr | 17 | 32 |
| 2893154 | 2825176 | OQMlUR | 245 | 435 |
| 3523633 | 3455655 | oQmLUr | 19 | 39 |
+---------+---------+--------+------+---------+
6 rows in set (2.39 sec)
As can be seen from the above results, it takes 2.09 seconds to query the student information with the student number "3455655". It takes 2.39 seconds to query the student information with the student name "oQmLUr". It has reached the order of seconds, indicating that the current query efficiency is relatively low.
We will analyze the reasons in the following section.
2.Analysis
show status like 'slow_queries';
Additional information:
In addition to the above variables, there is also a system variable that controls the slow query log: min_examined_row_limit. The meaning of this variable is, query
扫描过的最少记录数. This variable and the query execution time together form the condition for determining whether a query is a slow query. If the number of records scanned by the query is greater than or equal to the value of this variable, and the query execution time exceeds the value of long_query_time, then the query will be recorded in the slow query log; otherwise, it will not be recorded in the slow query log.mysql> show variables like 'min%'; +------------------------+-------+ | Variable_name | Value | +------------------------+-------+ | min_examined_row_limit | 0 | +------------------------+-------+ 1 row in set (0.00 sec)You can also modify the value of "min_examined_row_limit" by modifying the "my.ini" file as needed.
4.5 Slow query log analysis tool: mysqldumpslow
In a production environment, if you want to manually analyze logs, find and analyze SQL, it is obviously a laborious task. MySQL provides log analysis tools mysqldumpslow.
View the help information of mysqldumpslow
mysqldumpslow --help
The specific parameters of the mysqldumpslow command are as follows:
- -a: Do not abstract numbers into N and strings into S
- -s: indicates how to sort:
- c: number of visits
- l: lock time
- r: return record
- t: query time
- al: average lock time
- ar: average number of records returned
- at: average query time (default mode)
- ac: average number of queries
- -t: Returns the previous number of data;
- -g: followed by a regular matching pattern, case-insensitive
Example: We want to sort by query time and view the first five SQL statements, so we can write:
mysqldumpslow -s t -t 5 /var/lib/mysql/atguigu01-slow.log
[root@bogon ~]# mysqldumpslow -s t -t 5 /var/lib/mysql/atguigu01-slow.log
Reading mysql slow query log from /var/lib/mysql/atguigu01-slow.log
Count: 1 Time=2.39s (2s) Lock=0.00s (0s) Rows=13.0 (13), root[root]@localhost
SELECT * FROM student WHERE name = 'S'
Count: 1 Time=2.09s (2s) Lock=0.00s (0s) Rows=2.0 (2), root[root]@localhost
SELECT * FROM student WHERE stuno = N
Died at /usr/bin/mysqldumpslow line 162, <> chunk 2.
Common references for work:
#得到返回记录集最多的10个SQL
mysqldumpslow -s r -t 10 /var/lib/mysql/atguigu-slow.log
#得到访问次数最多的10个SQL
mysqldumpslow -s c -t 10 /var/lib/mysql/atguigu-slow.log
#得到按照时间排序的前10条里面含有左连接的查询语句
mysqldumpslow -s t -t 10 -g "left join" /var/lib/mysql/atguigu-slow.log
#另外建议在使用这些命令时结合 | 和more 使用 ，否则有可能出现爆屏情况
mysqldumpslow -s r -t 10 /var/lib/mysql/atguigu-slow.log | more
4.6 Close slow query log
Except that it needs to be turned on for tuning, it is better not to turn it on normally.
There are two ways to stop the slow query log function of the MySQL server:
Method 1: Permanent method
[mysqld]
slow_query_log=OFF
Or, comment out or delete the slow_query_log item
mysqld]
#slow_query_log =OFF
Restart the MySQL service and execute the following statement to query the slow log function.
SHOW VARIABLES LIKE '%slow%'; #查询慢查询日志所在目录
SHOW VARIABLES LIKE '%long_query_time%'; #查询超时时长
Method 2: Temporary method
Use the SET statement to set. (1) Stop the MySQL slow query log function. The specific SQL statements are as follows.
SET GLOBAL slow_query_log=off;
(2) Restart the MySQL service and use the SHOW statement to query the slow query log function information. The specific SQL statement is as follows
SHOW VARIABLES LIKE '%slow%';
#以及
SHOW VARIABLES LIKE '%long_query_time%';
4.7 Delete slow query logs
mysql> show variables like '%slow_query_log%';
+---------------------+----------------------------+
| Variable_name | Value |
+---------------------+----------------------------+
| slow_query_log | ON |
| slow_query_log_file | /var/lib/mysql/my-slow.log |
+---------------------+----------------------------+
2 rows in set (0.07 se
It can be seen from the execution results that the directory of the slow query log defaults to the MySQL data directory, and it can be in this directory 手动删除慢查询日志文件. Use the command mysqladmin flush-logsto regenerate the query log file. The specific command is as follows. After execution, the slow query log file will be regenerated in the data directory.
# 不使用这个命令，没办法自己创建
mysqladmin -uroot -p flush-logs slow
## 这个命令可以重置其他日志 例如undo日志
hint
Slow query logs are deleted and rebuilt using the mysqladmin flush-logs command. When using it, be sure to note that once this command is executed, the slow
query log will only exist in the new log file. If the old query log is needed, it must be backed up in advance.
5. View SQL execution cost: SHOW PROFILE
Show profile is discussed in the "Logical Architecture" chapter, and is here as a review.
Show Profile is a tool provided by MySQL that can be used to analyze what SQL has done in the current session and the resource consumption of execution. It can be used for SQL tuning measurement. 默认情况下处于关闭状态, and save the latest 15 running results.
We can enable this feature at the session level
mysql> show variables like 'profiling';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| profiling | OFF |
+---------------+-------+
1 row in set (0.34 sec)
profiling='ON’Turn on show profile by setting :
mysql> set profiling = 'ON';
Query OK, 0 rows affected, 1 warning (0.06 sec)
mysql> show variables like 'profiling';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| profiling | ON |
+---------------+-------+
1 row in set (0.13 sec)
Then execute the relevant query statements. Then to see what profiles the current session has, use the following command:
mysql> show profiles;
+----------+------------+-------------------------------------+
| Query_ID | Duration | Query |
+----------+------------+-------------------------------------+
| 1 | 0.13515975 | show variables like 'profiling' |
| 2 | 0.06386950 | select * from student_info limit 10 |
+----------+------------+-------------------------------------+
2 rows in set, 1 warning (0.01 sec)
You can see that there are a total of 2 queries in the current session. If we want to see the cost of the most recent query, we can use:
mysql> show profile;
+--------------------------------+----------+
| Status | Duration |
+--------------------------------+----------+
| starting | 0.029330 |
| Executing hook on transaction | 0.001174 |
| starting | 0.002804 |
| checking permissions | 0.002918 |
| Opening tables | 0.009026 |
| init | 0.001605 |
| System lock | 0.000503 |
| optimizing | 0.000013 |
| statistics | 0.007651 |
| preparing | 0.000084 |
| executing | 0.005307 |
| end | 0.000017 |
| query end | 0.000178 |
| waiting for handler commit | 0.000028 |
| closing tables | 0.001087 |
| freeing items | 0.000399 |
| cleaning up | 0.001748 |
+--------------------------------+----------+
17 rows in set, 1 warning (0.04 sec)
We can also check the cost of the specified Query ID, for example, show profile for query 2the query results are the same. In SHOWPROFILE we can view the overhead of different parts, such as cpu, block.io, etc.:
mysql> show profile cpu,block io for query 2;
+------------------------------+--------+----------+------------+-------------+
| Status |Duration| CPU_user |Block_ops_in|Block_ops_out|
+------------------------------+--------+----------+------------+-------------+
| starting |0.029330| 0.017180 | 49712| 0|
| Executing hook on transaction|0.001174| 0.001079 | 3624| 0|
| starting |0.002804| 0.002169 | 4728| 0|
| checking permissions |0.002918| 0.002437 | 8168| 0|
| Opening tables |0.009026| 0.005841 | 14120| 0|
| init |0.001605| 0.000392 | 80| 0|
| System lock |0.000503| 0.000130 | 24| 0|
| optimizing |0.000013| 0.000010 | 0| 0|
| statistics |0.007651| 0.003072 | 4160| 0|
| preparing |0.000084| 0.000071 | 0| 0|
| executing |0.005307| 0.001609 | 568| 0|
| end |0.000017| 0.000011 | 0| 0|
| query end |0.000178| 0.000047 | 8| 0|
| waiting for handler commit |0.000028| 0.000025 | 0| 0|
| closing tables |0.001087| 0.000279 | 56| 0|
| freeing items |0.000399| 0.000259 | 8| 0|
| cleaning up |0.001748| 0.000381 | 56| 0|
+------------------------------+--------+----------+------------+-------------+
17 rows in set, 1 warning (0.01 sec)
If the execution is relatively long, it may be that the code is not written well. Use explain to continue to query the problem.
Common query parameters for show profile:
① ALL: Display all overhead information.
② BLOCK IO: Displays block IO overhead.
③ CONTEXT SWITCHES: Context switching overhead.
④ CPU: Displays CPU overhead information.
⑤ IPC: Displays sending and receiving overhead information.
⑥ MEMORY: Display memory overhead information.
⑦ PAGE FAULTS: Display page error overhead information.
⑧ SOURCE: Displays overhead information related to Source_function, Source_file, Source_line.
⑨ SWAPS: Displays the cost information of the number of swaps.
Conclusions that need to be paid attention to in daily development:
converting HEAP to MyISAM: The query result is too large, the memory is not enough, and the data has been moved to the disk.creating tmp table:Create a temporary table. Copy the data to the temporary table first, and then delete the temporary table after use.Copying to tmp table on disk:Copy the temporary table in memory to disk, be careful!locked.
If any of the above four results appear in the show profile diagnostic results, the SQL statement needs to be optimized.
Notice:
However, the SHOW PROFILE command will be deprecated, and we can view it from the profiling data table in information_schema.
6. Analyze the query statement: EXPLAIN
6.1 Overview
**After locating the slow query SQL, we can use the EXPLAIN or DESCRIBE tool to analyze the query statement in a targeted manner. **The usage of the DESCRIBE statement is the same as the EXPLAIN statement, and the analysis results are also the same.
执行计划There is an optimizer module in MySQL that is specifically responsible for optimizing SELECT statements. Its main function is to provide the Query requested by the client with the data retrieval method it thinks is optimal (he thinks it is the optimal data retrieval method) by calculating and analyzing the statistical information collected in the system. It may not be what the DBA thinks is optimal, this part is the most time-consuming).
This execution plan shows how to specifically execute the query, such as what is the order of multi-table connections, what access method is used for each table to specifically execute the query, etc. MySQL provides us with EXPLAINstatements to help us view the specific execution plan of a certain query statement. Understanding the EXPLAINvarious output items of the statement can improve the performance of our query statements in a targeted manner.
1.What can be done?
- Table reading order
- The operation type of the data read operation.
- Which indexes can be used
- Which indexes are actually used
- References between tables
- How many rows in each table are queried by the optimizer
Official website introduction
https://dev.mysql.com/doc/refman/5.7/en/explain-output.html
https://dev.mysql.com/doc/refman/8.0/en/explain-output.html
Version status
- Before MySQL 5.6.3, only EXPLAIN SELECT could be used; after MYSQL 5.6.3, it can be used
EXPLAINSELECT.UPDATEDELETE - In versions before 5.7, you
partitionsneed to useexplain partitionsthe command if you want to display it; you needfilteredto useexplain extendedthe command if you want to display it. After version 5.7, explain directly displays the information in partitions and filtered by default.
6.2 Basic syntax
The syntax of the EXPLAIN or DESCRIBE statement is as follows:
EXPLAIN SELECT select_options
# 或者 两个是一样的
DESCRIBE SELECT select_options
If we want to see the execution plan of a certain query, we can add one in front of the specific query statement EXPLAIN, like this:
mysql> EXPLAIN SELECT 1;
EXPLAINThe functions of each column output by the statement are as follows:
| List | describe |
|---|---|
| id | In a large query statement, each SELECT keyword corresponds to a unique id. |
| select_type | The type of query corresponding to the SELECT keyword |
| table | Table Name |
| partitions | Matching partition information |
| type | Access method for single table (important) |
| possible_keys | Possible indexes |
| key | actual index used |
| key_len | The actual index length used |
| ref | When using the index column equivalence query, the object information that matches the index column for equivalence. |
| rows | Estimated number of records to be read |
| filtered | Percentage of remaining records in a table after filtering by search conditions |
| Extra | some additional information |
6.3 Data preparation
1. Create a table
CREATE TABLE s1 (
id INT AUTO_INCREMENT,
key1 VARCHAR(100),
key2 INT,
key3 VARCHAR(100),
key_part1 VARCHAR(100),
key_part2 VARCHAR(100),
key_part3 VARCHAR(100),
common_field VARCHAR(100),
PRIMARY KEY (id),
INDEX idx_key1 (key1),
UNIQUE INDEX idx_key2 (key2),
INDEX idx_key3 (key3),
INDEX idx_key_part(key_part1, key_part2, key_part3)
) ENGINE=INNODB CHARSET=utf8;
CREATE TABLE s2 (
id INT AUTO_INCREMENT,
key1 VARCHAR(100),
key2 INT,
key3 VARCHAR(100),
key_part1 VARCHAR(100),
key_part2 VARCHAR(100),
key_part3 VARCHAR(100),
common_field VARCHAR(100),
PRIMARY KEY (id),
INDEX idx_key1 (key1),
UNIQUE INDEX idx_key2 (key2),
INDEX idx_key3 (key3),
INDEX idx_key_part(key_part1, key_part2, key_part3)
) ENGINE=INNODB CHARSET=utf8;
2. Set the parameter log_bin_trust_function_creators
Create a function. If an error is reported, you need to enable the following command: Allow creation of function settings:
set global log_bin_trust_function_creators=1; # 不加global只是当前窗口有效。
3. Create functions
DELIMITER //
CREATE FUNCTION rand_string1 ( n INT )
RETURNS VARCHAR ( 255 ) #该函数会返回一个字符串
BEGIN
DECLARE
chars_str VARCHAR ( 100 ) DEFAULT 'abcdefghijklmnopqrstuvwxyzABCDEFJHIJKLMNOPQRSTUVWXYZ';
DECLARE
return_str VARCHAR ( 255 ) DEFAULT '';
DECLARE
i INT DEFAULT 0;
WHILE
i < n DO
SET return_str = CONCAT(
return_str,
SUBSTRING( chars_str, FLOOR( 1+RAND ()* 52 ), 1 ));
SET i = i + 1;
END WHILE;
RETURN return_str;
END //
DELIMITER ;
4. Create a stored procedure
Create a stored procedure to insert data into the s1 table:
DELIMITER //
CREATE PROCEDURE insert_s1 (IN min_num INT (10),IN max_num INT (10))
BEGIN
DECLARE i INT DEFAULT 0;
SET autocommit = 0;
REPEAT
SET i = i + 1;
INSERT INTO s1 VALUES(
(min_num + i),
rand_string1(6),
(min_num + 30 * i + 5),
rand_string1(6),
rand_string1(10),
rand_string1(5),
rand_string1(10),
rand_string1(10));
UNTIL i = max_num
END REPEAT;
COMMIT;
END //
DELIMITER ;
Create a stored procedure to insert data into the s2 table:
DELIMITER //
CREATE PROCEDURE insert_s2 (IN min_num INT ( 10 ),IN max_num INT ( 10 ))
BEGIN
DECLARE i INT DEFAULT 0;
SET autocommit = 0;
REPEAT
SET i = i + 1;
INSERT INTO s2 VALUES(
( min_num + i ),
rand_string1 ( 6 ),
( min_num + 30 * i + 5 ),
rand_string1 ( 6 ),
rand_string1 ( 10 ),
rand_string1 ( 5 ),
rand_string1 ( 10 ),
rand_string1 ( 10 ));
UNTIL i = max_num
END REPEAT;
COMMIT;
END //
DELIMITER ;
5. Call the stored procedure
Adding s1 table data: adding 10,000 records:
CALL insert_s1(10001,10000); # id 10002~20001
Adding s2 table data: adding 10,000 records:
CALL insert_s2(10001,10000);# id 10002~20001
6.4 The functions of each EXPLAIN column
In order to give everyone a better experience, we have adjusted EXPLAINthe order of the output columns.
1. table
Table Name
No matter how complex our query statement is, 包含了多少个表it needs to be performed on each table in the end 单表访问. Therefore,
MySQL stipulates that each record output by the EXPLAIN statement corresponds to the access method of a single table . The table of this record The column represents the
table name of the table (sometimes not the real table name, it may be abbreviated)
#1. table：表名
#查询的每一行记录都对应着一个单表
explain select count(*) from s1;
#s1:驱动表 s2:被驱动表
EXPLAIN SELECT * FROM s1 INNER JOIN s2;
# 驱动表和被驱动表是 优化器决定的，他认为哪个比较好久用哪个
How many tables are used, how many records there will be
2. id
Normally, there is one select and one ID. There are exceptions. The query optimizer has optimized it.
mysql> EXPLAIN SELECT * FROM s1 WHERE key1 = 'a';
mysql> EXPLAIN SELECT * FROM s1 INNER JOIN s2;
mysql> EXPLAIN SELECT * FROM s1 WHERE key1 IN (SELECT key1 FROM s2) OR key3 = 'a';
Query optimizer optimization
######查询优化器可能对涉及子查询的查询语句进行重写,转变为多表查询的操作########
EXPLAIN SELECT * FROM s1 WHERE key1 IN (SELECT key2 FROM s2 WHERE common_field = 'a');
Running result: There is only one id because the query optimizer has optimized it.
Union deduplication
I originally thought of one select and one ID, but I expected two.
#Union去重
# union 去重，union all 不去重
EXPLAIN SELECT * FROM s1 UNION SELECT * FROM s2;
# union all 不去重 所以不需要放在临时表里面
mysql> EXPLAIN SELECT * FROM s1 UNION ALL SELECT * FROM s2;
summary:
- If the IDs are the same, they can be considered as a group and executed sequentially from top to bottom.
- In all groups, the larger the id value is, the higher the priority is and the execution is performed first.
- Point of concern: Each ID number represents an independent query. The fewer the number of queries in a SQL statement, the better.
3. select_type
A large query statement can contain several SELECT keywords, 每个SELECT关键字代表着一个小的查询语句and the FROM clause of each SELECT keyword can contain several tables (these tables are used for connection queries). 每一张表都对应着执行计划输出中的一条记录For the same SELECT keyword, For the tables in , their id values are the same.
MySQL defines an attribute called for each small query represented by the SELECT keyword select_type, which means that as long as we know the value of a certain small query select_type属性, we know this 小查询在整个大查询中扮演了一个什么角色. Let's see
select_typewhat values can take. Please see the official documentation :
| name | describe |
|---|---|
| SIMPLE | Simple SELECT (not using UNION or subqueries) |
| PRIMARY | Outermost SELECT |
| UNION | Second or later SELECT statement in a UNION |
| UNION RESULT | Result of a UNION |
| SUBQUERY | First SELECT in subquery |
| DEPENDENT SUBQUERY | First SELECT in subquery, dependent on outer query |
| DEPENDENT UNION | Second or later SELECT statement in a UNION, dependent on outer query |
| DERIVED | Derived table |
| MATERIALIZED | Materialized subquery |
| UNCACHEABLE SUBQUERY | A subquery for which the result cannot be cached and must be re-evaluated for each row of the outer query |
| UNCACHEABLE UNION | The second or later select in a UNION that belongs to an uncacheable subquery (see UNCACHEABLE SUBQUERY) |
-
SIMPLE# 查询语句中不包含`UNION`或者子查询的查询都算作是`SIMPLE`类型 EXPLAIN SELECT * FROM s1; #连接查询也算是`SIMPLE`类型 EXPLAIN SELECT * FROM s1 INNER JOIN s2; -
PRIMARYandUNIONwithUNION RESULT-
UNION RESULTMySQL chooses to use a temporary table to complete
UNIONthe deduplication of queries. The query for this temporary tableselect_typeis asUNION RESULTshown in the example above.
#对于包含`UNION`或者`UNION ALL`或者子查询的大查询来说，它是由几个小查询组成的，其中最左边的那个 #查询的`select_type`值就是`PRIMARY` #对于包含`UNION`或者`UNION ALL`的大查询来说，它是由几个小查询组成的，其中除了最左边的那个小查询 #以外，其余的小查询的`select_type`值就是`UNION` #`MySQL`选择使用临时表来完成`UNION`查询的去重工作，针对该临时表的查询的`select_type`就是`UNION RESULT`Test sql:
EXPLAIN SELECT * FROM s1 UNION SELECT * FROM s2; -
EXPLAIN SELECT * FROM s1 UNION ALL SELECT * FROM s2;
T关键字代表的那个查询的select_type就是SUBQUERY` EXPLAIN SELECT * FROM s1 WHERE key1 IN (SELECT key1 FROM s2) OR key3 = ‘a’;
DEPENDENT SUBQUERY
dependent subquery
#如果包含子查询的查询语句不能够转为对应的`semi-join`的形式，并且该子查询是相关子查询，
#则该子查询的第一个`SELECT`关键字代表的那个查询的`select_type`就是`DEPENDENT SUBQUERY`
EXPLAIN SELECT * FROM s1
WHERE key1 IN (SELECT key1 FROM s2 WHERE s1.key2 = s2.key2) OR key3 = 'a';
#注意的是，select_type为`DEPENDENT SUBQUERY`的查询可能会被执行多次。
DEPENDENT UNION
#在包含`UNION`或者`UNION ALL`的大查询中，如果各个小查询都依赖于外层查询的话，那除了
#最左边的那个小查询之外，其余的小查询的`select_type`的值就是`DEPENDENT UNION`。
EXPLAIN SELECT * FROM s1
WHERE key1 IN (SELECT key1 FROM s2 WHERE key1 = 'a' UNION SELECT key1 FROM s1 WHERE key1 = 'b');
# 这里优化器会重构成exist
DERIVED
derived: derived, derived
#对于包含`派生表`的查询，该派生表对应的子查询的`select_type`就是`DERIVED`
EXPLAIN SELECT *
FROM (SELECT key1, COUNT(*) AS c FROM s1 GROUP BY key1) AS derived_s1 WHERE c > 1;
MATERIALIZED
materialized: English [məˈtɪəri:əˌlaɪzd] materialized
#当查询优化器在执行包含子查询的语句时，选择将子查询物化之后与外层查询进行连接查询时，
#该子查询对应的`select_type`属性就是`MATERIALIZED`
EXPLAIN SELECT * FROM s1 WHERE key1 IN (SELECT key1 FROM s2); #子查询被转为了物化表
-
Don’t understand: Why is there no atomization in the above subquery?
EXPLAIN SELECT * FROM s1 WHERE key1 IN (SELECT key1 FROM s2) OR key3 = 'a'; # 这个怎么不物化 -
UNCACHEABLE SUBQUERYuncacheable
-
UNCACHEABLE UNION
4. partitions (optional)
- Represents the hit situation in the partitioned table. For non-partitioned tables, this item is NULL. Under normal circumstances, the value of the partitions column in the execution plan of our query statement is NULL.
- https://dev.mysql.com/doc/refman/5.7/en/alter-table-partition-operations.html
- If you want to know more about it, you can test it as follows. Create partition table:
-- 创建分区表，
-- 按照id分区，id<100 p0分区，其他p1分区
CREATE TABLE user_partitions (
id INT auto_increment,
NAME VARCHAR(12),PRIMARY KEY(id))
PARTITION BY RANGE(id)(
PARTITION p0 VALUES less than(100),
PARTITION p1 VALUES less than MAXVALUE
);
DESC SELECT * FROM user_partitions WHERE id>200;
Query the records with ID greater than 200 (200>100, p1 partition) and view the execution plan. The partitions are p1, which conforms to our partition rules.
5. type ☆
A record in the execution plan represents MySQL's access to a certain table 执行查询时的访问方法, also known as "access type". The columns in it typeindicate what the access method is, which is a more important indicator. For example, seeing typea column value of ref, indicates that MySQL is about to use refthe access method to execute s1a query on the table.
The complete access method is as follows: system, const, eq_ref, ref, fulltext, ref_or_null, index_merge, unique_subquery, index_subquery, range, index, ALL.
Let’s explain it in detail:
-
systemWhen
只有一条记录the statistics in the table and the storage engine used by the table are accurate, such as MyISAM and Memory, then the access method to the table issystem. For example, let's create a newMyISAMtable and insert a record into it:mysql> CREATE TABLE t(i int) Engine=MyISAM; Query OK, 0 rows affected (0.05 sec) mysql> INSERT INTO t VALUES(1); Query OK, 1 row affected (0.01 sec)Then we look at the execution plan for querying this table:
mysql> EXPLAIN SELECT * FROM t; +----+-------------+-------+------------+--------+ | id | select_type | table | partitions | type | +----+-------------+-------+------------+--------+ | 1 | SIMPLE | t | NULL | system | +----+-------------+-------+------------+--------+ 1 row in set, 1 warning (0.00 sec)If it is innodb, it will become ALL because the innodb system does not store the number field. . MyISAM will store such a field
-
const#当我们根据主键或者唯一二级索引列与常数进行等值匹配时，对单表的访问方法就是`const` EXPLAIN SELECT * FROM s1 WHERE id = 10005; EXPLAIN SELECT * FROM s1 WHERE key2 = '10066'; -
eq_ref#在连接查询时，如果被驱动表是通过主键或者唯一二级索引列等值匹配的方式进行访问的 #（如果该主键或者唯一二级索引是联合索引的话，所有的索引列都必须进行等值比较），则 #对该被驱动表的访问方法就是`eq_ref` EXPLAIN SELECT * FROM s1 INNER JOIN s2 ON s1.id = s2.id;mysql> EXPLAIN SELECT * FROM s1 INNER JOIN s2 ON s1.id = s2.id; +----+------------+------+--------+---------+---------+------------------+------+ | id | select_type| table| type | key | key_len | ref | rows | +----+------------+------+--------+---------+---------+------------------+------+ | 1 | SIMPLE | s1 | ALL | NULL | NULL | NULL | 9895 | | 1 | SIMPLE | s2 | eq_ref | PRIMARY | 4 | atguigudb1.s1.id | 1 | +----+------------+------+--------+---------+---------+------------------+------+ 2 rows in set, 1 warning (0.00 sec)It can be seen from the results of the execution plan that MySQL intends to use s2 as the driving table and s1 as the driven table, focusing on the access
method of s1, indicating that it can be accessedeq_refwhen accessing the s1 table .通过主键的等值匹配 -
ref#当通过普通的二级索引列与常量进行等值匹配时来查询某个表，那么对该表的访问方法就可能是`ref` EXPLAIN SELECT * FROM s1 WHERE key1 = 'a';Results of the:
mysql> EXPLAIN SELECT * FROM s1 WHERE key1 = 'a'; +----+-------+------+---------------+----------+---------+ | id | table | type | possible_keys | key | key_len | +----+-------+------+---------------+----------+---------+ | 1 | s1 | ref | idx_key1 | idx_key1 | 303 | +----+-------+------+---------------+----------+---------+ 1 row in set, 1 warning (0.00 sec)Tips: Indexing can only be done if the types are the same.
EXPLAIN SELECT * FROM s1 WHERE key2 = 10066; # 这个是不会走索引的 因为key2 是字符串 # 类型不一样，mysql会加函数，进行隐式转换，一旦加上函数，就不会走索引了。 -
fulltextFull text index
-
ref_or_null#当对普通二级索引进行等值匹配查询，该索引列的值也可以是`NULL`值时，那么对该表的访问方法 #就可能是`ref_or_null` EXPLAIN SELECT * FROM s1 WHERE key1 = 'a' OR key1 IS NULL; -
index_merge#单表访问方法时在某些场景下可以使用`Intersection`、`Union`、 #`Sort-Union`这三种索引合并的方式来执行查询 EXPLAIN SELECT * FROM s1 WHERE key1 = 'a' OR key3 = 'a';result
mysql> EXPLAIN SELECT * FROM s1 WHERE key1 = 'a' OR key3 = 'a'; +---+-------+------------+-------------------+--------+-------------------------------+ | id| table | type | key | key_len| Extra +---+-------+------------+-------------------+--------+-------------------------------+ | 1| s1 | index_merge| idx_key1,idx_key3 | 303,303| Using union(idx_key1,idx_key3)| +---+-------+------------+-------------------+--------+-------------------------------+ 1 row in set, 1 warning (0.01 sec)typeIt can be seen from the value of the column in the execution planindex_mergethat MySQL intends to use index merging to execute queries
ons1the table. -
unique_subquery#`unique_subquery`是针对在一些包含`IN`子查询的查询语句中，如果查询优化器决定将`IN`子查询 #转换为`EXISTS`子查询，而且子查询可以使用到主键进行等值匹配的话，那么该子查询执行计划的`type` #列的值就是`unique_subquery` EXPLAIN SELECT * FROM s1 WHERE key2 IN (SELECT id FROM s2 WHERE s1.key1 = s2.key1) OR key3 = 'a'; -
index_subqueryEXPLAIN SELECT * FROM s1 WHERE common_field IN (SELECT key3 FROM s2 where s1.key1 = s2.key1) OR key3 = 'a'; -
range#如果使用索引获取某些`范围区间`的记录，那么就可能使用到`range`访问方法 EXPLAIN SELECT * FROM s1 WHERE key1 IN ('a', 'b', 'c'); #同上 EXPLAIN SELECT * FROM s1 WHERE key1 > 'a' AND key1 < 'b'; -
index#当我们可以使用索引覆盖，但需要扫描全部的索引记录时，该表的访问方法就是`index` EXPLAIN SELECT key_part2 FROM s1 WHERE key_part3 = 'a';Index coverage,
INDEX idx_key_part(key_part1, key_part2, key_part3）these 3 constitute a composite indexkey_part3 is in the composite index, and the queried field is also in the index. Simply traverse the index directly to find the data.
Thinking: The advantage is that the index stores less data, and the less data means fewer pages, which can reduce IO.
-
ALL
mysql> EXPLAIN SELECT * FROM s1;
Generally speaking, except for Allthis access method, all other access methods can use indexes. Except for index_mergethe access method, all other access methods can only use one index at most.
summary:
The resulting values from best to worst are:
system > const > eq_ref > ref >
fulltext > ref_or_null > index_merge >unique_subquery > index_subquery > range >
index > ALL
The goal of SQL performance optimization: at least reach the range level, the requirement is the ref level, preferably the consts level. (
Required by Alibaba Development Manual)
6. possible_keys和key
In the execution plan output by the EXPLAIN statement, the column indicates which indexes are executed possible_keyson a certain table in a certain query statement . 单表查询时可能用If an index exists on a field involved in a general query, the index will be listed, but may not be used by the query. What are the indexes keyrepresented by the column ? If it is NULL, no index is used. 实际用到For example, consider the following query:
mysql> EXPLAIN SELECT * FROM s1 WHERE key1 > 'z' AND key3 = 'a';
mysql> EXPLAIN SELECT * FROM s1 WHERE key1 > 'z' AND key3 = 'a';
+----+-------------+------+------+-------------------+----------+
| id | select_type | table| type | possible_keys | key |
+----+-------------+------+------+-------------------+----------+
| 1 | SIMPLE | s1 | ref | idx_key1,idx_key3 | idx_key3 |
+----+-------------+------+------+-------------------+----------+
1 row in set, 1 warning (0.01 sec)
The value of the possible_keys column of the above execution plan is idx_key1,idx_key3, indicating that the query may use two indexes, and the value of the key column is idx_key1, indicating that after the query optimizer calculates the cost of using different indexes, it finally decides to useidx_key3idx_key3idx_key3
Only one index can be used. So he has to choose one and use it. If you look at the above
index_mergeor, you will go through index merging.
7. key_len ☆
-
key_len: the actual index length used (ie: number of bytes)
-
The smaller the key_len, the better the indexing effect. This is what I learned earlier. Shorter key_len is more efficient.
-
But in the joint index, key_len is hit once plus the length. The longer the value, the higher the accuracy and the better the effect.
#7.
EXPLAIN SELECT * FROM s1 WHERE id = 10005;
## 结果key_len =4
mysql> EXPLAIN SELECT * FROM s1 WHERE key2 = 10126;
## 结果key_len = 5
key2 is a unique index of type int. . Because there may also be a null value, null occupies a field. 4+1 = 5
mysql> EXPLAIN SELECT * FROM s1 WHERE key1 = 'a';
## 结果key_len = 303
Reason: idx_key_part(key_part1, key_part2, key_part3）It is the sum of three 100 fields. Each field can be empty, so 101*3 = 303
mysql> EXPLAIN SELECT * FROM s1 WHERE key_part1 = 'a';
+----+------------+------+---------------+--------------+---------+-------+------+
| id | select_type| type | possible_keys | key | key_len | ref | rows |
+----+------------+------+---------------+--------------+---------+-------+------+
| 1 | SIMPLE | ref | idx_key_part | idx_key_part | 303 | const | 1 |
+----+------------+------+---------------+--------------+---------+-------+------+
1 row in set, 1 warning (0.00 sec)
The result key_len is 303
mysql> EXPLAIN SELECT * FROM s1 WHERE key_part1 = 'a' AND key_part2 = 'b';
+----+------------+-----+---------------+--------------+---------+------------
| id | select_type|type | possible_keys | key | key_len | ref
+----+------------+-----+---------------+--------------+---------+------------
| 1 | SIMPLE |ref | idx_key_part | idx_key_part | 606 | const,const
+----+------------+-----+---------------+--------------+---------+------------
1 row in set, 1 warning (0.00 sec)
Result key_606
The joint index is hit twice here, with higher accuracy and better effect.
practise:
The length calculation formula of key_len:
varchar(10)变长字段且允许NULL = 10 * ( character set：utf8=3,gbk=2,latin1=1)+1(NULL)+2(变长字段)
varchar(10)变长字段且不允许NULL = 10 * ( character set：utf8=3,gbk=2,latin1=1)+2(变长字段)
char(10)固定字段且允许NULL = 10 * ( character set：utf8=3,gbk=2,latin1=1)+1(NULL)
char(10)固定字段且不允许NULL = 10 * ( character set：utf8=3,gbk=2,latin1=1)
8. ref
# 8. ref：当使用索引列等值查询时，与索引列进行等值匹配的对象信息。
#比如只是一个常数或者是某个列。
mysql> EXPLAIN SELECT * FROM s1 WHERE key1 = 'a';
+----+-------------+------+------+---------------+----------+---------+-------+
| id | select_type | table| type | possible_keys | key | key_len | ref |
+----+-------------+------+------+---------------+----------+---------+-------+
| 1 | SIMPLE | s1 | ref | idx_key1 | idx_key1 | 303 | const |
+----+-------------+------+------+---------------+----------+---------+-------+
The type is type =ref, compared with const (constant)
mysql> EXPLAIN SELECT * FROM s1 INNER JOIN s2 ON s1.id = s2.id;
+---+------+--------+---------------+--------+------------------+-----
| id| table| type | possible_keys | key | ref | rows
+---+------+--------+---------------+--------+------------------+-----
| 1| s1 | ALL | PRIMARY | NULL | NULL | 9895
| 1| s2 | eq_ref | PRIMARY | PRIMARY| atguigudb1.s1.id | 1
+---+------+--------+---------------+--------+------------------+-----
The type is type =eq_ref, compared with atguigudb1.s1.id
mysql> EXPLAIN SELECT * FROM s1 INNER JOIN s2 ON s2.key1 = UPPER(s1.key1);
+----+------+------+---------------+----------+---------+------+------+----------------------+
| id | table| type | possible_keys | key | key_len | ref | rows |Extra |
+----+------+------+---------------+----------+---------+------+------+----------------------+
| 1 | s1 | ALL | NULL | NULL | NULL | NULL | 9895 |NULL |
| 1 | s2 | ref | idx_key1 | idx_key1 | 303 | func | 1 |Using index condition |
+----+------+------+---------------+----------+---------+------+------+----------------------+
Compare to a methodfunc
9. rows ☆
# 9. rows：预估的需要读取的记录条数
# `值越小越好`
# 通常与filtered 一起使用
EXPLAIN SELECT * FROM s1 WHERE key1 > 'z';
The smaller the rows value, the more likely it is that the data is in one page, so the io will be smaller.
10. filtered
The bigger the better
The value of filtered refers to the percentage of rows that return results to the rows that need to be read (the value of the rows column).
My own understanding: For example, after reading 100 rows. filtered is 10%, which means that 100 rows need to be filtered. . . . . . . . . This is completely unfounded because of my own obscenity, so I can only understand it this way.
If you are using a single table scan performed by an index, then when calculating, you need to estimate how many records meet other search conditions except the search conditions using the corresponding index.
EXPLAIN SELECT * FROM s1 WHERE key1 > 'z' AND common_field = 'a';
For single-table queries, the value of this filtered column is meaningless . For us 更关注在连接查询中驱动表对应的执行计划记录的filtered值, it determines the number of times the driven table is to be executed (ie: rows * filtered)
EXPLAIN SELECT * FROM s1 INNER JOIN s2 ON s1.key1 = s2.key1 WHERE s1.common_field = 'a';
11. Extra ☆
As the name suggests, Extracolumns are used to describe some additional information, including additional information that is not suitable for display in other columns but is very important. We can use this additional information 更准确的理解MySQL到底将如何执行给定的查询语句. There are dozens of additional information provided by MySQL. Let’s take a look at the highlights.
-
No tables usedThis additional information will be prompted when the query statement does not have a FROM clause, such as:
mysql> EXPLAIN SELECT 1;
mysql> EXPLAIN SELECT * FROM s1 WHERE 1 != 1;
-
Using where
When we use a full table scan to execute a query on a certain table, and
WHEREthere are search conditions for the table in the clauses of the statement,Extrathe above additional information will be prompted in the column.
EXPLAIN SELECT * FROM s1 WHERE common_field = 'a';
When there are other conditions besides index, this prompt will also appear.
#当使用索引访问来执行对某个表的查询，并且该语句的`WHERE`子句中
#有除了该索引包含的列之外的其他搜索条件时，在`Extra`列中也会提示上述额外信息。
explain SELECT * FROM s1 WHERE key1 = 'fUhcQU' and common_field = 'uDHCOnalcF';
- No matching min/max row`
When there is an aggregate function in the query list MIN, MAXbut there are no WHERErecords that match the search conditions in the clause, this additional information will be prompted.
# 数据库不存在 QLjKYOx
EXPLAIN SELECT MIN(key1) FROM s1 WHERE key1 = 'QLjKYOx';
# 数据库存在 QLjKYO
EXPLAIN SELECT MIN(key1) FROM s1 WHERE key1 = 'QLjKYO';
Using index
When our query list and search conditions only contain columns that belong to a certain index, that is, when a covering index can be used, Extrathe additional information will be prompted for the column.
For example, the following query only needs to be used idx_key1and does not require table return operations:
EXPLAIN SELECT key1 FROM s1 WHERE key1 = 'a';
Using index condition
Although index columns appear in some search conditions, the index cannot be used to view the courseware and understand the index conditions.
SELECT * FROM s1 WHERE key1 > 'z' AND key1 LIKE '%a';
mysql> EXPLAIN SELECT * FROM s1 WHERE key1 > 'z' AND key1 LIKE '%a';
-
Step 1. Here key1 > 'z' went through the index and found 378 pieces of data. . .
Step 2. key1 LIKE '%b'; This condition is still the key1 index, so next we only need to traverse these 378 indexes. Which ones match '%a'
Step 3. Filter out valid indexes through step 2. . This is Using index condition.
Step 4. Return the indexes that meet the conditions to query the table.
Full description:
The
key1 > 'z'index can be used, butkey1 LIKE '%a 'the index cannot be used. In previous versions of MySQL, this query was executed according to the following steps:- First, according to the condition key1 > 'z',
idx_key1obtain the corresponding secondary index record from the secondary index. - Return to the table based on the primary key value in the secondary index record obtained in the previous step, find the complete user record, then check whether the record meets this
key1 LIKE '%a'condition, and add the qualified record to the final result set.
However, although
key1 LIKE ‘%a'a range interval cannot be formed to participaterangein the execution of the access method, this condition only involveskey1columns after all, so MySQL has improved the above steps:-
First, based on
key1 > 'z'this condition, locateidx_key1the corresponding secondary index record in the secondary index. -
For the specified secondary index record, instead of rushing back to the table, first check whether the record meets
key1 LIKE ‘%a'this condition. If this condition is not met, there is no need to return the secondary index record to the table at all. -
key1 LIKE '%a'Perform table return operations for secondary index records that meet this condition.
We say that the table return operation is actually a random IO, which is relatively time-consuming. Therefore, although the above modifications only improve a little, it can save a lot of costs in the table return operation. MySQL calls their improvement
索引条件下推(English name: Index Condition Pushdown). If this feature is to be used during the execution of the query statement索引条件下推, it will be displayed in the Extra columnUsing index condition - First, according to the condition key1 > 'z',
-
Using join buffer (Block Nested Loop)Fields without indexes are associated with tables.
During the execution of the join query, when the driven table cannot effectively use the index to speed up the access, MySQL will generally allocate a
join buffermemory block named for it to speed up the query, which is what we are talking about.基于块的嵌套循环算法mysql> EXPLAIN SELECT * FROM s1 INNER JOIN s2 ON s1.common_field = s2.common_field;
-
Not existsWhen we use a left (outer) join, if the clause contains a search condition
WHEREthat requires a column of the driven table to be equal to a value, and that column is not allowed to store values, then in the Extra column of the execution plan of the table Additional information will be promptedNULLNULLNot existsEXPLAIN SELECT * FROM s1 LEFT JOIN s2 ON s1.key1 = s2.key1 WHERE s2.id IS NULL; # 都表关联了，，关联字段怎么会等于 is null
Using intersect(...) 、 Using union(...) 和 Using sort_union(...) -
ExtraIf a prompt appears in the execution plan columnUsing intersect(...), it means that you are ready to useIntersectthe index. -
Execute the query by merging. The name in parentheses
...indicates the index name that needs to be merged; -
If a prompt appears
Using union(...), it means that you are ready to useUnionindex merging to execute the query; -
A prompt appears
Using sort_union(...)indicating thatSort-Unionthe query is to be executed using index merging.EXPLAIN SELECT * FROM s1 WHERE key1 = 'a' OR key3 = 'a';
-
Zero limit -
Using filesortIn some cases, indexes can be used to sort records in the result set, such as the following query:
EXPLAIN SELECT * FROM s1 ORDER BY key1 LIMIT 10;This query statement can use
idx_key1the index to directly retrieve the 10 records of the key1 column, and then perform the table operation. However, in many cases, the sorting operation cannot use the index, and can only be sorted in memory (when there are few records) or on the disk (when there are many records). MySQL uses this method of sorting in memory or on disk. Collectively called file sorting (English name:filesort).Using filesortIf a query requires file sorting to be executed, a prompt will be displayed in the Extra column of the execution plan.EXPLAIN SELECT * FROM s1 ORDER BY common_field LIMIT 10; -
Using temporaryDuring the execution of many queries, MySQL may use temporary tables to complete some functions, such as deduplication, sorting, etc. For example, when we execute many queries containing, , and other clauses, if we cannot effectively use the
DISTINCTindexGROUP BYtoUNIONUpon completion of the query, MySQL will most likely seek to execute the query by creating an internal temporary table. If an internal temporary table is used in the query, a promptExtrawill be displayed in the execution plan column.Using temporaryEXPLAIN SELECT DISTINCT common_field FROM s1;
#执行计划中出现`Using temporary`并不是一个好的征兆，因为建立与维护临时表要付出很大成本的，所以
#我们`最好能使用索引来替代掉使用临时表`。比如：扫描指定的索引idx_key1即可
EXPLAIN SELECT key1, COUNT(*) AS amount FROM s1 GROUP BY key1;
12. Summary
- EXPLAIN does not consider various caches
- EXPLAIN cannot display the optimization work done by MySQL when executing the query
- EXPLAIN doesn't tell you about triggers, stored procedures, or how user-defined functions affect your query.
- Some statistics are estimates, not exact values
7. Further use of EXPLAIN
7.1 EXPLAIN four output formats
Let’s talk about the output format of EXPLAIN. EXPLAIN can output four formats: 传统格式, JSON格式, TREE格式and 可
视化输出. Users can choose the format that suits them according to their needs.
1. Traditional format
The traditional format is simple and straightforward, with the output being a table summarizing the query plan.
mysql> EXPLAIN SELECT s1.key1, s2.key1 FROM s1 LEFT JOIN s2 ON s1.key1 = s2.key1 WHERE
s2.common_field IS NOT NULL;
+----+-------------+-------+------------+------+---------------+-------
| id | select_type | table | partitions | type | possible_keys | key
+----+-------------+-------+------------+------+---------------+-------
| 1 | SIMPLE | s2 | NULL | ALL | idx_key1 | NULL
| 1 | SIMPLE | s1 | NULL | ref | idx_key1 | idx_ke
+----+-------------+-------+------------+------+---------------+-------
2 rows in set, 1 warning (0.00 sec)
2. JSON format
The statement output introduced in the first format EXPLAINlacks an important attribute that measures the quality of the execution plan - 成本. The JSON format is the output information in four formats 最详尽的格式, which contains execution cost information.
-
JSON format: Add it between the EXPLAIN word and the real query statement
FORMAT=JSON.EXPLAIN FORMAT=JSON SELECT .... -
Correspondence between EXPLAIN's Column and JSON: (from MySQL 5.7 documentation)
In this way, we can get an execution plan in json format, which contains the cost of the plan, such as this:
mysql> EXPLAIN FORMAT=JSON SELECT * FROM s1 INNER JOIN s2 ON s1.key1 = s2.key2 WHERE s1.common_field = 'a' \G
*************************** 1. row ***************************
EXPLAIN: {
"query_block": {
"select_id": 1,
"cost_info": {
"query_cost": "1360.07"
},
"nested_loop": [
{
"table": {
"table_name": "s1",
"access_type": "ALL",
"possible_keys": [
"idx_key1"
],
"rows_examined_per_scan": 9895,
"rows_produced_per_join": 989,
"filtered": "10.00",
"cost_info": {
"read_cost": "914.80",
"eval_cost": "98.95",
"prefix_cost": "1013.75",
"data_read_per_join": "1M"
},
"used_columns": [
"id",
"key1",
"key2",
"key3",
"key_part1",
"key_part2",
"key_part3",
"common_field"
],
"attached_condition": "((`atguigudb1`.`s1`.`common_field` = 'a') and (`atguigudb1`.`s1`.`key1` is not null))"
}
},
{
"table": {
"table_name": "s2",
"access_type": "eq_ref",
"possible_keys": [
"idx_key2"
],
"key": "idx_key2",
"used_key_parts": [
"key2"
],
"key_length": "5",
"ref": [
"atguigudb1.s1.key1"
],
"rows_examined_per_scan": 1,
"rows_produced_per_join": 989,
"filtered": "100.00",
"index_condition": "(cast(`atguigudb1`.`s1`.`key1` as double) = cast(`atguigudb1`.`s2`.`key2` as double))",
"cost_info": {
"read_cost": "247.38",
"eval_cost": "98.95",
"prefix_cost": "1360.08",
"data_read_per_join": "1M"
},
"used_columns": [
"id",
"key1",
"key2",
"key3",
"key_part1",
"key_part2",
"key_part3",
"common_field"
]
}
}
]
}
}
1 row in set, 2 warnings (0.01 sec)
We use #the form followed by comments to explain EXPLAIN FORMAT=JSONthe output content of the statement, but you may
have questions. cost_infoThe costs in " " look strange. How are they calculated? Let’s look at the “ ” part s1of the table first :cost_info
"cost_info": {
"read_cost": "914.80",
"eval_cost": "98.95",
"prefix_cost": "1013.75",
"data_read_per_join": "1M"
}
-
read_costIt is composed of the following two parts:- IO cost
- CPU cost of detecting
rows × (1 - filter)records
Tips: rows and filter are the output columns of the execution plan we introduced earlier. In the execution plan in JSON format, rows is equivalent to rows_examined_per_scan, and the filtered name remains unchanged.
-
eval_costThis is how it is calculatedThe cost of detecting
rows × filterrecords. -
prefix_costThis is the cost of querying the table alones1, that is:read_cost + eval_cost -
data_read_per_joinIndicates the amount of data that needs to be read in this query.
The "cost_info" part of the s2 table looks like this:
"cost_info": {
"read_cost": "247.38",
"eval_cost": "98.95",
"prefix_cost": "1360.08",
"data_read_per_join": "1M"
}
Since s2the table is a driven table, it may be read multiple times. The read_costand here eval_costare the values accumulated after accessing s2the table multiple times
. Everyone is mainly concerned about the value inside prefix_costwhich represents the estimated cost of the entire connection query, that is The sum of the costs after a single query of the table and multiple queries of the table, that
is s1:s2
247.38 + 98.95 + 1013.75 = 1360.08
3. TREE format
TREE format is a new format introduced after version 8.0.16. It mainly describes how to query based on the query 各个部分之间的关系and各部分的执行顺序
mysql> EXPLAIN FORMAT=tree SELECT * FROM s1 INNER JOIN s2 ON s1.key1 = s2.key2 WHERE s1.common_field = 'a'\G
*************************** 1. row ***************************
EXPLAIN: -> Nested loop inner join (cost=1360.08 rows=990)
-> Filter: ((s1.common_field = 'a') and (s1.key1 is not null)) (cost=1013.75 rows=990)
-> Table scan on s1 (cost=1013.75 rows=9895)
-> Single-row index lookup on s2 using idx_key2 (key2=s1.key1), with index condition: (cast(s1.key1 as double) = cast(s2.key2 as double)) (cost=0.25 rows=1)
1 row in set, 1 warning (0.00 sec)
4. Visual output
Visual output, you can visually view the MySQL execution plan through MySQL Workbench. By clicking the magnifying glass
icon of Workbench, you can generate a visual query plan.
The above figure shows the tables in join order from left to right. Red boxes indicate 全表扫描, while green boxes indicate use 索引查找. For each table,
the indexes used are displayed. Also note that above the box for each table are estimates of the number of rows discovered by each table access and the
cost of accessing that table.
7.2 Use of SHOW WARNINGS
mysql> EXPLAIN SELECT s1.key1, s2.key1 FROM s1 LEFT JOIN s2 ON s1.key1 = s2.key1 WHERE
s2.common_field IS NOT NULL;
Use explain immediately after using explainSHOW WARNINGS \G
mysql> SHOW WARNINGS\G
*************************** 1. row ***************************
Level: Note
Code: 1003
Message: /* select#1 */ select `atguigudb1`.`s1`.`key1` AS `key1`,`atguigudb1`.`s2`.`key1` AS `key1` from `atguigudb1`.`s1` join `atguigudb1`.`s2` where ((`atguigudb1`.`s1`.`key1` = `atguigudb1`.`s2`.`key1`) and (`atguigudb1`.`s2`.`common_field` is not null))
1 row in set (0.00 sec)
You can see the statements actually executed by the query optimizer
Sticking it out may not necessarily allow it to run
You can see that SHOW WARNINGSthe displayed information has three fields, namely Level, Code, and Message. Our most common
information is the information with Code 1003. When the Code value is 1003, the information displayed in the Message field is similar to the statement after rewriting查询优化器 our query statement . For example, our query above was originally a left (outer) join query, but there is a condition of s2.common_field IS NOT NULL, which will cause the query optimizer to optimize the left (outer) join query into an inner join query, from the field It can also be seen that the original LEFT JOIN has become a JOIN (inner join).SHOW WARNINGSMessage
8. Analyze the optimizer execution plan: trace
OPTIMIZER_TRACEIt is a tracking feature introduced in MySQL 5.6. It can track various decisions made by the optimizer (such as methods of accessing tables, various cost calculations, various transformations, etc.) and record the tracking results into the table INFORMATION_SCHEMA.OPTIMIZER_TRACE.
This feature is turned off by default. Enable trace, set the format to JSON, and set the maximum memory size that trace can use to avoid complete display due to the default memory being too small during the parsing process.
SET optimizer_trace="enabled=on",end_markers_in_json=on;
set optimizer_trace_max_mem_size=1000000;
After turning it on, the following statements can be analyzed:
- SELECT
- INSERT
- REPLACE
- UPDATE
- DELETE
- EXPLAIN
- SET
- DECLARE
- CASE
- IF
- RETURN
- CALL
Test: Execute the following SQL statement
select * from student where id < 10;
Finally, query information_schema.optimizer_trace to know how MySQL executes SQL:
select * from information_schema.optimizer_trace\G
mysql> select * from information_schema.optimizer_trace\G
*************************** 1. row ***************************
# //第1部分：查询语句
QUERY: select * from student where id < 10
//第2部分：QUERY字段对应语句的跟踪信息
TRACE: {
"steps": [
{
"join_preparation": { /*预备工作*/
"select#": 1,
"steps": [
{
"expanded_query": "/* select#1 */ select `student`.`id` AS `id`,`student`.`stuno` AS `stuno`,`student`.`name` AS `name`,`student`.`age` AS `age`,`student`.`classId` AS `classId` from `student` where (`student`.`id` < 10)"
}
] /* steps */
} /* join_preparation */
},
{
"join_optimization": {
/*进行优化*/
"select#": 1,
"steps": [
{
"condition_processing": {
/*条件处理*/
"condition": "WHERE",
"original_condition": "(`student`.`id` < 10)",
"steps": [
{
"transformation": "equality_propagation",
"resulting_condition": "(`student`.`id` < 10)"
},
{
"transformation": "constant_propagation",
"resulting_condition": "(`student`.`id` < 10)"
},
{
"transformation": "trivial_condition_removal",
"resulting_condition": "(`student`.`id` < 10)"
}
] /* steps */
} /* condition_processing */
},
{
"substitute_generated_columns": {
/*替换生成的列*/
} /* substitute_generated_columns */
},
{
"table_dependencies": [ /* 表的依赖关系*/
{
"table": "`student`",
"row_may_be_null": false,
"map_bit": 0,
"depends_on_map_bits": [
] /* depends_on_map_bits */
}
] /* table_dependencies */
},
{
"ref_optimizer_key_uses": [ /* 使用键*/
] /* ref_optimizer_key_uses */
},
{
"rows_estimation": [ /*行判断*/
{
"table": "`student`",
"range_analysis": {
"table_scan": {
"rows": 3945207,
"cost": 404306
} /* table_scan */,/*表扫描*/
"potential_range_indexes": [
{
"index": "PRIMARY",
"usable": true,
"key_parts": [
"id"
] /* key_parts */
}
] /* potential_range_indexes */,
"setup_range_conditions": [
] /* 设置条件范围 */,
"group_index_range": {
"chosen": false,
"cause": "not_group_by_or_distinct"
} /* group_index_range */,
"skip_scan_range": {
"potential_skip_scan_indexes": [
{
"index": "PRIMARY",
"usable": false,
"cause": "query_references_nonkey_column"
}
] /* potential_skip_scan_indexes */
} /* skip_scan_range */,
"analyzing_range_alternatives": {
/*分析范围选项*/
"range_scan_alternatives": [
{
"index": "PRIMARY",
"ranges": [
"id < 10"
] /* ranges */,
"index_dives_for_eq_ranges": true,
"rowid_ordered": true,
"using_mrr": false,
"index_only": false,
"in_memory": 0.159895,
"rows": 9,
"cost": 1.79883,
"chosen": true
}
] /* range_scan_alternatives */,
"analyzing_roworder_intersect": {
"usable": false,
"cause": "too_few_roworder_scans"
} /* analyzing_roworder_intersect */
} /* analyzing_range_alternatives */,
"chosen_range_access_summary": {
/*选择范围访问摘要*/
"range_access_plan": {
"type": "range_scan",
"index": "PRIMARY",
"rows": 9,
"ranges": [
"id < 10"
] /* ranges */
} /* range_access_plan */,
"rows_for_plan": 9,
"cost_for_plan": 1.79883,
"chosen": true
} /* chosen_range_access_summary */
} /* range_analysis */
}
] /* rows_estimation */
},
{
"considered_execution_plans": [/*考虑执行计划*/
{
"plan_prefix": [
] /* plan_prefix */,
"table": "`student`",
"best_access_path": {
/*最佳访问路径*/
"considered_access_paths": [
{
"rows_to_scan": 9,
"access_type": "range",
"range_details": {
"used_index": "PRIMARY"
} /* range_details */,
"resulting_rows": 9,
"cost": 2.69883,
"chosen": true
}
] /* considered_access_paths */
} /* best_access_path */,
"condition_filtering_pct": 100, /*行过滤百分比*/
"rows_for_plan": 9,
"cost_for_plan": 2.69883,
"chosen": true
}
] /* considered_execution_plans */
},
{
"attaching_conditions_to_tables": { /*将条件附加到表上*/
"original_condition": "(`student`.`id` < 10)",
"attached_conditions_computation": [
] /* attached_conditions_computation */,
"attached_conditions_summary": [ /*附加条件概要*/
{
"table": "`student`",
"attached": "(`student`.`id` < 10)"
}
] /* attached_conditions_summary */
} /* attaching_conditions_to_tables */
},
{
"finalizing_table_conditions": [
{
"table": "`student`",
"original_table_condition": "(`student`.`id` < 10)",
"final_table_condition ": "(`student`.`id` < 10)"
}
] /* finalizing_table_conditions */
},
{
"refine_plan": [ /*精简计划*/
{
"table": "`student`"
}
] /* refine_plan */
}
] /* steps */
} /* join_optimization */
},
{
"join_execution": { /*执行*/
"select#": 1,
"steps": [
] /* steps */
} /* join_execution */
}
] /* steps */
}
/
/*第3部分：跟踪信息过长时，被截断的跟踪信息的字节数。*/
MISSING_BYTES_BEYOND_MAX_MEM_SIZE: 0 /*丢失的超出最大容量的字节*/
/*第4部分：执行跟踪语句的用户是否有查看对象的权限。当不具有权限时，该列信息为1且TRACE字段为空，一般在
调用带有SQL SECURITY DEFINER的视图或者是存储过程的情况下，会出现此问题。*/
INSUFFICIENT_PRIVILEGES: 0 /*缺失权限*/
1 row in set (0.01 sec)
9. MySQL monitoring analysis view-sys schema
9.1 Sys schema view summary
1. Host-related : Starting with host_summary, it mainly summarizes IO delay information.
2. Innodb related : Starting with innodb, it summarizes the innodb buffer information and information about transactions waiting for innodb locks.
3. I/O related : Starting with io, it summarizes the waiting for I/O and I/O usage.
4. Memory usage : starting with memory, showing the memory usage from the perspective of host, thread, event, etc.
5. Connection and session information : processlist and session-related views summarize session-related information.
6. Table related : Views starting with schema_table show the statistical information of the table.
7. Index information : statistics on index usage, including redundant indexes and unused indexes.
8. Statement-related : Starting with statement, it contains statement information for performing full table scan, using temporary tables, sorting, etc.
9. User related : The view starting with user counts the file I/O and execution statement statistics used by the user.
10. Information related to waiting events : starting with wait, showing the delay of waiting events.
9.2 Sys schema view usage scenarios
Index status
#1. 查询冗余索引
select * from sys.schema_redundant_indexes;
#2. 查询未使用过的索引
select * from sys.schema_unused_indexes;
#3. 查询索引的使用情况
select index_name,rows_selected,rows_inserted,rows_updated,rows_deleted
from sys.schema_index_statistics where table_schema='dbname' ;
table related
# 1. 查询表的访问量
select table_schema,table_name,sum(io_read_requests+io_write_requests) as io from
sys.schema_table_statistics group by table_schema,table_name order by io desc;
# 2. 查询占用bufferpool较多的表
select object_schema,object_name,allocated,data
from sys.innodb_buffer_stats_by_table order by allocated limit 10;
# 3. 查看表的全表扫描情况
select * from sys.statements_with_full_table_scans where db='dbname';
Statement related
#1. 监控SQL执行的频率
select db,exec_count,query from sys.statement_analysis
order by exec_count desc;
#2. 监控使用了排序的SQL
select db,exec_count,first_seen,last_seen,query
from sys.statements_with_sorting limit 1;
#3. 监控使用了临时表或者磁盘临时表的SQL
select db,exec_count,tmp_tables,tmp_disk_tables,query
from sys.statement_analysis where tmp_tables>0 or tmp_disk_tables >0
order by (tmp_tables+tmp_disk_tables) desc;
IO related
#1. 查看消耗磁盘IO的文件
select file,avg_read,avg_write,avg_read+avg_write as avg_io
from sys.io_global_by_file_by_bytes order by avg_read limit 10;
Innodb related
#1. 行锁阻塞情况
select * from sys.innodb_lock_waits;
risk warning:
When querying through the sys library, MySQL will consume a lot of resources to collect relevant information. In severe cases, business requests may be blocked, causing failures. It is recommended that those in production
不要频繁query sys or performance_schema and information_schema to complete monitoring, inspection and other tasks.