1. Slow SQL optimization ideas
- Slow Query Logging Slow SQL
- explain analyzes SQL execution plan
- profile analysis execution time
- Optimizer Trace analysis details
- Identify the problem and take appropriate action
1. Slow Query Logging Slow SQL
How to locate slow SQL?
We can view slow SQL through the slow query log .
①: Enable slow query log:
SET global slow_query_log = ON;:Set the status of slow query opening (ON: on; OFF: off)slow_query_log_file: Set the location where slow query logs are storedSET global log_queries_not_using_indexes = ON;: Record query SQL that does not use indexes. The premise isslow_query_logthat the value is ON, otherwise it will not workSET long_query_time = 10;: Set the slow query threshold in seconds. If the SQL execution time exceeds the threshold, it is a slow query and is recorded in the log file.
②: Check the slow query log configuration:
show variables like 'slow_query_log%show variables like 'long_query_time'
③: Slow query log analysis tool:
mysqldumpslow: This tool is a slow query analysis tool that comes with slow query. Generally, as long as mysql is installed, this tool will be available.
# 取出使用最多的10条慢查询
mysqldumpslow -s c -t 10 /var/run/mysqld/mysqld-slow.log
# 取出查询时间最慢的3条慢查询
mysqldumpslow -s t -t 3 /var/run/mysqld/mysqld-slow.log
# 得到按照时间排序的前10条里面含有左连接的查询语句
mysqldumpslow -s t -t 10 -g “left join” /database/mysql/slow-log
# 按照扫描行数最多的
mysqldumpslow -s r -t 10 -g 'left join' /var/run/mysqld/mysqld-slow.log
Note: mysqldumpslowThe analysis results using will not display the specific and complete sql statement, but will only display the composition structure of sql;
if:SELECT * FROM sms_send WHERE service_id=10 GROUP BY content LIMIT 0, 1000;
Count: 1 Time=1.91s (1s) Lock=0.00s (0s) Rows=1000.0 (1000), vgos_dba[vgos_dba]@[10.130.229.196]
SELECT * FROM sms_send WHERE service_id=N GROUP BY content LIMIT N, N;
There are actually many tools, and we are not limited to this one. There are also tools such as pt-query-digest, mysqlslaetc. These are all small tools that can locate slow query logs.
Reasons for slow query:
- Full table scan: explain analysis type attribute all
- Full index scan: explain analysis of type attribute index
- Poor index filtering: depends on index field selection, data volume and status, and table design
- Frequent table query overhead: use select * as little as possible and use covering indexes
＜Transfer＞Detailed explanation of slow query mysqldumpslow
2. explain View and analyze the execution plan of SQL
After locating the SQL with low query efficiency, you can use explain to view the execution plan of the SQL.
When explain is used with SQL, MySQL displays information from the optimizer about the statement's execution plan. Namely: MySQL explains how it will process the statement, including information about how to join the tables and in what order:
Generally speaking, we need to focus ontype、key、rows、extra
13.1 type
type represents the connection type, an important indicator of index performance. The following performance, in order from best to worst:system > const > eq_ref > ref > ref_or_null > index_merge > unique_subquery > index_subquery > range > index > ALL
- NULL : Indicates no need to access the table, the fastest
- system : This type requires only one piece of data in the database table. It is a special case of the const type and will not appear under normal circumstances.
- const : Data can be found through one index. It is generally used as a condition for primary key or unique index. This type of scanning is extremely efficient and very fast.
- eq_ref : often used for primary key or unique index scans, generally refers to related queries using primary keys
- ref : often used for non-primary key and unique index scans
- ref_or_null : This connection type is similar to ref, except that MySQL will additionally search for rows containing NULL values.
- index_merge : The index merge optimization method is used, and the query uses more than two indexes
- unique_subquery : similar to eq_ref, the condition uses in subquery
- index_subquery : different from unique_subquery, used for non-unique indexes and can return duplicate values
- range : Commonly used for range queries, such as: between...and or In and other operations
- index : full index scan
- all : full table scan
13.2 possible_keys
Represents the index that can be used when querying (the index name is displayed),Only the index that may be used, not the index that is actually used
13.3 key
This column represents the actual index used. Generally speaking, possible_keyswe will look together in a row.
13.4 rows
The MySQL query optimizer will use statistical information to estimate how many rows of records need to be scanned by SQL to obtain the results. In principle, the fewer rows, the higher the efficiency. You can intuitively understand the efficiency of SQL.
13.5 extra
This field contains additional information about how MySQL parses the query, and will typically have these values:
- Using filesort : Indicates sorting by files, which usually occurs when the specified sorting and index sorting are inconsistent. Generally seen in order by statements. Suggested optimization
- Using temporary : indicates that a temporary table is used, and the performance is particularly poor and needs to be optimized. Generally seen in group by statements or union statements
- Using index : Indicates that a covering index is used
- Using where : indicates that where condition filtering is used, and data needs to be queried through the index back to the table.
- Using index condition : New index push-down after MySQL5.6. Perform data filtering at the storage engine layer instead of filtering at the service layer, and use indexing of existing data to reduce data returned to the table.
- NULL : The queried column is not covered by the index
Summarize:
| extra | where condition | select field |
|---|---|---|
| null | where filter condition is the leading column of the index | The queried column is not covered by the index |
| Using index | where filter condition is the leading column of the index | The queried column is covered by the index |
| Using where; Using index | where filter condition is one of the index columns but not the leading column or where filter condition is a range of the leading column of the index column | The queried column is covered by the index |
| Using where; | where filter condition is not an index column | - |
| Using where; | where filter condition is not the leading column of the index, but a range of the leading column of the index column (>) | Query column is not covered by index |
| Using index condition | where is a range of the leading column of the index column (<, between) | Query column is not covered by index |
Two sorting situations:
| extra | Appear scene |
|---|---|
| Using filesort | Filesort is mainly used for sorting query data result sets. First, MySQL will use the memory size of sort_buffer_size for sorting. If the result set exceeds the size of sort_buffer_size, the sorted chunk will be transferred to the file, and finally multi-way merge sorting will be used. Complete sorting of all data. |
| Using temporary | MySQL uses temporary tables to save temporary structures for subsequent processing. MySQL first creates a temporary table of the heap engine. If there is too much temporary data and exceeds the size of max_heap_table_size, the temporary table will be automatically converted into a table of the MyISAM engine for use. . |
filesort can only be applied to a single table. If data from multiple tables needs to be sorted, MySQL will first use using temporary to save the temporary data, then use filesort on the temporary table to sort, and finally output the results.
13.6 select_type
select_type: Indicates the type of query.
Commonly used values are as follows:
- SIMPLE : Indicates that the query statement does not contain subqueries or UNION
- PRIMARY : Indicates that this query is the outermost query
- UNION : Indicates that this query is the second or subsequent query of UNION
- DEPENDENT UNION : The second or subsequent query statement in UNION uses external query results
- UNION RESULT : The result of UNION
- SUBQUERY : SELECT subquery statement
- DEPENDENT SUBQUERY : The SELECT subquery statement depends on the results of the outer query
The most common query type is the UNION query SIMPLE, which means our query has no subqueries and does not use UNION queries.
13.7 filtered
This column is a percentage value, which is the percentage of the number of record rows finally queried through the query conditions and the number of record rows scanned through the type field. To put it simply, this field represents the proportion of the remaining records that meet the conditions after filtering the data returned by the storage engine.
13.8 key_len
Indicates the number of bytes of the index used by the query (you can determine whether all combined indexes are used)
The calculation rules of key_len are as follows:
- String type: The string length is related to the character set: latin1 = 1, gbk = 2, utf8 = 3, utf8mb4 = 4
char(n):n * character set lengthvarchar(n):n * character set length + 2 bytes
- Numeric type
TINYINT: 1 byteSMALLINT:2 bytesMEDIUMINT:3 bytesINT,FLOAT:4 bytesBIGINT,DOUBLE:8 bytes
- time type
DATE:3 bytesTIMESTAMP: 4 bytesDATETIME:8 bytes
- Field properties
- The NULL attribute occupies 1 byte. If a field is set to NOT NULL, there will be no such item.
3. Profile analysis execution time
explain only sees the estimated execution plan of SQL . If you want to know the real execution thread status and time consumption of SQL, you need to use profiling.
After the profiling parameter is turned on, subsequent SQL statements executed will record their resource overhead, including IO, context switching, CPU, memory, etc. We can further analyze the bottlenecks of the current slow SQL based on these overheads and further optimize
Check whether profiling is turned on:
show variables like '%profil%'
Turn on profiling:
set profiling=ON
Use profiling:
show profiles
show profiles will display multiple statements recently sent to the server. The number is profiling_history_sizedefined by the variable. The default is 15. If we need to see the analysis of a single SQL, we can use show profile to view the analysis of the most recent SQL, or we can use show profile for query id(the id is the QUERY_ID in show profiles) to view the analysis of a specific SQL statement:
4. Optimizer Trace analysis details
profile can only view the execution time of SQL, but cannot see the actual execution process information of SQL, that is, it does not know how the MySQL optimizer selects the execution plan. At this time, we can use it Optimizer Trace, which can track the entire process of parsing and optimizing the execution of the execution statement.
Turn on:
set optimizer_trace="enabled=on";
Viewing and analyzing its execution tree will include three stages:
- join_preparation: preparation phase
- join_optimization: analysis phase
- join_execution: execution phase
5. Identify the problem and take appropriate measures
Confirm the problem and take corresponding measures.
- Most slow SQL is related to indexes, such as no indexing, index ineffectiveness, unreasonableness, etc. At this time, we can optimize the index
- We can also optimize SQL statements, such as some problems with too many in elements (in batches), deep paging problems (based on the last data filtering, etc.), and perform time segmented queries
- SQL cannot be optimized well. You can use ES or data warehouse instead.
- If the amount of data in a single table is too large, resulting in slow query, you can consider splitting databases and tables.
- If the database is flushing dirty pages and causing slow queries, consider whether some parameters can be optimized and discuss the optimization plan with the DBA.
- If the amount of existing data is too large, consider whether some data can be archived