Query optimization for mysql performance tuning

Written before: this is the second part of msql performance tuning

MySQL performance optimization query

A MySQL query process

How do we optimize sql? First of all, we need to know that sql optimization is mainly to solve the problem of query optimization, so we start with the query of the database. The following picture shows the execution path of the query:

① The client sends the query to the server;

② The server checks the query cache, and if found, returns the result from the cache, otherwise proceeds to the next step.

③ Server analysis and preprocessing.

④ The query optimizer optimizes the query

⑤ Generate an execution plan, and the execution engine calls the storage engine API to execute the query

⑥ The server sends the result back to the client.

Query cache
Before parsing a query statement, if the query cache is open, MySQL will first check whether the query hits the data in the query cache. If the cache hits the data, it will directly get the result from the cache and return it to the client. In this case, the query will not be parsed, no execution plan will be generated, and it will not be executed.

Syntax parsing and preprocessor
MySQL parses SQL statements by keywords and generates a corresponding "parse tree". The MySQL parser will validate and parse the query using MySQL syntax rules.

Query optimizer
After the grammar book is verified and legal, the optimizer converts it into a query plan. A statement can be executed in many ways, and finally returns the same result. The role of the optimizer is to find the best execution plan among them.

Query execution engine
In the analysis and optimization phase, MySQL will generate an execution plan corresponding to the query, and MySQL's query execution engine will complete the entire query according to the execution plan. The most commonly used and most frequently used engines are the MyISAM engine and the InnoDB engine. The default storage engine starting with MySQL 5.5 has been changed to innodb.

Second query optimization

In the previous query process analysis, we have a general understanding of how MySQL is executed, and we will come to the parts involved one by one later. Now let's start with the query optimization part.

SQL is the most important part of our communication with the database, so when we are tuning, we need to spend a lot of time on SQL tuning. Common analysis methods include slow query logs and EXPLAIN analysis queries. By locating and analyzing performance bottlenecks, the performance of the database system can be better optimized.

slow query

Add two configuration parameters under the [mysqld] line in the configuration file my.cnf[linux] or my.ini[windows]

    log-slow-queries=C:/ProgramData/MySQL/MySQL Server 5.5/Data/mysqldata/slow-query.log
    long_query_time=5

The log-slow-queries parameter is the location where the slow query logs are stored. Generally, this directory must have the write permission of the mysql running account. Generally, this directory is set as the data storage directory of mysql;
5 in long_query_time=5 means that the query exceeds five seconds to record;

You can also add the log-queries-not-using-indexes parameter to my.cnf or my.ini, indicating that queries that do not use indexes are recorded.

SHOW VARIABLES LIKE 'long_query_time%';   -- 查看当前多少秒算慢
show global status like '%Slow_queries%'; --查询当前系统中有多少条慢查询记录

Slow query analysis

We can open the log file to see which SQLs are inefficiently executed. From the log, we can find the SQL whose query time exceeds 5 seconds, and the SQL whose query time is less than 5 seconds does not appear in this log.

If there is a lot of content recorded in the slow query log, you can use the mysqldumpslow tool (included with the MySQL client installation) to classify and summarize the slow query log. mysqldumpslow subtotals the log files, showing the summary results after the aggregation.

Enter the log storage directory and run:

[root@mysql_data]# mysqldumpslow slow-query.log
Reading mysql slow query log fromslow-query.log
Count: 2 Time=11.00s (22s) Lock=0.00s (0s)Rows=1.0 (2), root[root]@mysql
select count(N) from t_user; 

mysqldumpslowCommand

/path/mysqldumpslow -s c -t 10/database/mysql/slow-query.log

This will output the 10 SQL statements with the most records, where:

-s, indicates which way to sort, c, t, l, r are sorted according to the number of records, time, query time, and the number of returned records, respectively, ac, at, al, ar, indicate the corresponding flashback

-t, is the meaning of top n, that is, how many pieces of data are returned;

-g, you can write a regular matching pattern after it, which is not case-sensitive;

E.g:

/path/mysqldumpslow -s r -t 10/database/mysql/slow-log

Get the 10 queries that return the most recordsets.

/path/mysqldumpslow -s t -t 10 -g “leftjoin” /database/mysql/slow-log

Get the top 10 query statements with left joins sorted by time.

Using the mysqldumpslow command, we can get all the query statements we need very clearly. Monitoring, analyzing, and optimizing MySQL query statements is a very important step in MySQL optimization. After the slow query log is enabled, the performance of mysql will be affected to a certain extent due to the logging operation, but it can be enabled in stages to locate performance bottlenecks.

EXPLAIN execution plan (parse plan)

EXPLAIN can help developers analyze SQL problems. EXPLAIN shows how MySQL uses SQL execution plans, which can help developers write more optimized query statements. To use the method, just add Explain before the select statement:

EXPLAIN SELECT * FROM products

The resulting columns are described as follows:

1) id

SELECT identifier. This is the SELECT query sequence number. This is not important. It just identifies that there are multiple queries in the executed sql statement

2) select_type

Indicates the type of SELECT statement.

1. simple: Simple select (does not use unions or subqueries).
2. primary: the outermost selection.
3. union: The second or subsequent select statement in union.
4. dependent union: The second or subsequent select statement in the union, depending on the outer query.
5. union result: The result of the union.
6. subquery: The first select in the subquery.
7. dependent subquery: The first select in the subquery, which depends on the outer query.
8. derived: The select of the derived table (subquery of the from clause).

3. table

Shows which table the data for this query is about.

4) type 【Important】

Distinguishing index, this is the important column showing what type of join is used. The connection types from best to worst are:

system > const > eq_ref > ref > ref_or_null > index_merge > unique_subquery > index_subquery > range > index > ALL

Generally speaking, it is necessary to ensure that the query reaches at least the range level, preferably ref, and index.

1. system: There is only one row in the table, which is a special column of const type, which usually does not appear, and this can also be ignored.
2. const: The data table has at most one matching row, because only one row of data is matched, so it is very fast
3. eq_ref: The mysql manual says this: "For each combination of rows from the preceding table, read a row from that table. This is probably the best join type, except for the const type. It is used on all Parts are used by joins and the index is UNIQUE or PRIMARY KEY". eq_ref can be used for conditional comparison using =, and the column is an indexed column.
4. ref: The query condition index is neither UNIQUE nor PRIMARY KEY. ref can be used on indexed columns with the = or < or > operators.
5. ref_or_null: This join type is like ref, but with the addition that MySQL can specifically search for rows containing NULL values. This join type optimization is often used in solving subqueries.
6. index_merge: This join type indicates that the index merge optimization method is used. In this case, the key column contains the list of used indexes, and key_len contains the longest key element of the used indexes.
7. unique_subquery: This type replaces the ref of the IN subquery of the following form: value IN (SELECT primary_key FROM single_table WHERE some_expr) unique_subquery is an index lookup function that can completely replace subqueries and is more efficient.
8. index_subquery: This join type is similar to unique_subquery. IN subqueries can be replaced, but only for non-unique indexes in subqueries of the form: value IN (SELECT key_column FROM single_table WHERE some_expr)
9. range: Retrieve only rows in a given range, using an index to select rows.
10. index: This join type is the same as ALL, except that only the index tree is scanned. This is usually faster than ALL because index files are usually smaller than data files.
11. ALL: For each combination of rows from the previous table, a full table scan is performed. (Worst performance, remove all, at least index)

5) possible_keys

Indicates which index MySQL can use to find rows in this table. If empty, there is no associated index. To improve performance at this time, you can check the WHERE clause to see if some fields are referenced, or the search field is not suitable for indexing.

6) key

The actual index used. If NULL, no index is used. If it is primary, it means that the primary key is used.

7) key_len

The longest index width. If the key is NULL, the length is NULL. Shorter lengths are better without loss of accuracy.

(Limited ability, relatively understand the structure of the index, have the ability to tune the index, the algorithm is relatively strong)

8) ref

Shows which column or constant is used with key to select rows from the table.

9) rows

Displays the number of rows that MySQL thinks it must examine when executing the query. The smaller the number of rows, the higher the efficiency (whether there is an index)

10) Extra

Execution status description, this column contains the details of MySQL's solution query, the value of this column may have more than one of the following situations

• Distinct: MySQL stops searching for more rows for the current row combination after finding the first matching row.
• Not exists: MySQL can perform LEFT JOIN optimization on the query. After finding a row that matches the LEFT JOIN criteria, it will not check more rows in the table for the previous row combination.
• range checked for each record (index map: #): MySQL did not find a good index to use, but found that if the column value from the previous table is known, maybe a partial index can be used.
• 【Using filesort】: MySQL requires an extra pass to figure out how to retrieve rows in sorted order. In sorting, this means that real data records need to be read when sorting [read data file IO]
• Using index: Retrieve column information in a table from reading the actual row using only the information in the index tree without further searching.
• [Using temporary]: In order to solve complex queries, MySQL needs to create a temporary table to accommodate the results, which is inefficient.
• The Using where:WHERE clause is used to restrict which row matches the next table or is sent to the client.
• Using sort_union(…), Using union(…), Using intersect(…): These functions illustrate how to merge index scans for the index_merge join type.
• Using index for group-by: Similar to the Using index method of accessing a table, Using index for group-by means that MySQL has found an index that can be used to query all columns of a GROUP BY or DISTINCT query without additionally searching the hard disk to access the actual table data in .

Written later: The next article is about to write about index optimization