1. Explain
The Explain command is the first recommended command to solve database performance. Most performance problems can be solved simply by this command. Explain can be used to view the execution effect of SQL statements, which can help select better indexes and optimize query statements. , To write better optimized statements.
Explain语法：explain select … from … [where …]
For example: explain select * from news;
Output: +----+-------------+-------+-------+-------------------+---------+---------+-------+------| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |+----+-------------+-------+-------+-------------------+---------+---------+-------+------
The following is an understanding of each attribute:
1. id: This is the query sequence number of SELECT
2. Select_type: select_type is the type of select, which can be the following:
SIMPLE: simple SELECT (does not use UNION or subqueries, etc.)
PRIMARY: the outermost SELECT
UNION: the second or subsequent SELECT statement in UNION
DEPENDENT UNION: The second or subsequent SELECT statement in UNION, depending on the query outside
UNION RESULT: The result of UNION.
SUBQUERY: the first SELECT in the subquery
DEPENDENT SUBQUERY: The first SELECT in the subquery depends on the query outside
DERIVED: SELECT of the derived table (subqueries of the FROM clause)
3. Table: Shows which table the data in this row is about
4. Type: This column is the most important. It shows which category is used in the connection and whether the index is used or not. It is one of the key items to analyze the performance bottleneck using the Explain command.
The resulting values from good to bad are:
system > const > eq_ref > ref > fulltext > ref_or_null > index_merge > unique_subquery > index_subquery > range > index > ALL
Generally speaking, it is necessary to ensure that the query reaches at least the range level, and it is best to reach ref, otherwise performance problems may occur.
a.ALL: Full Table Scan, MySQL will traverse the entire table to find matching rows. b.index: Full Index Scan. The difference between index and ALL is that the index type only traverses the index tree. c. range: Index range scan. The scan of the index starts at a certain point and returns rows that match the range. Obvious index range scans are queries with between or with <,> in the where clause. When mysql uses an index to find a series of values, such as IN() and OR lists, the range (range scan) will also be displayed. Of course, there is a difference in performance. d. ref_or_null: The connection type is like ref, but MySQL can specifically search for rows containing NULL values. e. index_merge: This connection type indicates that the index merge optimization method is used. f. unique_subquery: This type replaces the ref of the IN subquery of the following form: value IN (SELECT primary_key FROM single_table WHERE some_expr). unique_subquery is an index search function that can completely replace subqueries and is more efficient. g. index_subquery: This connection type is similar to unique_subquery. You can replace the IN subquery, but it is only suitable for non-unique indexes in the following forms of subquery: value IN (SELECT key_column FROM single_table WHERE some_expr) h.ref: that is, the linker cannot get only one record based on the key value, and the index is the most The left prefix or index is not a primary key or unique index. This is a good connection type when only a few matching records are queried based on the key value. i. eq_ref: Similar to ref, the difference is that the index used is a unique index. For each index key value, there is only one record in the table that matches. In simple terms, the primary key or unique key is used as the association condition in the multi-table connection. j. const, system: When MySQL optimizes a certain part of the query and converts it to a constant, these types of access are used. If you put the primary key in the where list, MySQL can convert the query into a constant. Note: system is a special case of the const type. When the query table has only one row, use system k.NULL: MySQL decomposes the statement during the optimization process, and does not even need to access the table or index during execution, for example, select the smallest from an index column The value can be searched through a separate index.
5. Possible_keys: The column indicates which index MySQL can use to find rows in the table
6. Key: Shows the key (index) that MySQL actually decides to use. If no index is selected, the key is NULL
7. key_len: Shows the key length that MySQL decides to use. If the key is NULL, the length is NULL. The length of the index used. Without loss of accuracy, the shorter the length, the better
8. ref: Shows which column or constant is used together with the key to select rows from the table.
9, rows: Shows the number of rows that MySQL thinks it must check when executing a query.
10. Extra: Contains detailed information for MySQL to resolve queries and is also one of the key reference items.
2. Slow query log
When the query does not return a result for a certain period of time, it will be recorded in the slow query log. It is not turned on by default.
Turn it on manually when sampling. It can help us find out SQL statements that execute slowly. 1. Check whether the slow SQL log is enabled (on means it is enabled): show variables like'slow_query_log'; 2. Check how many seconds slower than the SQL execution will be recorded in the log file show variables like 'long_query_time'; 3. You can use fuzzy search to view all variable information containing query show variables like'%query%';
4. Whether to enable slow query log
slow_query_log=1
5. Specify the save path and file name, the default is the data file directory,
slow_query_log_file="bxg_mysql_slow.log" #
--Specify how many seconds to return the result of the query as a slow query long_query_time=1
6. Record all query statements that do not use the index
log_queries_not_using_indexes=1
7. Record those slow queries caused by searching more than 1000 times
min_examined_row_limit=1000
8. Record those slow optimize table, analyze table and alter table statements
log_slow_admin_statements=1
9. Record slow queries generated by Slave
log_slow_slave_statements=1
The command line modification configuration mode does not need to be restarted to take effect, but it will automatically become invalid after restart.
1、set global slow_query_log=1; 2、set global slow_query_log_file='bxg_mysql_slow.log'; 3、set long_query_time=1; 4、set global log_queries_not_using_indexes=1; 5、set global min_examined_row_limit=1000; 6、set global log_slow_admin_statements=1; 7、set global log_slow_slave_statements=1;
Three. Cache
There are two ways to close the cache, one is temporary and the other is permanent. Temporarily execute directly on the command line
set global query_cache_size=0; set global query_cache_type=0; --If the cache is turned off in the configuration file, the cache cannot be turned on by command
Permanently modify the configuration file my.cnf and add the following configuration.
query_cache_type=0 query_cache_size=0
In addition, we can also use the sql_no_cache keyword to directly disable the cache in the sql statement, and turn on the cache
In the case of we make some changes to the sql statement
Select sql_no_cache count( ) from pythonlearn.lianjia; - Do not cache Select sql_cache count( ) from pythonlearn.lianjia; - Cache (can also be omitted, the default cache is already enabled)
4. Prepare test data
-- user table
CREATE TABLE person ( id bigint(20) unsigned NOT NULL, fname varchar(100) NOT NULL, lname varchar(100) NOT NULL, age tinyint(3) unsigned NOT NULL, sex tinyint(1) unsigned NOT NULL, PRIMARY KEY (id) ) ENGINE=INNODB DEFAULT CHARSET=utf8
--User Department Table
CREATE TABLE department ( id bigint(20) unsigned NOT NULL, department varchar(100) NOT NULL, PRIMARY KEY (id) ) ENGINE=INNODB DEFAULT CHARSET=utf8
- User address table
CREATE TABLE address ( id bigint(20) unsigned NOT NULL, address varchar(100) NOT NULL, PRIMARY KEY (id) ) ENGINE=INNODB DEFAULT CHARSET=utf8
- Create a stored procedure for adding test data in batches
DELIMITER $$ DROP PROCEDURE IF EXISTS generate$$ CREATE PROCEDURE generate(IN num INT) BEGIN DECLARE chars VARCHAR(100) DEFAULT 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789';
DECLARE fname VARCHAR(10) DEFAULT '';
DECLARE lname VARCHAR(25) DEFAULT '';
DECLARE id INT UNSIGNED;
DECLARE len INT;
SET id=1;
DELETE FROM person;
WHILE id <= num DO
SET len = FLOOR(1 + RAND()*10);
SET fname = '';
WHILE len > 0 DO
SET fname = CONCAT(fname,SUBSTRING(chars,FLOOR(1 + RAND()*62),1));
SET len = len - 1;
END WHILE;
SET len = FLOOR(1+RAND()*25);
SET lname = '';
WHILE len > 0 DO
SET lname = CONCAT(lname,SUBSTR(chars,FLOOR(1 + RAND()*62),1));
SET len = len - 1;
END WHILE;
INSERT INTO person VALUES (id,fname,lname, FLOOR(RAND()*100), FLOOR(RAND()*2));
SET id = id + 1;
END WHILE;END $$ DELIMITER ;
Execute stored procedure
- Stop the transaction SET autocommit = 0; - Call the stored procedure CALL generate(1000000); SET autocommit = 1; - Restart the transaction
5. Use of Profiling
If you want to optimize a Query, you need to know exactly where the performance bottleneck of this Query is, whether it consumes too much CPU calculations, or requires too many IO operations? If you want to be able to clearly understand this information, you can get it through the Query Profiler function. Query Profiler is a query diagnostic analysis tool that comes with MYSQL. It can analyze the performance bottleneck of a SQL statement. Usually we use the explain and slow query log that cannot be accurately analyzed, but the Query Profiler can locate the various resource consumption conditions of a SQL statement execution, such as CPU, IO, etc., and the time spent on the SQL execution Wait.
usage
(1) By executing the "set profiling" command, you can turn on and off the QueryProfiler function mysql> SET global profiling=on; (set profiling=1) (2) View related variables show VARIABLES like'%profiling%'; (3) Save settings The default number is 15 and the maximum is 100 mysql> set profiling_history_size=100; (4) After the Query Profiler function is turned on, MySQL will automatically record the profile information of all executed Query, and execute n Query as the test below. select * from person limit 10000,100; (3) Get the summary information of multiple Query profiles saved in the current system mysql> show profiles; (4) Get detailed profile information for a single Query. You can get the detailed profile information of a Query during the execution process according to the Query_ID in the profile information. For example, to view the detailed information of cpu and io, show profile cpu, block io for query 501; show profile ALL for query 501; ALL: display all information|BLOCK IO: block device IO input and output times|CONTEXT SWITCHES: context switch related overhead|CPU : User and system CPU usage | IPC: Displays the consumption related to sending and receiving messages | MEMORY: Memory consumption (this version is not currently implemented) |PAGE FAULTS: Displays the overhead related to major and minor page faults|SOURCE: Display and Source_function, Source_file,
Six. sql statement optimization
Common sql optimization suggestions
1. Avoid SELECT * The more data read from the database, the slower the query will become. And if your database server and
If the WEB server is two independent servers, this will also increase the load of network transmission. select * from person where lname='x8RJWmQX'; select id from person where lname='x8RJWmQX'; 2. Avoid using in the where clause! = or <> operator should try to avoid using in the where clause! = or <> operator, otherwise the engine gives up using the index and performs a full table scan. EXPLAIN select * from person where fname !='sss'; 3. Try to avoid full table scan to optimize the query, and try to avoid full table scan. First, consider establishing indexes on the columns involved in where and order by. 4. Use UNION instead of OR Use OR statement: select * from person where fname ='LVc1oJjd' or fname='bjRdlVo'; Use UNION statement, the returned result is the same as above, but faster: select * from person where fname ='LVc1oJjd' Union select * from person where fname='bjRdlVo'; Explain and analyze the two sql separately: The result of the OR statement and the result of the UNION statement. Let’s compare the important indicators and find that the main difference is type and ref Both of these. type shows the access type, which is a more important indicator. The resulting value from good to bad is: system> const> eq_ref> ref> fulltext> ref_or_null> index_merge> unique_subquery> index_subquery> The type value of range> index> ALL UNION statement is generally ref, and the type value of OR statement is range. It can be seen that this is a very obvious gap. The ref value of the UNION statement is const, and the type value of the OR statement is null. const means constant value reference, which is very fast. The difference between these two shows that UNION is better than OR, and it can be understood from our intuitive feeling that although both of these methods use indexes, UNION uses a clear value to search in the index, and the goal is very clear. , OR needs to compare two values, and the target is relatively fuzzy, so OR is behind in a trance. 5. Like statement avoids leading percent signs and leading percent signs will cause index failure select * from person where fname like'%LVc1o%'; follow the index select * from person where fname like'LVc1o%'; 6. Avoid Performing expression operations on fields in the where clause should try to avoid performing expression operations on fields in the where clause, which will cause the engine to abandon the use of indexes and perform full table scans. Such as: select id from t where num/2=100 should be changed to: select id from t where num=100*2 7. Avoid performing functional operations on fields in the where clause. Try to avoid performing field operations in the where clause. Function operation, which will cause the engine to give up using the index and perform a full table scan. Such as: select id from t where substring(name,1,3)='abc' should be changed to: select id from t where name like'abc%' 8. Try to use numeric fields as much as possible. Use numeric fields as much as possible. If fields that only contain numerical information, try not to design them as characters, which will reduce the performance of query and connection and increase Storage overhead. This is because the engine compares each character in the string one by one when processing queries and concatenations. For numeric types, only one comparison is sufficient.
9. Paging optimization of large data volume uses limit for paging, and the efficiency is low after turning to more than 10,000 pages. The reason is that the limit offset will be searched row by row, it is searched first and then skipped. select * from person limit 999900,100; - Slow, it takes more than 0.4 seconds a. From business logic optimization, it is not allowed to turn over 100 pages, for example, Baidu can generally turn to about 70 pages. b. Technical optimization method one select * from person where id>999900 limit 100; This is very fast, about 0.001s, because the id index is used but there are prerequisites for this use, the id is continuous, and the data in the middle cannot be deleted, otherwise the id of 999900 is not the 999900 record . c. Technical optimization method 2: If you must query with limit offset, use delayed association select id from person limit 999900 ,100; in this way, only the id column is queried, index coverage is achieved, and select p.* from person p inner join soon (select id from person limit 999900 ,100) as tmp on p.id=tmp.id; Get the detailed information of each record after paging through the internal connection