Analysis and Summary of MySQL Common Query Optimization Techniques

data preparation

CREATE TABLE staffs(
id INT PRIMARY KEY AUTO_INCREMENT,
`name` VARCHAR(24)NOT NULL DEFAULT'' COMMENT'姓名',
`age` INT NOT NULL DEFAULT 0 COMMENT'年龄',
`pos` VARCHAR(20) NOT NULL DEFAULT'' COMMENT'职位',
`add_time` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT'入职时间'
)CHARSET utf8 COMMENT'员工记录表';

insert into staffs(NAME,age,pos,add_time) values('z3',22,'manager',NOW());
insert into staffs(NAME,age,pos,add_time) values('July',23,'dev',NOW());
insert into staffs(NAME,age,pos,add_time) values('2000',23,'dev',NOW());


create index idx_staffs_nameAgePos on staffs(name,age,pos)

1. Compound indexes follow the best left prefix rule

If multiple cases are indexed, the leftmost prefix rule is obeyed. Refers to the query starting from the front leftmost column of the index and not skipping columns in the index.

In addition, for compound indexes, MySQL queries cannot use the columns on the right side of the range condition in the index, that is, the index columns after the range are invalid.

Example: A compound index uses a range condition while following the best left prefix rule.

EXPLAIN SELECT * FROM staffs where name = 'Alice' AND age > 21 AND pos = 'HR';

key_len=198, indicating that only the two indexes of name and age are valid, and the latter pos does not use the index.

Without the use of range queries, the execution plan of the compound query is as follows:

the query level reaches ref, and the name, age and pos index columns are used.

2. Do not do anything on indexed columns

Not doing any operation (calculation, function, (automatic or manual) type conversion) on the indexed column will cause the index to fail and go to a full table scan.

However, using the function operation on the value to be queried can use the index normally, such as using the CONCAT function to concatenate strings.

3. Try to achieve index coverage

The concept of covering index: also known as index covering, the data column of the select can only be obtained from the index, without reading the data row, MySQL can use the index to return the fields in the select list, without having to read the data file again according to the index.

4. Try not to use the unequal sign (!= or <>)

MySQL's inability to use indexes when using unequal (!= or <>) results in a full table scan. So try to avoid using the != or <> operators in the WHERE clause. MySQL uses indexes only for the following operators: <, <=, =, >, >=, BETWEEN, IN, and sometimes LIKE.

5. Try to avoid is null and is not null

The use of is null and is not null will make the index unusable. You should try to avoid NULL value judgment for fields in the WHERE clause. NULL is the default value when creating a table, but most of the time you should use NOT NULL, or use a special value, such as 0, -1 as the default value.

6. Pay attention to like fuzzy matching

Like starts with a wildcard ('$abc...') MySQL index failure will become a full table scan operation

, but there are always business scenarios that must start with a wildcard for fuzzy search. The corresponding solution is to use index coverage, that is, the query field can be Primary keys and other indexed fields, thus avoiding full table scans.

Simple example:

7. Try not to use or

You should try to avoid using OR to join conditions in the WHERE clause, otherwise it will cause the engine to give up using the index and perform a full table scan. You can use UNION to merge queries, such as:

select id from t where num=10 union all select id from t where num=20

Use union query:

8. Be aware of data type issues

It is necessary to pay attention to the data type problem. If the condition column is a string, the condition value should be the same as the attribute of the condition column to avoid index invalidation caused by implicit conversion.

-Notice:

1. If the condition column is a string type and the condition value is a number, the index will be invalid.

2. But if the condition column is an integer and the condition value is a string, the index will not be affected.
   
   age is an integer, but if the query condition column uses a string, the index will still be used.
   

9. order by keyword optimization

9.1 Basic principles

ORDER BY clause, try to use the Index method to sort, and avoid using the FileSort method to sort. The sorting operation is done on the index column as much as possible, and for compound indexes, the best left prefix rule for index building should be followed

9.2 Optimization strategy

1. Increase the setting of the sort_buffer_size parameter
2. Increase the setting of the max_length_for_sort_data parameter

9.3 Summary of order by use

MySQL has two sorts: file sort or scan-ordered index sort. MySQL can use the same index for sorting and querying.

10. Group by keyword optimization

The essence of groupby is to sort first and then group, so pay attention to the same point as order by. When the index column cannot be used, increase the setting of the max_length_for_sort_data parameter + increase the setting of the sort_buffer_size parameter. where is higher than having, and the conditions that can be written in where are not limited by having.

Counter example:

select * from order
group by user_id
having user_id <= 200;

This writing method first groups all orders according to user id, and then uses having to filter users whose user id is greater than or equal to 200. Grouping is a relatively time-consuming operation, and we should first narrow the data range in the where-limited conditions.

select * from order
where user_id <= 200
group by user_id;

11. Replace union with union all

After the union keyword is used in the union query, the deduplicated data can be obtained. Using the union all keyword, you can get all data, including duplicate data. The process of deduplication needs to be traversed, sorted and compared, which is more time-consuming than union all and consumes more CPU resources.

Unless there are some special scenarios, such as after union all, duplicate data appears in the result set, and duplicate data is not allowed in business scenarios, then union can be used.

12. Incremental query

In normal work, we often need to traverse the database to modify some data. When the amount of data to be searched is large, we can sort by id and time, query only one batch of data at a time, and save the largest id of this query. and time, reserved for the next query.
Counter example:

select * from order
where （查询条件）

Normal example:

select * from order
where id>#{lastId} and create_time >= #{lastCreateTime} （and 其他条件)
limit 100;

13. The number of indexes should not exceed 5 as much as possible

Indexes can significantly improve the performance of querying SQL, but the more the number of indexes, the better. Because when new data is added to the table, an index needs to be created for it at the same time, and the index requires additional storage space, which is accompanied by a certain performance consumption.
MySQL uses the structure of the B+ tree to save the index. During the insert, update and delete operations, the B+ tree index needs to be updated. If there are too many indexes, it will consume a lot of extra performance.

Alibaba's developer manual stipulates that the number of indexes in a single table should be controlled within 5 as much as possible, and the number of fields in a single index should not exceed 5.

So how to optimize the number of indexes?

1. You can build a joint index, but don't build a single key index
2. Introduce databases or search engines such as Elastic Seach, HBase or MongoDB to realize some query functions and reduce the pressure on MySQL

Summarize

Finally, with the composite index as the column, the case of whether the index is used in the specific query conditions is as follows:

The formula to help memory:

Expansion:
Detailed performance optimization strategy: 52 SQL statement performance optimization strategy
https://juejin.cn/post/7028937747087753246