MySQL paging query optimization with sorting

1. Script

1.1 Script to create table

CREATE TABLE `student` (
  `id` int(11) NOT NULL AUTO_INCREMENT COMMENT '主键',
  `user_no` varchar(50) DEFAULT '' COMMENT '学号',
  `user_name` varchar(50) DEFAULT '' COMMENT '姓名',
  `score` decimal(10,2) DEFAULT NULL COMMENT '分数',
  `create_time` date DEFAULT NULL COMMENT '创建时间',
  `update_time` date DEFAULT NULL COMMENT '更新时间',
  `remark` varchar(200) DEFAULT '' COMMENT '备注',
  PRIMARY KEY (`id`),
  KEY `idx_score` (`score`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COMMENT='学生表';

1.2 Requirements

Sort the results in descending order, and query the fields student number (user_no), name (user_name), and score (score) to make a sorted paging query.

1.3 Stored procedures - automatically execute data

use test;
DROP PROCEDURE if EXISTS BatchInsert;
delimiter $$
CREATE PROCEDURE BatchInsert(IN initId INT, IN loop_counts INT)
BEGIN
    DECLARE Var INT;
    DECLARE ID INT;
    SET Var = 0;
    SET ID = initId;
    set autocommit=0; -- 关闭自动提交事务，提高插入效率
    WHILE Var < loop_counts DO
        INSERT INTO `student` (`user_no`,`user_name`,`score`,`create_time`,`update_time`,`remark`) 
        VALUES (CONCAT('学号',ID),CONCAT('姓名',ID), floor(1 + rand()*100) , 
				DATE_ADD('2020-1-01 11:29:00', INTERVAL ROUND(RAND() * 1000 + 1) DAY),
				DATE_ADD('2020-1-01 11:29:00', INTERVAL ROUND(RAND() * 1000 + 1) DAY),
				CONCAT('备注',ID));
        SET ID = ID + 1;
        SET Var = Var + 1;
    END WHILE;
    COMMIT;
END$$;

delimiter ;  
CALL BatchInsert(1, 2000000);  

2. SQL to be optimized

	#浅分页
	EXPLAIN
	select user_no, user_name, score from student order by score desc limit 5, 20;
	#深分页
	EXPLAIN
	select user_no, user_name, score from student order by score desc limit 80000, 20;

Disadvantages: No matter deep paging or shallow paging. The overall performance is relatively poor.
Analysis: 20 pieces of data are also queried. Because of the large offset, the number of record rows scanned by deep paging query is more than that of shallow paging, so the performance is relatively poor.

3. Optimization plan 1

Add order by sort field index script

#增加order by 排序字段索引脚本
ALTER TABLE student ADD index idx_score (score);

Advantages: The speed of shallow paging queries has been greatly improved.
Disadvantages: The speed of deep paging queries has not changed significantly.
Returning to the table requires costs, and sorting also requires costs. MySQL helped us optimize it, and we chose the best of the two.
Reasons for the slowness of deep paging queries: partial The cost of table return is small if the offset is small, and the cost of table return if the offset is large is high.
Score field index data structure

4. Optimization plan 2

Add joint indexes to order by and select fields

#给我们order by 和 select字段加上联合索引
ALTER TABLE student ADD index idx_score_name_no (score,user_name, user_no);

Advantages: The speed of deep paging queries has been significantly improved.
Disadvantages: If query fields are added due to changes in requirements, it will affect the performance of SQL. We cannot always change the joint index of our query fields.

5. Optimization plan 3

Add an index to the sorting field and manually return the table (need to delete the index in plan 2 to reduce the impact of other indexes)
Delete script

#给排序字段增加索引，并手动回表
drop index idx_score_name_no on student;

execute sql

EXPLAIN
select user_no, user_name, score from student t1 join
(select id from student order by score desc limit 80000,20) t2 on t1.id= t2.id;  

Advantages: The speed of deep paging query has been improved.
Disadvantages: It is not recommended when there are too many subquery result sets.

6. Optimization plan 4

Optimize according to our score index structure; the score field index is first sorted in ascending order of score, and then in ascending order of id when the score is the same. The front end of the paging query passes the id and score of the last row of data on the previous page
.

EXPLAIN
 SELECT
	id, user_no, user_name, score 
FROM
	student 
WHERE id < 30000000  AND score <= 100.00 
	ORDER BY score DESC  LIMIT 20

7. Summary

Paging query optimization with sorting

1. Shallow paging can add an index to the order by field
2. Deep paging can add joint indexes to order by and select fields.
3. You can force the index to be retrieved by manually returning the table.
4. Start from the business side to limit his paging queries or modify the front-end and back-end interactions (pass the id and score of the last piece of data on each page)