MySQL8 only supported one join algorithm - nested loop before, and a new algorithm hash join was introduced in MySQL8, which is more efficient than nested loop. (There will be time to introduce this join algorithm later)
1. MySQL driver table and driven table and join optimization
First understand which table is the driving table (also called the outer table) and which table is the driven table (also called the inner table) during the join connection:
- When using left join, the left table is the driving table and the right table is the driven table
- When using right join, the right table is the driving table, and the left table is the driving table
- When using join, the mysql optimizer will choose a table with a relatively small amount of data as the driving table, and a large table as the driven table
1.1) How to select the driving table and the driven table for the join query
In SQL optimization, large tables are always driven by small tables. For example: A is a small table and B is a large table. When using left join, it should be written as follows:
select * from A a left join B b on a.code=b.codeIn this way, table A is the driving table, and table B is the driven table.
1) The meaning of the drive table:
In nested loop join and hash join, it is used to obtain data first, and based on the data in this table, gradually obtain data in other tables, until the first table that finally finds all the data that meets the conditions is called drive table.
The driving table is not necessarily a table, but may be a data set, that is, a sub-collection is composed of data rows that meet the conditions in a certain table, and then this sub-collection is used as a data source for connecting other tables. This sub-collection is the real driving table. Sometimes for the sake of brevity, the table that first obtains the sub-collection according to the conditions is directly called the driving table.
If there are three or more tables, the join algorithm will be used to obtain the result sets of the first and second tables, and the result sets will be used as the outer data, and the result sets will be traversed to query data in the third table.
2) Small table as driving table:
We often say that the driving table must be a small table, which means that the sub-set obtained according to the conditions must be small, not that the entity table itself must be small. If the sub-set obtained by a large table is small, the large table can also be referred to as drive table. because:
小表驱动大表：需要通过140多次的扫描
for(140条){
for(20万条){
}
}
大表驱动小表：要通过20万次的扫描
for(20万条){
for(140条){
}
}So it can also be concluded that if the amount of data in table A and table B is about the same, it doesn't matter who you choose as the driving table.
Look at an example: more than 140 pieces of data in table A, and about 200,000 pieces of data in table B
select * from A a left join B b on a.code=b.code
执行时间：7.5s
select * from B b left join A a on a.code=b.code
执行时间：19s3) View who is the driver table through explain:
The EXPLAIN analysis can be used to determine who is the driving table in SQL. The table in the first row analyzed by the EXPLAIN statement is the driving table.
1.2) The driving table and driven table indexes are used:
The join query has an index condition:
- The driver table has an index and will not use the index
- The driven table will use the index to create an index
In the case of driving a large table with a small table, indexing the large table will greatly improve the execution speed.
Test 1: Create indexes for table A and table B
分析：EXPLAIN select * from A a left join B b on a.code=b.code
Only table B code uses the index
Test 2: What happens if only the code in table A is indexed?
In this case, the A table index is invalid.
in conclusion:
- Drive big tables with small tables
- Create an index on the driven table
2、Nested-Loop Join
Before MySQL 8.0, only one JOIN algorithm, Nested-Loop Join (nested loop link), was supported. There are many variants of Nested-Loop Join, which can help MySQL perform JOIN operations more efficiently.
2.1) Simple Nested-Loop Join (simple nested loop connection)
To put it simply, the nested loop join algorithm is a double-layer for loop. By looping through the row data of the outer table and comparing it with all the row data of the inner table one by one to obtain the result, the pseudo code is as follows:
select * from user tb1 left join level tb2 on tb1.id=tb2.user_id
for (user_table_row ur : user_table) {
for (level_table_row lr : level_table) {
if (ur.id == lr.user_id)) {
//返回匹配成功的数据
}
}
}Features:
Nested-Loop Join is simple, rough and easy to understand. It is to obtain results by comparing data in a double-layer loop, but this algorithm is obviously too rough. If each table has 10,000 pieces of data, then the number of data comparisons = 10,000 * 1 Ten thousand = 100 million times, obviously this kind of query efficiency will be very slow.
Of course, mysql will definitely not join tables so rudely, so there are two optimization algorithms for Nested-Loop Join. When executing a join query, mysql will choose one of the two optimal join optimization algorithms according to the situation. A join query is performed.
2.2) Index Nested-Loop Join (index nested loop connection)
The optimization idea of Index Nested-Loop Join is mainly to reduce the matching times of inner table data. Simply put, Index Nested-Loop Join is to directly match the inner table index through the outer table matching conditions to avoid matching with inner table data. This greatly reduces the number of matches in the inner table, from the original number of matches = the number of rows in the outer table * the number of rows in the inner table to the number of rows in the outer table * The height of the inner table index greatly improves the performance of join.
select * from user tb1 left join level tb2 on tb1.id=tb2.user_id
# 伪代码
for (user_table_row ur : user_table) {
lookup level_user_id_index {
if (ur.id == lr.user_id)) {
//返回匹配成功的数据
}
}
}Note: The premise of using the Index Nested-Loop Join algorithm is that the matched fields must have an index established on the driven table (inner table).
2.3) Batched Key Access join (referred to as BKA)
There is a problem in the INLJ algorithm described above. If the index associated with the driven table is an auxiliary index, and the query field cannot be covered by the index, then a table return operation is required when assembling the data. However, if every matching record is returned to the table, the efficiency is definitely not high. Although the primary key index can be used in the table return, because the ids here are not necessarily in order, it is also a random scattered read . For this situation, MySQL provides an optimization measure, providing an algorithm called Batched Key Access join, that is, a batch primary key access join algorithm.
1) BKA algorithm:
The principle of the BKA algorithm is to first query the qualified records in the driving table according to the conditions and store them in the join buffer, then obtain the index records of the driven table according to the index, and store them in the read_rnd_buffer. If one of the join buffer or read_rnd_buffer is full, then process the data in the buffer first: sort the driven table index records in read_rnd_buffer in ascending order according to the primary key, and then rely on this ordered record to go back to the table query, because the primary key index The records are sorted in ascending order according to the primary key, which can improve the efficiency of returning to the table. To enable the BKA algorithm, batched_key_access needs to be enabled.
Explanation: You can understand MRR first and then look at BKA and it will be very clear (below). BKA is disabled by default, and it needs to be executed to enable it.
mysql> SET optimizer_switch='mrr=on,mrr_cost_based=on,batched_key_access=on';Test it out here:
set optimizer_switch='batched_key_access=off';
explain select a.*,b.* from b join a on b.key = a.key
Batched_key_access is not enabled here. You can see that a is the driver table, and there is an index on the b.key field. The query fields use a.*, b.*. The required data needs to be returned to the table, but the INLJ algorithm is still used. Now turn on batched_key_access and try:
set optimizer_switch='batched_key_access=on';
explain select a.*,b.* from b join a on b.key = a.key
It can be seen that the BKA algorithm is used this time, and the join buffer is used. If we do not select b.* for the query field, only the b.id field is returned, so that the covering index can be used:
set optimizer_switch='batched_key_access=on';
explain select a.*,b.id from b join a on b.key = a.key 
It is found that the BKA algorithm is no longer used.
2.4) Block Nested-Loop Join (caching block nested loop connection, referred to as BNL)
If there is no available index for the associated field of the driven table, then the BNL algorithm must be used. This algorithm needs to use the join buffer like the BKA algorithm, but does not use read_rnd_buffer. Whether to choose BNL or BKA, the key point is whether the driven table has an available index, if not, use BNL directly, if there is an index, but need to return to the table, use BKA.
1) BNL algorithm:
First find out the qualified records in the driver table according to the conditions, and store them in the join buffer. If the join buffer can only store 100 records, but if the driver table meets the conditions, the result set exceeds 100 records, then only 100 records can be retrieved. It is called a batch, and then the driven table is scanned in full, each row of the driven table is matched with the records in the join buffer, and the matched records are put into the final result set. After the driven table is scanned, the join buffer is cleared, and the remaining records are repeatedly obtained from the driven table and stored in the join buffer, then the driven table is scanned in full, and the data is matched in the join buffer, and the cycle is repeated until the data is matched.
Explanation: Before MySQL 8.0.18, when the index of the driven table cannot be used during join, BNL will be used for join. In MySQL 8.0.18 and later, the Hash join optimization algorithm is used in this case. Starting from MySQL 8.0.20, MySQL no longer uses the BNL algorithm, and uses the Hash Join algorithm in all cases where BNL was used before.
# SQL
select * from R join S on R.r=S.s
# 伪代码
for each tuple r in R do # 扫描外表R
store used columns as p from R in join buffer # 将部分或者全部R记录保存到join buffer中，记为p
for each tuple s in S do # 扫描内表S
if p and s satisfy the join buffer # p与s满足join条件
then output the tuple <p,s> # 返回结果集Processing flow:
- Traverse all the records (all fields of the SQL query) in the driver table that meet the filter conditions, and put them into the join buffer
- If all the records in the driving table satisfy the conditions, put them into the join buffer, traverse all the records in the driven table, and obtain the result set of records satisfying the join conditions
- If the join buffer cannot store all the driver table records at one time, you can read the records to the join buffer in batches and repeat the second step
Note: Version 5.6 and later, the block_nested_loop parameter in the optimizer management parameter optimizer_switch controls whether BNL is used for the optimizer. By default, it is enabled. If it is set to off, the optimizer will choose the NLJ algorithm when selecting the join method.
mysql> show variables like 'optimizer_switch'\G;
block_nested_loop=on # BNL优化，默认打开
mysql> set optimizer_switch='block_nested_loop=on'; # 开启BNL2) BNL introduction:
BNL is mainly for optimization when the associated field of the driven table has no index. (When the driven table has no index or the index fails, INLJ cannot be used, and mysql will optimize it through BNL.) If in the EXPLAIN output, when the Extra value contains Using When the join buffer (Block Nested Loop) and the type value is ALL, index or range, it means that BNL is used; it also means that the table associated fields of the driven table lack indexes or the indexes are invalid and cannot effectively use the indexes.
The BNL algorithm is an optimization of the SNLJ algorithm, and the algorithm BNL can be promoted to INLJ for optimization. For SQL scan data, the number of scans for the driver table is 1, and the number of scans for the driven table is the record size of the drive table/join_buffer_size; for the scan records of SQL, the number of rows scanned by SQL = the number of records in the drive table + (driver table records/join_buffer_size) * number of driven table records.
To a certain extent, increasing the size of join_buffer_size can improve the execution efficiency of SQL using the BNL algorithm:
# 手动调整 join_buffer_size 的大小
mysql> show variables like '%join_buffer_size%';
+------------------+--------+
| Variable_name | Value |
+------------------+--------+
| join_buffer_size | 262144 |
+------------------+--------+
1 row in set (0.04 sec)
mysql> set join_buffer_size=1024;
Query OK, 0 rows affected (0.04 sec)
mysql> show variables like '%join_buffer_size%';
+------------------+-------+
| Variable_name | Value |
+------------------+-------+
| join_buffer_size | 1024 |
+------------------+-------+
1 row in set (0.03 sec)3) BNL features:
- The join_buffer_size variable determines the buffer size.
- The join buffer can only be used when the join type is all, index, or range.
- Each join that can be buffered will allocate a buffer, which means that a query may eventually use multiple join buffers.
- The first nonconst table will not allocate a join buffer, even if its scan type is all or index.
- The join buffer will be allocated before the join, and will be released after the query is executed.
- Only the columns participating in the join will be saved in the join buffer, not the entire data row.
3、MRR
MRR, the full name of which is "Multi-Range Read Optimization", is a new feature of MySQL 5.6. Simply put: MRR converts "random disk reads" into "sequential disk reads", thereby improving the performance of index queries.
3.1) How does mysql read data from disk?
Perform a range query:
mysql > explain select * from stu where age between 10 and 20;
+----+-------------+-------+-------+------+---------+------+------+-----------------------+
| id | select_type | table | type | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+----------------+------+------+-----------------------+
| 1 | SIMPLE | stu | range | age | 5 | NULL | 960 | Using index condition |
+----+-------------+-------+-------+----------------+------+------+-----------------------+When this sql is executed, MySQL will go to the disk to read data (assuming the data is not in the data buffer pool) as shown in the figure below:
The red line in the figure is the entire query process, and the blue line is the movement route of the disk.
This picture is drawn according to the index structure of Myisam, but it is also applicable to Innodb. For Myisam, the left side is the secondary index of the field age, and the right side is where the complete row data is stored. The same is true for Innodb. Innodb is a cluster index (cluster index), so you only need to replace the right side with a B+ tree with complete data on the leaf node.
The following Myisam engine analysis:
- First go to the secondary index on the left to find the first record that meets the conditions (in fact, each node is a page, and a page can have many records, here we assume that each page has only one record), then go to the right to read Get the full record of this data.
- After reading, go back to the left and continue to find the next record that meets the conditions. After finding it, read it on the right. At this time, you find that this piece of data keeps up with the previous piece of data. In terms of physical storage location, it is far away!
- What to do, there is no way, you can only let the disk and the magnetic head do mechanical movement together to read this data for you. The third and fourth are the same. Every time data is read, the disk and the magnetic head have to travel a long way.
Explanation: MySQL actually reads data in units of "pages". Here we assume that these pieces of data happen to be located on different pages. In addition, the idea of "page" is actually derived from the discontinuous memory management mechanism of the operating system, and there are similar "segments".
Note: A 10,000 RPM (Revolutions Per Minute) mechanical hard disk can perform about 167 disk reads per second, so in extreme cases, MySQL can only return you 167 pieces of data per second, which is not enough Count the CPU queuing time.
3.2) Sequential read
At this point you know how luxurious random disk access is, so it is obvious to convert random access to sequential access:
mysql > set optimizer_switch='mrr=on';
Query OK, 0 rows affected (0.06 sec)
mysql > explain select * from stu where age between 10 and 20;
+----+-------------+-------+-------+------+---------+------+------+----------------+
| id | select_type | table | type | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+------+---------+------+------+----------------+
| 1 | SIMPLE | tbl | range | age | 5 | NULL | 960 | ...; Using MRR |
+----+-------------+-------+-------+------+---------+------+------+----------------+We enabled MRR, re-executed the sql statement, and found that there was an additional "Using MRR" in Extra.
Now the MySQL query process will become like this:
- For Myisam, before going to the disk to get the complete data, it will be sorted according to the rowid, and then read the disk sequentially.
- For Innodb, it will be sorted according to the clustered index key value, and then read the clustered index sequentially.
Sequential reading brings several benefits:
- The disk and the magnetic head no longer need to do mechanical movement back and forth;
- Can make full use of disk pre-reading: for example, when the client requests data of one page, the data of the following pages can also be returned together and put into the data buffer pool, so that if the data of the next page happens to be needed next time, it will not Need to read to disk again. The theoretical basis for this is the well-known principle of locality in computer science: when a piece of data is used, nearby data is usually used immediately.
- In a query, the data of each page will only be read from the disk once: after MySQL reads the data of the page from the disk, it will put the data into the data buffer pool. Read from disk, read directly from memory. But if it is not sorted, maybe after you read the data on page 1, you will read the data on pages 2, 3, and 4, and then you will read the data on page 1. At this time, you find that page 1 The data of the page has been removed from the cache, so the data of the first page has to be read from the disk again. After converting to sequential reading, you will continue to use the data on page 1. At this time, according to the cache removal mechanism of MySQL, the cache of this page will not be invalidated until you use up the data on this page, because it is Sequential reading, in the remaining process of this query, you are sure that the data on this page will not be used again, and you can say goodbye to the data on this page.
Sequential reading optimizes index reading to the greatest extent through these three aspects. Don't forget that the index itself is to reduce disk IO and speed up queries, and MRR is to further amplify the role of index in reducing disk IO.
3.3) About the configuration of MRR
There are two configurations related to MRR :
- mrr: on/off
- mrr_cost_based: on/off
1）mrr=on/off
The switch used to turn on the MRR, if you don't turn it on, the MRR will never be used.
mysql > set optimizer_switch='mrr=on';
2）mrr_cost_based=on/off
It is used to tell the optimizer whether to use MRR based on the cost of using MRR, consider whether it is worthwhile to use MRR (cost-based choice), and decide whether to use MRR in a specific SQL statement. Obviously, MRR is not necessary for queries that only return one row of data, and if you set mrr_cost_based to off, the optimizer will use MRR entirely, which is very stupid in some cases, so it is recommended that this configuration is still Set to on, after all, the optimizer is correct in most cases.
3）read_rnd_buffer_size
There is also a configuration read_rnd_buffer_size , which is used to set the size of the memory used to sort the rowid. Obviously, MRR is essentially an algorithm that trades space for time. It is impossible for MySQL to give you unlimited memory for sorting. If the read_rnd_buffer is full, it will first sort the full rowids and read them from the disk, then clear them, and then continue to put rowids in them until the read_rnd_buffer reaches the read_rnd_buffer configuration again. upper limit, and so on.
In addition, Mariadb, one of MySQL's branches, has made a lot of optimizations to MySQL's MRR. Interested students can read the recommended reading at the end of the article.
3.4) Epilogue
You can also see that MRR has a lot to do with indexes.
Indexing is an optimization made by MySQL to query. It organizes the originally disorganized data into an orderly structure, so that full table scanning becomes a rule-based query. The MRR we are talking about is an optimization of MySQL for index-based queries, which can be said to be an optimization of optimization.
To optimize MySQL queries, you must first know the MySQL query process; and to optimize index queries, you must know the principles of MySQL indexes. As I said in " How to Learn MySQL ", to optimize a technology and learn how to tune it, you must first understand its principles. The two are different levels.
Recommended reading: