Join query and performance optimization

In actual production, questions about the use of join statements generally focus on the following two categories:

Our DBA does not allow the use of join, what is the problem with using join?
If there are two tables with different sizes to join, which table should be used as the driving table?

Take an example to say something:

CREATE TABLE `t2` (
  `id` int(11) NOT NULL,
  `a` int(11) DEFAULT NULL,
  `b` int(11) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `a` (`a`)
) ENGINE=InnoDB;

create table t1 like t2;

1000 rows of data are inserted into table t2, and 100 rows of data are inserted into table t1.

Index Nested-Loop Join

There is such a query: select * from t1 straight_join t2 on (t1.a=t2.a);

If you use the join statement directly, the MySQL optimizer may choose table t1 or t2 as the driving table, which will affect the execution process of our analysis of the SQL statement. Therefore, in order to facilitate the analysis of performance problems during execution, I switch to straight_join to let MySQL use a fixed connection method to execute the query, so that the optimizer will only join in the way we specify . In this statement, t1 is the driven table and t2 is the driven table.

As you can see, in this statement, there is an index on the field a of the driven table t2, and this index is used in the join process, so the execution flow of this statement is as follows:

Read a row of data R from the table t1, and retrieve the a field from the data row R to search in the table t2;
Take out the rows that meet the conditions in table t2 and form a row with R as part of the result set;
Repeat steps 1 and 2 until the end of the loop of table t1 ends.

In form, this process is similar to the nested query when we write the program, and the index of the driven table can be used, so we call it "Index Nested-Loop Join", or NLJ for short. Throughout the execution process, the total number of scan lines is 200.

Assuming that join is not used, then we can only use single table query:

1. Execute select * from t1 to find out all the data in table t1, there are 100 rows;

2. Loop through these 100 rows of data:

Get the value of field a $Ra from each row R;
执行select * from t2 where a=$R.a；
The returned result and R form a row of the result set.

In this query process, 200 rows were scanned, but a total of 101 select statements were executed, which was 100 more interactions than direct join. In addition, the client has to splice SQL statements and results by itself. Obviously, it is better to use join.

During the execution of this join statement, the driving table is a full table scan, and the driven table is a tree search.

Assuming that the number of rows in the driven table is M , each time you check a row of data in the driven table, you must first search index a, and then search the primary key index. The approximate complexity of searching a tree each time is the logarithm of M with the base of 2, recorded as log2M, so the time complexity of searching a row on the driven table is 2*log2M (normal index search + primary key index search).

Assuming that the number of rows in the driving table is N , the execution process must scan the driving table N rows, and then for each row, it matches once on the driven table. Therefore, the approximate complexity of the entire execution process is N + N*2*log2M. Obviously, N has a greater impact on the number of scan rows, so the small table should be used as the driving table .

Through the above analysis we have got two conclusions:

Using join statement, the performance is better than the performance of forcibly splitting into multiple single tables to execute SQL statements;
If you use the join statement, you need to make the small table the driving table.

However, you need to note that the premise of this conclusion is that "the index of the driven table can be used."

Simple Nested-Loop Join

We change the SQL statement to this: select * from t1 straight_join t2 on (t1.a=t2.b);

Since there is no index on field b of table t2, a full table scan is required every time t2 is matched. If you only look at the results, this algorithm is correct, and this algorithm also has a name called "Simple Nested-Loop Join".

In this way, this SQL request will scan table t2 up to 1,000 times, and scan a total of 100*1000=100,000 rows. If these are two big tables, this algorithm looks too "cumbersome".

Of course, MySQL does not use this Simple Nested-Loop Join algorithm, but uses another algorithm called "Block Nested-Loop Join", referred to as BNL.

Block Nested-Loop Join

At this time, there is no available index on the driven table, and the algorithm flow is like this:

Read the data of table t1 into the thread memory join_buffer. Since we write select * in this statement, we put the entire table t1 into memory;
Scan table t2, take out each row in table t2, compare it with the data in join_buffer, and return those that meet the join conditions as part of the result set.

It can be seen that in this process, a full table scan is performed on both tables t1 and t2, so the total number of scanned rows is 1100. Since join_buffer is organized in an unordered array, 100 judgments are required for each row in table t2. The total number of judgments required in the memory is: 100*1000=100,000 times.

Compared with the previous simple Nested-Loop Join algorithm, the 100,000 judgments of this algorithm are memory operations, which are much faster and have better performance.

Let's take a look, in this case, which table should be selected as the driving table. Assuming that the number of rows in the small table is N and the number of rows in the large table is M, then in this algorithm:

Both tables do a full table scan, so the total number of scanned rows is M+N;
The number of judgments in the memory is M*N.

It can be seen that there is no difference between M and N in these two calculations. Therefore, at this time, whether to choose a large table or a small table as the driving table, the execution time is the same.

The size of join_buffer is set by the parameter join_buffer_size, and the default value is 256k. If you can't put all the data in the table t1, the strategy is very simple, which is to put it in sections. I changed join_buffer_size to 1200 and then executed it. The execution process becomes:

Scan table t1, read data rows sequentially and put them into join_buffer, after putting the join_buffer on the 88th row, continue to step 2;
Scan table t2, take out each row in t2, compare it with the data in join_buffer, and return those that meet the join conditions as part of the result set;
Clear join_buffer;
Continue to scan table t1, sequentially read the last 12 rows of data into join_buffer, and continue to step 2.

This process reflects the origin of the "Block" in the algorithm name, which means "join by block" . It can be seen that at this time, because table t1 is divided into two times and put into join_buffer, table t2 will be scanned twice. Although the join_buffer is divided into two times, the number of times to determine the equivalence condition is still the same, still (88+12)*1000=100,000 times.

Let's look at the selection of the driver table in this case. Assuming that the number of data rows in the driven table is N, it needs to be divided into K segments to complete the algorithm flow, and the number of data rows in the driven table is M. Note that K here is not a constant. The larger N is, the larger K will be. Therefore, K is expressed as λ*N, and the value range of λ is (0,1). Therefore, in the execution of this algorithm: the number of scanning lines is N+λ*N*M; the memory is judged N*M times .

Obviously, the number of memory judgments is not affected by which table is selected as the driving table. Considering the number of scanning lines, when the size of M and N are determined, N is smaller, and the result of the whole calculation will be smaller. The small table should be used as the driving table .

We just said that the larger the N, the larger the number of segments K. Then, when N is fixed, the larger the join_buffer_size, the smaller the number of segments, and the fewer the full table scans of the driven table.

That's why, you may see some suggestions telling you that if your join statement is very slow, increase the join_buffer_size .

Answer the opening two questions

Can I use the join statement?

If the Index Nested-Loop Join algorithm can be used, which means that the index on the driven table can be used, it is actually no problem;
If you use the Block Nested-Loop Join algorithm, the number of scan lines will be too much. Especially for join operations on large tables, this may have to scan the driven table many times, which will consume a lot of system CPU resources . So try not to use this kind of join.

So when you decide whether to use the join statement, you just look at the explain result and see if the word "Block Nested Loop" appears in the Extra field .

If you want to use join, should you choose a large table as the driving table or a small table as the driving table?

If it is the Index Nested-Loop Join algorithm, a small table should be selected as the driving table;
If it is the Block Nested-Loop Join algorithm: when the join_buffer_size is large enough, it is the same; when the join_buffer_size is not large enough (this situation is more common), a small table should be selected as the driving table.

Therefore, the conclusion of this question is that small tables should always be used as driving tables .

How to judge who is the "small watch" ?

1. select * from t1 straight_join t2 on (t1.b=t2.b) where t2.id<=50;
select * from t2 straight_join t1 on (t1.b=t2.b) where t2.id<=50;

Note that in order to prevent the driven tables of the two statements from using indexes, the join fields use the field b without index. But if the second statement is used, join_buffer only needs to be placed in the first 50 lines of t2, which is obviously better. So here, "the first 50 rows of t2" is the relatively small table, that is, the "small table".

2. select t1.b,t2.* from t1 straight_join t2 on (t1.b=t2.b) where t2.id<=100;
select t1.b,t2.* from t2 straight_join t1 on (t1.b=t2.b) where t2.id<=100;

In this example, both tables t1 and t2 have only 100 rows participating in the join. However, the data put into join_buffer in these two statements is different each time:

Table t1 only checks field b, so if t1 is put in join_buffer, only the value of b needs to be put in join_buffer;
Table t2 needs to check all the fields, so if you put table t2 in join_buffer, you need to put three fields id, a and b.

Here, we should choose table t1 as the driving table, "only one column of table t1 participating in the join" is the relatively small table.

Therefore, to be more precise, when deciding which table to be the driving table, the two tables should be filtered according to their respective conditions. After the filtering is completed, the total data volume of each field participating in the join is calculated, and the table with the smaller data volume is calculated. It is the "small table", which should be used as the driving table.

MRR optimization and BKA optimization

1. Multi-Range Read optimization (MRR). The main purpose of this optimization is to use sequential disk reading as much as possible.

Because most of the data is inserted in the increasing order of the primary key, we can think that if the query is in the increasing order of the primary key, the disk read is closer to the sequential read, which can improve the read performance.

select * from t1 where a>=1 and a<=100; a is a normal index. The execution flow of the statement becomes this:

According to index a, locate the record that meets the condition, and put the id value into read_rnd_buffer;
Sort the id in read_rnd_buffer in ascending order;
After sorting the id array, check records in the primary key id index in turn, and return as the result.

Here, the size of read_rnd_buffer is controlled by the read_rnd_buffer_size parameter. If the read_rnd_buffer is full in step 1, steps 2 and 3 will be executed first, and then read_rnd_buffer will be cleared. Then continue to find the next record of index a, and continue to loop.

Another thing to note is that if you want to use MRR optimization steadily, you need to set set optimizer_switch="mrr_cost_based=off" (the official document says that it is the current optimizer strategy. When judging the consumption, you will be more inclined to not Use MRR).

If from the explain result, we can see that the Extra field has more Using MRR, which means that MRR optimization is used.

Summary: The core that MRR can improve performance is that this query statement is a range query or a multi-value query on index a, which can get enough primary key ids, so that after sorting, go to the primary key index to check the data. Reflects the advantage of "sequence".

2. MySQL started to introduce the Batched Key Access (BKA) algorithm after version 5.6. This BKA algorithm is actually an optimization of the NLJ algorithm.

Because, the logic executed by the NLJ algorithm is: fetch values row by row from the driven table, and then join the driven table. In other words, for the driven table, a value is matched every time. At this time, the advantages of MRR are not used. In addition, we know that the role of join_buffer in the BNL algorithm is to temporarily store the data of the drive table. But it is not useful in the NLJ algorithm. Then, we can just reuse join_buffer into the BKA algorithm.

If you want to use the BKA optimization algorithm, you need to set before executing the SQL statement:

set optimizer_switch='mrr=on,mrr_cost_based=off,batched_key_access=on';

Among them, the function of the first two parameters is to enable MRR. The reason for this is that the optimization of the BKA algorithm depends on MRR.

Performance issues and optimization of BNL algorithm

When using the Block Nested-Loop Join (BNL) algorithm, the driven table may be scanned multiple times. If the driven table is a large cold data table, in addition to causing high IO pressure, what impact will it have on the system?

As mentioned earlier, InnoDB optimized the LRU algorithm of Bufffer Pool. However, if a join statement using the BNL algorithm scans a cold table multiple times, and the statement execution time exceeds 1 second, the data pages of the cold table will be moved to the head of the LRU linked list when the cold table is scanned again. This situation corresponds to the situation where the data volume of the cold table is less than 3/8 of the entire Buffer Pool and can be completely placed in the old area.

If the cold table is very large, another situation will occur: the data page normally accessed by the business has no chance to enter the young area. Because the normally accessed data page needs to be accessed again after 1 second to enter the young area. However, because our join statement reads the disk in a loop and eliminates memory pages, the data pages that enter the old area are likely to be eliminated within 1 second. In this way, the data pages in the young area of the Buffer Pool of this MySQL instance are not reasonably eliminated during this period.

Although the large table join operation has an impact on IO, the impact on IO will end after the statement is executed. However, the impact on the Buffer Pool is continuous, requiring subsequent query requests to slowly restore the memory hit rate.

In summary, the impact of the BNL algorithm on the system mainly includes three aspects:

The driven table may be scanned multiple times, occupying disk IO resources;
To determine the join condition, you need to perform M*N comparisons (M and N are the number of rows in the two tables respectively). If it is a large table, it will take up a lot of CPU resources;
It may cause the hot data of the Buffer Pool to be eliminated and affect the memory hit rate.

Before we execute the statement, we need to confirm whether to use the BNL algorithm through theoretical analysis and view the explain results. If it is confirmed that the optimizer will use the BNL algorithm, it needs to be optimized. The common optimization method is to add an index to the join field of the driven table to convert the BNL algorithm to the BKA algorithm.

If this statement is a low-frequency SQL statement at the same time, it would be wasteful to create an index on the field b of table t2 for this statement. However, if you use the BNL algorithm to join, as mentioned above, the BNL algorithm needs to be judged in memory, which will increase sharply with the large table, and the time-consuming is too slow.

At this time, we can consider using temporary tables. The general idea of using temporary tables is:

First put the data that meets the conditions in the large table in the temporary table;
In order to let join use the BKA algorithm, add indexes to the fields of the temporary table;
Let the driving table and the temporary table do a join operation.

This reduces the number of memory judgments while scanning a large table.

In general, whether adding an index on the original table or using an indexed temporary table, our idea is to enable the join statement to use the index on the driven table to trigger the BKA algorithm and improve query performance.

Extension 1-hash join

If what is maintained in join_buffer is not an unordered array but a hash table, then a large number of judgments become a small number of hash lookups, and the execution speed of the entire statement is much faster. Since the current version of MySQL does not support hash join, it can be simulated by the application side. Theoretically, the effect is better than the temporary table solution.

The implementation process is roughly as follows:

select * from t1; Get all the row data of small table t1, and store a hash structure on the business end, such as data structures such as set in C++ and array in PHP.
select * from t2 where ... Get multiple rows of data in table t2 that meet the conditions. Take these multiple rows of data line by line to the business end, and search for matching data in the hash structure data table. The row of data that meets the matching conditions is regarded as a row of the result set.

Extension 2-multi-table join

如：select * from t1 join t2 on (t1.a=t2.a) join t3 on (t2.b=t3.b) where t1.c>=X and t2.c>=Y and t3.c>=Z;

If it is rewritten as straight_join, how to specify the connection order and how to create indexes for the three tables.

The first principle is to use the BKA algorithm as much as possible . It should be noted that when using the BKA algorithm, it is not "first calculate the result of the join of two tables, and then join the third table", but directly nest the query.

The specific implementation is: among the three conditions t1.c>=X, t2.c>=Y, t3.c>=Z, select the table with the least data after filtering as the first driving table . At this time, the following two situations may occur.

In the first case, if the table t1 or t3 is selected, the remaining part is fixed.

If the driving table is t1, the connection sequence is t1->t2->t3, and the index must be created on the field of the driven table, that is, t2.a and t3.b;
If the driving table is t3, the connection sequence is t3->t2->t1, and indexes need to be created on t2.b and t1.a.

At the same time, we also need to create an index on the field c of the first driving table.

The second case is that if the first selected driving table is table t2, the filtering effect of the other two conditions needs to be evaluated.

In short, the overall idea is to try to make the data set of the driving table participating in the join as small as possible, because in this way our driving table will be smaller.

Content source: Lin Xiaobin "45 Lectures on MySQL Actual Combat"