In the previous article, I introduced you two algorithms join statement, respectively Index Nested-Loop Join (NLJ) and Block Nested-Loop Join (BNL).
We found that when using NLJ algorithm, in fact, the effect is good, more convenient than the application layer into multiple statements and then stitching the query results, and the performance will not be bad.
However, BNL algorithm performance when large tables join more backward, compare the number of times equal to the product of the two tables involved in the join line, it consumes CPU resources.
Of course, these two algorithms as well as continue to optimize the space, we are today to talk about this topic.
For ease of analysis, I created two tables t1, t2, and you started to today's problems.
create table t1(id int primary key, a int, b int, index(a));
create table t2 like t1;
drop procedure idata;
delimiter ;;
create procedure idata()
begin
declare i int;
set i=1;
while(i<=1000)do
insert into t1 values(i, 1001-i, i);
set i=i+1;
end while;
set i=1;
while(i<=1000000)do
insert into t2 values(i, i, i);
set i=i+1;
end while;
end;;
delimiter ;
call idata();
复制代码For convenience of explanation later quantization, my table t1, the insertion of the 1000 rows, each row a = value of 1001-id. That is, a field in a table t1 in the reverse order. At the same time, I inserted one million rows of data in table t2.
Multi-Range Read optimization
Before optimization program introduced join statement, and I need to introduce you to a point of knowledge, namely: Multi-Range Read optimization (MRR). The main objective is to optimize the use of sequential read the disk.
In the fourth article, I introduce you to InnoDB's index structure, referred to the concept of "back to the table". Let's take a look at this concept. Refers back to the table, InnoDB found primary key value in the ordinary index id of a post, then in accordance with a primary key value of the id to the primary key index to check up the entire process line data.
Then, some students asked in the message area, the process is back to the table to check data line by line, or in bulk check data?
Let's take a look at this issue. Suppose, I execute this statement:
select * from t1 where a>=1 and a<=100;
复制代码Primary key index is a B + tree, this tree, each row of data can be found in accordance with a primary key id. Thus, the table must be back rows of the search for the primary key index, the basic process shown in Figure 1.
Figure 1 Basic flow back to the table
As if the value of a query, then ascending order, id value becomes random, then there will be a random access performance is relatively poor. Although the "rows check" this mechanism can not be changed, but adjust the order of the query, or to accelerate.
Because most of the data are in accordance with the primary key is inserted in increasing order to get, so we can assume that, if the query words in ascending order of the primary key, read the disk closer reading order can improve read performance.
This is the design ideas MRR optimization. In this case, the execution flow statement has become such:
- The index a, is positioned to satisfy the condition of the recording, in the id value into read_rnd_buffer;
- The read_rnd_buffer the id be in ascending order;
- id sorted array, sequentially id to the primary key index to search records, and returned as a result.
Here, the size is controlled by read_rnd_buffer_size read_rnd_buffer parameters. If step 1, read_rnd_buffer filled, it will perform the first complete steps 2 and 3, and then empty the read_rnd_buffer. After continuing to find a next record index and the cycle continues.
Also need to note is that if you want to optimize the stable use of MRR, then you need to set set optimizer_switch = "mrr_cost_based = off". (Official document to say, now that the optimizer strategy, determine when consumed, will be more inclined not to use MRR, the mrr_cost_based set to off, is fixed using the MRR.)
The following two figures is the use of the implementation process and explain the results of MRR optimization.
FIG 2 MRR execution flow
The results explain the flow of execution in FIG. 3 MRR
From explain the results of Figure 3, we can see the Extra field more Using MRR, MRR represents the spend optimization. And since we do sort by id in read_rnd_buffer in, the final result set is obtained in accordance with the primary key id in ascending order, that is the reverse order of rows in the results of Figure 1.
Here, we summarize briefly.
The core can improve the performance of MRR that this query to do on the index is a range of a query (that is to say, this is a multi-value query), you can get enough primary key id. After sorting through this, go to the primary key index to check data in order to demonstrate the advantages of "order" of the.
Batched Key Access
MRR understand the principles of performance, we can understand the MySQL 5.6 version began after the introduction of Batched Key Access (BKA) the algorithm. The BKA algorithm, in fact, NLJ optimized algorithm.
We look at the flow chart NLJ algorithm used in the previous article:
FIG 4 Index Nested-Loop Join flowchart
Logic algorithm executed NLJ are: from the driving table t1, the value of a taken line by line, to the drive table t2 do is join. That is, for table t2, the value every time a match. At this time, the advantages of MRR would be irrelevant.
Then how can a one-time pass more and more value to the table t2 it? The method is, in one time t1 from the table to take more lines out pass along the table t2.
That being the case, we put the data table t1 taken out part first into a temporary memory. This temporary memory is not others, it is join_buffer.
By the previous article, we know the role of join_buffer at BNL in the algorithm, it is staging data-driven table. But we did not use the algorithm in NLJ. So, we just can reuse join_buffer to BKA algorithm.
5, is a flow chart of the algorithm BKA NLJ above optimization algorithm.
FIG 5 Batched Key Access Process
Figure I put in join_buffer data is P1 ~ P100, represents the field will only take queries need. Of course, if all of the data does not fit join buffer P1 ~ P100, the line 100 will put the data into segments on the execution flow of FIG.
Well, the BKA algorithm in the end how to enable it?
If you are using BKA optimization algorithm, you need to execute SQL statements before, first set
set optimizer_switch='mrr=on,mrr_cost_based=off,batched_key_access=on';
复制代码Among them, the role of the first two parameters is to enable the MRR. The reason for this is that BKA optimization algorithm depends on the MRR.
BNL algorithm performance issues
Having NLJ optimization algorithm, we will look at BNL optimization algorithm.
At the end of my previous articles, you leave thinking problem is that when the algorithm Block Nested-Loop Join (BNL), the table might be driven to do multiple scans. If this is the driving table is a large cold data table, in addition to outside pressure will lead to large IO, there will be any impact on the system?
In the first 33 articles, we talk LRU algorithm of InnoDB, when mentioned, due to the InnoDB Bufffer Pool to LRU algorithm is optimized, namely: first read from disk into memory data pages, will be the first on the old region. If after one second page of this data is no longer accessible, it will not be moved to the head of the LRU chain, so the impact on the Buffer Pool hit rate would be small.
However, if you use a join statement BNL algorithm, multiple scans of a cold table, but this time more than one second statement is executed, it will re-scan the table when cold, the cold table's data pages moved to the head of the LRU chain.
This case corresponds to, the amount of data is less than the entire cold Buffer Pool Table 3/8, the case can be fully inserted in the old region.
If this is a big cold table, there will be another situation: normal access to business data pages, lack of access to young area.
Due to the optimization mechanism, a normally accessed data pages, to enter the region young, after one second interval need to be accessed again. However, due to join us in the loop statement read out of the disk and memory pages, data pages into the old area, probably within one second to be eliminated. In this way, it will lead to the MySQL instance Buffer Pool During this time, the data page young area is not properly eliminated.
In other words, both of which will affect the normal operation of the Buffer Pool.
Although large table join operation IO impact, but at the end of the statement is executed, the impact on the IO will be over. However, the impact of the Buffer Pool is sustained, we need to rely on subsequent queries to slowly recover memory hit rate.
To reduce this effect, you can consider increasing the value of join_buffer_size reduce the number of driven table scan.
In other words, the impact on the system BNL algorithm mainly includes three aspects:
- Multiple scans may be driven table, taking up disk IO resources;
- Analyzing join conditions need to perform comparative M * N times (M, N are the number of rows of two tables), if a large table will take up a lot of CPU resources;
- It may result in data Buffer Pool heat is eliminated, affecting memory hit rate.
Before we execute the statement, required by means of theoretical analysis and view explain the results, confirm that you want to use BNL algorithm. If confirmed optimizer uses BNL algorithm, we need to do optimization. Common practice is to optimize, to join field is the driving table plus the indexes, the BNL algorithm turn into BKA algorithm.
Next, we look at the specific, this optimization how to do?
BNL turn BKA
In some cases, we can build the index to be driven directly on the table, then you can turn directly to the BKA algorithm.
But sometimes you do come across some cases not suitable for building an index on the table to be driven. For example, the following statement:
select * from t1 join t2 on (t1.b=t2.b) where t2.b>=1 and t2.b<=2000;
复制代码When we started the article, insert one million rows of data in table t2, but after filtration where conditions require the participation of only 2,000 lines of data join. If this statement is also a low frequency of SQL statement, then this statement creates an index on the field b table t2 is very wasted.
However, if BNL algorithm to join the case, the flow of execution of this statement is this:
- All fields of the table t1 is taken out, stored in join_buffer. This table is only 1000 lines, join_buffer_size default value is 256k, can be completely stored.
- Scan the table T2, the data taken with each row of data in comparison join_buffer, ● with not satisfied t1.b = t2.b, skip; ● If the condition t1.b = t2.b, then determining other conditions, i.e. meets t2.b in [1,2000] condition, if so, to return as part of the result set, otherwise skip.
I said in the previous article, for each row in the table t2 judge whether join when met, will need to traverse all the lines join_buffer. It is judged that the number of equivalent conditions is 1000 * 1 million = 10 million times, a lot of work in this judgment.
Figure 6 explain results
7 statement execution time
See, explain the results in Extra field shows the use of BNL algorithm. In my test environment, this statement needs to be performed 1 minute 11 seconds.
Create an index would be a waste of resources in the field b table t2, but does not create the index, then the equivalent of a conditional statement to determine one billion times, think it is a waste. Well, there is no way to do best of both worlds?
At this time, we can consider the use of temporary tables. The general idea is to use a temporary table:
- The table t2 satisfy the condition data in the temporary table tmp_t;
- In order to join BKA use algorithms to temporary table tmp_t field b plus an index;
- Let t1 table and tmp_t do join operations.
In this case, the wording corresponding SQL statement is as follows:
create temporary table temp_t(id int primary key, a int, b int, index(b))engine=innodb;
insert into temp_t select * from t2 where b>=1 and b<=2000;
select * from t1 join temp_t on (t1.b=temp_t.b);
复制代码Figure 8 is the implementation of the results of this sequence of statements.
FIG 8 is performed using the effect of temporary table
Can be seen, the sum of the whole process three statement execution time less than one second, compared to the preceding 1 minute 11 seconds, the performance has been greatly improved. Next, we look at consumption with this process:
- Execute insert statement temp_t table structure and insert data in the process, to table t2 do full table scan, the number of scanning lines here 100 million.
- After the join statement, scanning table t1, where the number of scanning lines is 1000; join the comparison process, we made 1,000 indexed queries. Compared to the join statement before optimization need to do a billion conditional, this optimization effect is obvious.
Overall, whether it is applied to the original table index, with the index or temporary tables, our idea is to let join statement can spend on the index driven table to trigger BKA algorithm to improve query performance.
Extended -hash join
See here you may find that the 10 billion times that the operation actually calculated above, looks a bit silly. If join_buffer which is not maintained by a disordered array, but a hash table, then it is not a billion judgment, but a million times hash lookup. In this case, the execution speed of the entire statement is much faster, right?
Indeed.
This, it is also one of the reasons MySQL optimizer and actuators has been criticized: do not support the hash join. And, MySQL official roadmap, also has yet to put this optimization rafts agenda.
In fact, this optimization idea, we can achieve in their own business side. Implementation process is as follows:
- All acquired data row 1000 of table t1, a hash stored in the business end of the structure, such as in C ++ set, PHP array such data structures.
- select * from t2 where b> = 1 and b <= 2000; 2000 acquires rows in table t2 satisfy the condition.
- This data line 2000, line by line to get the business end, the hash table data structure for a matching data. This data row matching condition is satisfied, it is set as a row of the result.
In theory, this process will be faster than the speed of execution of some of the temporary table program. If you are interested, you can verify it yourself.
summary
Today, I share with you the Index Nested-Loop Join (NLJ) and Block Nested-Loop Join (BNL) optimization method.
In these optimization methods:
- BKA is optimized MySQL has built-in support, it is recommended that you use the default;
- BNL low efficiency of the algorithm, I suggest you all try to turn into BKA algorithm. Optimization of direction is related to the driven field plus the table index;
- Improved program temporary table in advance to be able to filter out small data join statement, the effect is very good;
- MySQL does not support the current version of the hash join, but you can come out with their own simulation application side, in theory, the program is better than the temporary table.
Finally, I leave you with a question to think of it.
We are talking about join statement of these two articles, only related to the join of the two tables. Well, now there is a demand for three table join, assuming that these three tables table structure is as follows:
CREATE TABLE `t1` (
`id` int(11) NOT NULL,
`a` int(11) DEFAULT NULL,
`b` int(11) DEFAULT NULL,
`c` int(11) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB;
create table t2 like t1;
create table t3 like t2;
insert into ... // 初始化三张表的数据
复制代码We need to achieve the following statement of join logic:
select * from t1 join t2 on(t1.a=t2.a) join t3 on (t2.b=t3.b) where t1.c>=X and t2.c>=Y and t3.c>=Z;
复制代码Now in order to get the fastest execution speed, if let you design tables t1, t2, the index on t3, to support this join statement, which indexes you add it?
Meanwhile, if you want me to rewrite this statement with straight_join, with the index you create, you will need to arrange join order, what factors do you consider is the main?
You can write your programs and analysis in the comments section, I will end next article and you discuss this issue. Thank you for listening, you are welcome to send this share to more friends to read together.
On the issue of time
In the last article I left you last question has been answered in this article.
Reproduced in: https: //juejin.im/post/5d05eb7fe51d45777621bb78