Once the amount of data in Mysql reaches a certain level, it will reach the performance bottleneck, and we all know that the solution is to add indexes, but as for why the query performance can be instantly improved after adding indexes? I think many people know it but don't know why. In fact, this is all related to the data structure of the index. Below we will gradually understand the principle of Mysql index acceleration query.
The index mentioned above is related to its data structure, so what data structure is it? We will design this data structure as ourselves, starting with the binary tree.
Binary tree
We all know that a binary tree can sort our data, and then speed up the search for the data we think of, because a binary tree is regular, the child node on the left must be less than or equal to its parent node, and the child node on the right must be greater than its parent node. Then this condition seems to be similar to the implementation of Mysql index, as shown below
The value corresponding to each index column will be added to the binary tree, and the position of this value corresponds to the disk address of the corresponding data row (similar to a key-value pair), when we want to find data with col2=89 (Select * from t where t..col2 = 89), mysql finds that the index is added to the condition column we are querying, then it will look for it in the index file, that is, look for the binary tree above, according to The rule of the binary tree, we only need to query twice (the index is also on the disk, that is, the disk I/O is done twice) to locate the query to 89, and get the disk address corresponding to 89.
You may think that such an index implementation is also perfect. In fact, this is far more than that, because although MySQL uses a tree data structure for the number of index pages, it is not a binary tree, because there is a serious binary tree. The disadvantage is that when the added data is sequential, it will become a linear table structure. As shown below
In this way, when we want to find col2=8, it is equivalent to querying all the index values of col2, which is a full table query. Therefore, the underlying design of such an index is obviously not applicable.
Red black tree
The red-black tree is a kind of balanced binary tree. Above we can see that if we use a binary tree as an index implementation, when the inserted data is sequentially increasing or decreasing, then the tree will become a linked list. We can consider using red-black trees to optimize and solve this problem
You can see that the red-black tree in the above figure will self-balance to prevent problems similar to ordinary binary trees. But the red-black tree also has an obvious disadvantage, that is, the height will become larger and larger as the amount of data increases. If the height of the red-black tree reaches 20, and the data we want to query is at the leaf node, then There will be 20 disk I/Os, which is very costly.
B-Tree
B-Tree may be able to solve the problem of the height of red-black trees. The characteristics of B-Tree are listed below
1. Define that any non-leaf node has at most M children, and M>2;
2. The number of children of the root node is [2, M];
3. The number of children of non-leaf nodes except the root node is [ M/2, M];
4. Each node stores at least M/2-1 (rounded up) and at most M-1 keywords; (at least 2 keywords)
5. Keywords of non-leaf nodes Number = Number of pointers to the son -1;
6. Key words of non-leaf nodes: K[1], K[2], …, K[M-1]; and K[i] <K[i +1];
7. Pointers of non-leaf nodes: P[1], P[2], …, P[M]; where P[1] points to a subtree whose key is less than K[1], P[M ] Point to the subtree whose key is greater than K[M-1], and other P[i] point to the subtree whose key belongs to (K[i-1], K[i]);
8. All leaf nodes are at the same level ；
See the picture below for details (taken from other blogs)
B-Tree reduces the height of the tree by storing multiple key values in one layer (one page) and reduces disk I/O by loading the index data of one page in the disk into the memory. When the search comes to a certain node (also called a page of data), mysql will load all the keys in this node from the disk to the memory, and then find the key in the memory through a binary query, if not found Just go to the child nodes that meet the range of the key to find, continue to load the child node data from the disk to the memory, and repeat the previous step until the key is found.
If you need to find 28 above, starting from the root node, 17<28<35, so get the pointer P2, find the disk according to the pointer P2, find the corresponding key of another page, load it into the memory to find 28, if you can’t find it because 26<28<30, get the pointer P2 of this page, go to the disk according to the pointer P2 to find the key of another page and load it into the memory, and find 28.
But B-Tree also has the disadvantage that each node stores key + specific data, and the degree (a page of data) involves IO reading, so if the degree is designed to be large, it will affect the efficiency of IO reading. So mysql internally controls the size of each page, we can check it through SHOW GLOBAL STATUS LIKE'Innodb_page_size'
We can see that the size of each page set by mysql is 16384b=16kb, which means that the maximum index value from the disk can only be load each time
16kb, due to this limitation and each page of the B-Tree stores key+data, if the data is larger, then the key stored on each page will become less, and finally the depth will be uncontrollable. So mysql has improved the B-Tree.
B+Tree
The bottom layer of mysql index is maintained by B+Tree. The biggest difference between B+Tree and B-Tree is that the data of B+Tree is all placed in leaf nodes, and non-leaf nodes are not responsible for storing data, which means that the leaf nodes store the data of our entire table or the corresponding Disk address (MyISAM stores the disk address of the row corresponding to the index value, and InnoDB directly stores the table data, as described in detail below), we can find all the data by traversing the leaf nodes. We mentioned above that due to the size limit of the degree (one page), storing data in the degree will reduce the number of key storage, and B+Tree puts all the data on the leaf nodes to solve this problem, which can increase The number of keys stored in the degree reduces the height of the tree.
The difference between the index storage structure in MyISAM and InnoDB engines
We have analyzed how to use B+Tree to handle indexes in Mysql, so for different storage engines, Mysql's index storage structure is also different. Let's specifically talk about the two commonly used indexes in MyISAM and InnoDB. What is the storage structure under the storage engine?
Let's talk about MyISAM first. Let's look at how the table of the MyISAM storage engine is stored on the disk.
For a table of MyISAM storage engine, its composition is mainly composed of three file formats, namely frm, MYD, and MYI. frm stores the structure information of the table, MYD stores the table data, and MYI stores the table index.
Then, for the MyISAM table, the process of querying data is similar to the following figure:
If we select * from t t.Col1 = 15, when we execute this statement, Mysql will find that Col is indexed, then it will go to the MYI file of the corresponding table to find the B+Tree that maintains the index value , And then find the corresponding key value in B+Tree through the method we mentioned above, get the corresponding value value and then go to the table corresponding MYD file to check the data of a certain row. Therefore, the MyISAM index file and the data file are separated. Because of this, the indexes created by the tables of the MyISAM storage engine are all non-clustered indexes .
Looking at InnoDB again, the table created by the InnoDB storage engine is mainly composed of two file types, frm, ibd, frm storage table structure information, ibd stores index + table data. Obviously the biggest difference from MyISAM is that the index and table data are placed in the same file, as shown in the following figure (the main index is organized into data):
It can be found that the InnoDB table finds all the values of other fields in the row of the index value through the index . Under normal circumstances, the primary key of the table under InnoDB will create a clustered index (also known as the primary index) by default, and a table can only allow the existence of a clustered index, because once the data is stored, there can only be one order. (For InnoDB, the main index, primary key, and clustered index mentioned below are all in one way of saying) InnoDB table data is organized by the primary index, creating a primary key also creates a primary index by default, but InnoDB It is not mandatory to have a primary key, so there is a question here, doesn't it mean that the data is organized by the primary index? If there is no primary key, how is the data organized? In fact, if we do not create a primary key under the InnoDB table, mysql will choose the first unique index that does not contain NULL values as the primary key index. If there is no unique index, then mysql will choose the built-in 6-byte ROWID as the hidden index. In this way, with the clustered index, you can organize the data of the table.
The above also said that a table can only have one clustered index and the data storage order can only have one kind, then there is another question here, the above B+Tree is organized according to the clustered index, then set up other secondary indexes (secondary index It is the same way as the secondary index) How to organize the B+Tree of the index to quickly find the data? First of all, InnoDB and MyISAM both have a primary index and a secondary index. For InnoDB, the B+Tree of its primary index organizes data as shown in the figure above, while the secondary index is different. Its leaves The node stores not the data, but the stored key field + primary key value, which means that if you go according to the secondary index, you will eventually go to the leaf node of B+Tree to get the value of the primary key and then go to the B of the primary index. +Tree to find the corresponding data, as shown below:
Summary: Whether the primary index or the secondary index of MyISAM B+Tree leaf nodes are pointers to the data on the disk, it can be seen that the primary index in MyISAM is not different from the secondary index, the primary index is just to ensure This field is the only one that is not empty. There is no need to set the primary key in MyISAM, which is no different from B+Tree after the primary key is set; InnoDB can directly find the data through the B+Tree of the primary index, because the data is linked to the B of the primary index. Below the leaf node of +Tree, and the leaf node of B+Tree of the secondary index stores the index field +key value, get the key value and then go to the B+Tree of the main index to finally get the corresponding data.
Interview question: Why do InnoDB tables have to have primary keys, and it is recommended to use integer auto-increment primary keys?
There are three small questions about this question:
1. Why is a primary key necessary?
The answer to this question is as above.
2. Is it recommended to use an integer primary key?
The rules of B+Tree searching for data have also been mentioned above. The range of the data to be searched is constantly shortened by judgment until the data is found, because every page of B+Tree is ordered, and the value size is required for each page. The judgment then finds the next page, and if the primary key is an integer, the size comparison between direct numbers will be faster than the size between strings, which can shorten the time for determining the size of each page.
3. Auto-incremented primary key?
There is a two-way pointer between each leaf node of B+Tree to point to each other, and each value of the leaf node is increasing from left to right, then if the primary key is self-increasing, each value will be automatically Add to the rightmost node or create a new node, and if the primary key is not self-incrementing, the value will be added to the middle leaf node, this time the displacement of other values will be generated. This efficiency comparison is directly added. Much lower.