InnoDB index rules in MySQL

This article is a summary of the column of "45 Lectures on MySQL Practice" by Lin Xiaobin.

General index

The appearance of the index is actually to improve the efficiency of data query, just like a book catalog. There are also many ways to implement indexing. Here is a brief introduction to several models of indexing:

• Hash table, based on hash table implementation, is suitable for scenarios where there is only equivalent query.
• Ordered arrays have excellent performance in equivalent query and range query scenarios, but the insertion cost is extremely high, and it is only suitable for static storage engines.
• The binary search tree is characterized by: the values ​​of all nodes in the left subtree of the parent node are less than the value of the parent node, and the values ​​of all nodes in the right subtree are greater than the value of the parent node. The query time complexity is O(log(N)), and the update time complexity is O(log(N)). Binary trees are the most efficient for searching, but in fact, most database storage does not use binary trees. The reason is that the index is not only stored in memory, but also written to disk.
• In a multi-branch tree, each node has multiple sons, and the size between the sons is guaranteed to increase from left to right. It can make a query read the disk as little as possible, that is, reduce the height of the tree and the number of random IOs. Take an integer field index of InnoDB as an example, N = 1200, the tree height is 4, you can store 1200^3 = 1.7 billion. Considering that the data block at the root of the tree is always in memory, the index of an integer field on a 1 billion-row table requires only 3 disk accesses to find a value. In fact, the second level of the tree has a high probability of being in memory, so the average number of accesses to the disk is even less.

In MySQL, indexes are implemented at the storage engine layer, so there is no uniform index standard, that is, indexes of different storage engines work differently. Even if multiple storage engines support the same type of index, the underlying implementation may be different. Among them, the InnoDB storage engine is the most widely used in MySQL databases.

InnoDB's index model

In InnoDB, tables are stored in the form of indexes according to the order of the primary key. The tables in this storage method are called index-organized tables . And because we mentioned earlier, InnoDB uses the B+ tree index model, so the data is stored in the B+ tree. Among them, each index corresponds to a B+ tree in InnoDB. Further, according to the content of the leaf node, the index type is divided into primary and non-primary key index key index.

• The leaf node of the primary key index stores the entire row of data. In InnoDB, the primary key index is also called a clustered index (clustered index).
• The content of the leaf node of the non-primary key index is the value of the primary key. In InnoDB, non-primary key indexes are also called secondary indexes.

So, what is the difference between a query based on a primary key index and an ordinary index?

• The primary key query method , that is, the judgment condition in the where clause is the primary key field, only the B+ tree of ID needs to be searched;
• Ordinary index query mode , that is, the judgment condition in the where clause is an ordinary index field, so you need to search the ordinary index tree first to get the value of the primary key, and then search the primary key index tree again. This process is called returning to the table .

Index maintenance

In order to maintain the orderliness of the index, the B+ tree needs to do necessary maintenance when inserting new values . You may need to move data logically to make room for new data. What's worse is that if the data page is full, according to the algorithm of the B+ tree, you need to apply for a new data page, and then move part of the data. This process is called page splitting . In this case, performance will naturally suffer. The page splitting operation also affects the utilization of data pages. The data that was originally placed on one page is now divided into two pages, and the overall space utilization is reduced by about 50%. Of course, where there is division, there is merger. When the utilization of two adjacent pages is reduced due to data deletion , the data pages will be merged. The process of merging can be considered the inverse process of the splitting process .

The difference between auto-incremented primary key and business field primary key:

The self-incrementing primary key refers to the primary key defined on the self-incrementing column, which is generally defined in the table building statement: NOT NULL PRIMARY KEY AUTO_INCREMENT. The data insertion mode of the self-incrementing primary key is in line with the incremental insertion scenario we mentioned earlier. Every time a new record is inserted, it is an append operation , it does not involve moving other records, and does not trigger the split of leaf nodes.

It is often not easy to ensure orderly insertion if fields with business logic are used as primary keys, so the cost of writing data is relatively high. In addition, we know that the smaller the primary key length, the smaller the leaf nodes of the ordinary index, and the smaller the space occupied by the ordinary index. Therefore, in terms of performance and storage space, auto-incrementing the primary key is often a more reasonable choice.

Are there any scenarios suitable for using business fields as primary keys directly?

There is only one index, and it is the only index. Since there are no other indexes, there is no need to consider the size of the leaf nodes of other indexes.

Covering index

Perform a select query, if the where clause condition field already includes the query field requirements, you can directly provide the query results without returning to the table, this is the covering index. Since a covering index can reduce the number of tree searches and significantly improve query performance , using a covering index is a common performance optimization method.

When using a covering index, a joint index is generally used. Although the maintenance of the index field is costly, the covering index is used here, and there is no need to go back to the table to check the entire row of records, reducing the execution time of the statement.

Leftmost prefix principle

In actual business, there is no need to design an index for every query. You can combine the joint index and use the "leftmost prefix" of the index to locate records. Index items are sorted according to the order of the fields that appear in the index definition.

If you are looking for all the people whose names are "Zhang" in the first word, the condition of your SQL statement is "where name like'Zhang%'". At this time, you can also use this index to find the first record that meets the condition, and then traverse backwards until the condition is not met.

Therefore, not just the full definition of the index, as long as the leftmost prefix is ​​satisfied, the index can be used to speed up retrieval. The leftmost prefix can be the leftmost N fields of the joint index , or the leftmost M characters of the string index .

When building a joint index, how to arrange the order of the fields in the index?

The evaluation criterion is the reusability of the index. Because the leftmost prefix can be supported, when the joint index (a, b) already exists, there is generally no need to create an index on a separately. Therefore, the first principle is that if one less index can be maintained by adjusting the order, then this order is often a priority. If there are only b statements in the query condition, the (a, b) joint index cannot be used. At this time, you have to maintain another index, which means you need to maintain both (a, b) and (b) at the same time Indexes.

At the same time, the principle to be considered is space. For example, the name and age fields, the name field is larger than the age field, then I suggest you create a joint index of (name, age) and a single field index of (age).

Index push down optimization

Take the joint index (name, age) as an example. If there is a demand now: Retrieve all the boys whose name is Zhang and the age is 10 years old in the table. So, the SQL statement is written like this:

mysql> select * from tuser where name like '张%' and age=10 and ismale=1;

Before MySQL 5.6, when searching the index tree, this statement could only use "Zhang" to find the first record that satisfies the conditions, and then start returning to the table one by one. Find the data row on the primary key index and compare the field values.

The index condition pushdown introduced by MySQL 5.6 can first make judgments on the fields contained in the index during the index traversal process , and directly filter out records that do not meet the conditions and reduce the number of return to the table. Here, InnoDB judges whether age is equal to 10 within the (name, age) index. For records that are not equal to 10, it directly judges and skips, and then returns to the table.

In summary, look at an example:

There is such a table, the table structure definition is similar to this:

CREATE TABLE `geek` (
  `a` int(11) NOT NULL,
  `b` int(11) NOT NULL,
  `c` int(11) NOT NULL,
  `d` int(11) NOT NULL,
  PRIMARY KEY (`a`,`b`),
  KEY `c` (`c`),
  KEY `ca` (`c`,`a`),
  KEY `cb` (`c`,`b`)
) ENGINE=InnoDB;

Is the table structure reasonable? How to modify if it is unreasonable?

(1) The clustered index organization order of primary keys a, b is equivalent to order by a, b, that is, first sort by a, then sort by b, and c is unordered.

(2) The organization of index ca is sorted by c first, then sorted by a, and the primary key b is recorded at the same time. In fact, the effect is the same as that of index c. Just leave one of the two.

(3) The organization of index cb is sorted by c first, then sorted by b, and the primary key a is recorded at the same time.

The choice between ordinary index and unique index

We can take a look at the execution process of the query statement and update statement of the two indexes .

Inquire:

select id from T where k=5 k is the index field

The process of searching on the index tree first starts from the root of the tree through the B+ tree, and searches for the leaf nodes by layer, and then it can be considered that the data page is located inside the data page through the dichotomy.

• For ordinary indexes, after finding the first record that meets the condition, you need to find the next record until the first record that does not meet the k=5 condition is encountered.
• For a unique index, because the index defines uniqueness, after finding the first record that meets the condition, it will stop searching.

The performance gap caused by this difference is actually minimal. the reason:

InnoDB data is read and written in units of data pages, instead of reading the record itself from the disk, but in units of pages, it is read into memory as a whole. In InnoDB, the size of each data page is 16KB by default. When the record with k=5 is found, the data page it is on is all in the memory. Then, for the ordinary index, the "find and judge the next record" operation that needs to be done more requires only one pointer search and one calculation. Of course, if the record k=5 happens to be the last record of the data page, then to remove the next record, the next data page must be read. This operation will be slightly more complicated. For integer fields, a data page can put nearly a thousand keys, so the probability of this situation will be very low. Therefore, when we calculate the average performance difference, we can still consider this operating cost to be negligible for the current CPU.

Update:

1. Change buffer. When a data page needs to be updated, if the data page is in memory, it is updated directly, and if the data page is not in memory, InnoDB will cache these update operations in the change buffer without affecting data consistency. In this way, there is no need to read this data page from the disk.

When the next query needs to access the data page, the data page is read into the memory , and then the operations related to this page in the change buffer are executed . In this way, the correctness of the data logic can be guaranteed. It should be noted that although the name is called change buffer, it is actually data that can be persisted. In other words, if the change buffer is copied in memory, it will also be written to disk .

The operation change buffer is applied to the original data page , get the latest result of the process is called merge. In addition to accessing this data page to trigger a merge, the system has background threads that will merge periodically . During the normal shutdown of the database, the merge operation is also performed.

The change buffer uses the memory in the buffer pool, so it cannot increase indefinitely. The size of the change buffer can be dynamically set by the parameter innodb_change_buffer_max_size . When this parameter is set to 50, it means that the size of the change buffer can only occupy up to 50% of the buffer pool.

Obviously, if the update operation can be recorded in the change buffer first to reduce random disk read IO, the execution speed of the statement will be significantly improved. Moreover, data reading into the memory needs to occupy the buffer pool, so this method can also avoid occupying memory and improve memory utilization.

2. For a unique index, all update operations must first determine whether this operation violates the uniqueness constraint. This must be determined by reading the data page into the memory. If all have been read into the memory, it will be faster to update the memory directly, so the unique index update cannot use the change buffer.

When the target page to be updated is in the memory , the difference between the ordinary index and the unique index on the performance of the update statement is just a judgment and only consumes a small amount of CPU time.

When the target page of the record to be updated is not in the memory , for the unique index, the data page needs to be read into the memory, it is judged that there is no conflict, the value is inserted, and the statement execution ends; for the ordinary index, the record will be updated In the change buffer, the statement execution ends.

3. In all scenarios of normal indexing, can the use of change buffer play an acceleration role?

Because the merge is the time when the data is actually updated, and the main purpose of the change buffer is to cache the recorded change actions, so before a data page is merged, the more changes recorded in the change buffer (that is, the more changes are required on this page) The more updates are made), the greater the benefits.

For businesses that write more and read less, the probability that the page will be accessed immediately after writing is relatively small, and the use of change buffer is the best. Commonly used in this business model are billing and log systems.

Conversely, assuming that the update mode of a business is to query immediately after writing, even if the conditions are met, the update will be recorded in the change buffer first, but the merge process will be triggered immediately because the data page will be accessed soon. In this way, the number of random access to IO will not decrease, but will increase the maintenance cost of the change buffer .

Conclusion :

Normal index and unique index are the same in terms of query capabilities, and the main consideration is the impact on update performance. Therefore, I suggest you try to choose ordinary index.

If all updates are immediately followed by queries for this record, then you should close the change buffer. In other cases, change buffer can improve update performance.

In actual use, you will find that the combined use of ordinary indexes and change buffers is very obvious for updating and optimizing tables with large amounts of data.

Actual inspection:

First of all, business correctness is the priority.

The premise of our article is to discuss performance issues under the condition that " business code has been guaranteed not to write duplicate data" . If the business cannot be guaranteed, or the business requires the database as a constraint, then there is no choice and a unique index must be created. The significance of this article is that if a large amount of data is inserted slowly and the memory hit rate is low, it can provide you with an additional troubleshooting idea.

Then, in some "archive library" scenarios, you can consider using ordinary indexes.

For example, online data only needs to be retained for half a year, and then historical data is stored in the archive library. At this time, archiving data is to ensure that there is no unique key conflict. To improve archiving efficiency, you can consider changing the unique index in the table to a normal index.

The difference between change buffer and redo log

We want to execute this insert statement on the table:

mysql> insert into t(id,k) values(id1,k1),(id2,k2);

Here, we assume the current state of the k index tree. After finding the position, the data page where k1 is located is in the memory (InnoDB buffer pool), and the data page where k2 is located is not in the memory. As shown in the figure is the update state diagram with change buffer.

                                      

Analyzing this update statement, you will find that it involves four parts: memory, redo log (ib_log_fileX), data table space (t.ibd), system table space (ibdata1).

(1) Page 1 in the memory, directly update the memory;

(2) Page 2 is not in the memory, just in the change buffer area of ​​the memory, record the message "I want to insert a line into Page 2";

(3) Record the above two actions in the redo log.

After doing the above, the transaction can be completed. Therefore, you will see that the cost of executing this update statement is very low, that is, two memories are written, and then one disk is written (the two operations are combined to write one disk), and it is written sequentially. At the same time, the two dotted arrows in the figure are background operations and do not affect the update response time .

We are now going to execute select * from t where k in (k1, k2). Here, I drew the flowchart of these two read requests. If the read statement occurs shortly after the update statement and the data in memory is still there, then the two read operations at this time have nothing to do with the system table space (ibdata1) and redo log (ib_log_fileX).

                                    

(1) When reading Page 1, directly return from memory.

(2) When you want to read Page 2, you need to read Page 2 from the disk into the memory, and then apply the operation log in the change buffer to generate a correct version and return the result.

If you want to simply compare the benefits of these two mechanisms in improving update performance, redo log mainly saves the IO consumption of random disk writes (converted to sequential writing) , while the main saving of change buffer is the IO of random read disks. Consumption .

Loss of change buffer:

When the machine is powered off and restarted, will the change buffer be lost? Loss of change buffer is not a trivial matter. If data is read from disk, there is no merge process, which is equivalent to data loss. Will this happen?

Although only the memory is updated, when the transaction is committed, we also record the change buffer operation in the redo log, so the change buffer can be retrieved when the crash is recovered.

The execution flow of merge is as follows:

 (1) Read data pages from disk to memory (old version data pages);

 (2) Find out the change buffer records of this data page (there may be more than one) from the change buffer, apply them in turn, and get the new version of the data page;

 (3) Write redo log, this redo log contains data changes and change buffer changes.

At this time, the data page and the disk location corresponding to the change buffer in the memory have not been modified, and they are dirty pages. Afterwards, each of them flushes their physical data back, which is another process.

The optimizer chooses the logic of the index

The purpose of the optimizer to choose an index is to find an optimal execution plan and execute the statement with the least cost. In the database, the number of scanned rows is one of the factors that affect the execution cost. The fewer rows scanned, the fewer times to access disk data and the less CPU resources are consumed. Of course, the number of scanned rows is not the only criterion. The optimizer will also make a comprehensive judgment based on factors such as whether to use temporary tables and whether to sort .

So, the question here is: how to judge the number of scan lines?

Before MySQL actually starts executing the statement, it can only estimate the number of records based on statistical information. This statistical information is the "discrimination" of the index. The more different values ​​on an index, the better the discrimination of the index. And the number of different values ​​on an index is called "cardinality". In other words, the larger the base, the better the discrimination of the index .

You can use the show index from t method to see the cardinality of an index. How does MySQL get the cardinality of the index?

Here is a brief introduction to the method of MySQL sampling statistics. Because the entire table is taken out for row statistics, although accurate results can be obtained, but the cost is too high, so you can only choose "sampling statistics".

When sampling statistics, InnoDB selects N data pages by default, counts the different values ​​on these pages to get an average value, and then multiplies it by the number of pages in this index to get the base of this index. The data table will be continuously updated, and the index statistics will not be fixed. Therefore, when the number of changed data rows exceeds 1/M, it will automatically trigger a new index statistics.

In MySQL, there are two ways to store index statistics, which can be selected by setting the value of the parameter innodb_stats_persistent:

• When set to on, the statistics information will be stored persistently. At this time, the default N is 20 and M is 10.
• When set to off, it means that the statistical information is only stored in the memory. At this time, the default N is 8, and M is 16.

In fact, index statistics are only an input. For a specific statement, the optimizer has to judge how many rows to scan to execute the statement itself, and at the same time, the cost of returning to the table needs to be included in the use of ordinary indexes.

Since the statistical information is incorrect, correct it. The analyze table t command can be used to re-calculate index information.

How to solve the index exception?

• One way is to use force index to forcefully select an index. The main problem with this is the timeliness of the change, because the wrong index selection is still rare, so the force index is usually not written first during development. Instead, you will not modify the SQL statement and add force index until there is a problem online. But after the modification, it needs to be tested and released. For the production system, this process is not agile .
• The second method is that we can consider modifying the statement to guide MySQL to use the index we expect. This modification is not a general optimization method, because there is a certain coincidence.
• The third method is that in some rare scenes, we can create a more appropriate index to provide the optimizer for selection, or delete the misused index.

Index the string field

Suppose, you are now maintaining a system that supports mailbox login. The user table is defined as follows:

mysql> create table SUser(
ID bigint unsigned primary key,
email varchar(64), 
... 
)engine=innodb; 

MySQL supports prefix indexes , that is, you can define part of the string as an index. By default, if you create an index statement without specifying the prefix length, the index will contain the entire string.

对应的创建索引语句：
mysql> alter table SUser add index index1(email);
或
mysql> alter table SUser add index index2(email(6));

Since each mailbox field in the email(6) index structure only takes the first 6 bytes, the space occupied will be smaller . This is the advantage of using prefix indexes. However, the loss that this brings at the same time is that it may increase the number of additional record scans .

Using prefix index, if you can define the length, you can save space without adding too much additional query cost.

The question is, is there any way to determine how long I should use the prefix?

When building an index, the focus is on the degree of discrimination. The higher the degree of discrimination, the better. Because the higher the degree of discrimination, the less repeated key values.

mysql> select 
  count(distinct left(email,4)）as L4,
  count(distinct left(email,5)）as L5,
  count(distinct left(email,6)）as L6,
  count(distinct left(email,7)）as L7,
from SUser;

Of course, using prefix index is likely to lose discrimination, so you need to pre-set an acceptable loss ratio , such as 5%. Then, in the returned L4~L7, find a value that is not less than L * 95%. Assuming that both L6 and L7 are satisfied here, you can choose a prefix length of 6.

The effect of prefix index on covering index

select id,email from SUser where email='[email protected]';

For example, if you use index1 (that is, the index structure of the entire email string) for such a query, you can use the covering index to return the result directly after finding the result from index1, without having to go back to the ID index to check it again. If you use index2 (that is, the email(6) index structure), you have to go back to the ID index to determine the value of the email field. Even if you modify the definition of index2 to the prefix index of email(18) , although index2 already contains all the information, InnoDB still has to go back to the id index and check again, because the system is not sure whether the definition of the prefix index is truncated Complete information .

When the distinction of prefixes is not good enough, and only the equivalent query is based on this field, what should we do?

• Method one, use reverse order storage. Save the field upside down.
• Method two, use the hash field. Create an integer field to store the check code of the field, and create an index on the integer field. But the check code may conflict, so the where part of the query statement must determine whether the values ​​are exactly the same.

The similarities and differences between the two methods of using reverse storage and using hash fields:

(1) None of them support range query, they can only support equivalent query.

(2) From the perspective of the extra space occupied, the reverse storage method on the primary key index does not consume additional storage space, and the hash field method requires an additional field. However, if the reversed field is too long, the cost is almost offset by the additional hash field.

(3) In terms of CPU consumption, the reverse function requires an additional call to the reverse function each time it is written and read, while the hash field method requires an additional call to the crc32() function. If you only look at the computational complexity of these two functions, the additional CPU resources consumed by the reverse function will be smaller .

(4) In terms of query efficiency, the query performance using the hash field method is relatively more stable . Because the value calculated by crc32 has a probability of conflict, but the probability is very small, it can be considered that the average number of scan rows per query is close to 1. After all, the reverse storage method still uses the prefix index method, which means that it will increase the number of scan lines.