Adapted from the MySQL engine architecture and performance optimization https://url.wx-coder.cn/IF5HH , declared in reference documentation List https://parg.co/htL Awesome MySQL .

Principles and Applications of MySQL indexes: the index type, storage structures and lock

In the data structures and algorithms - index https://url.wx-coder.cn/O07eI one, we discussed the basic algorithm B + Tree, LSM-Tree indexes such files as well as full-text index, this article will be for file indexes in a relational database to explore the practical application.

Index (Index) is a database system to help efficiently get the data structure of the data, and the database index is essentially based on additional write operation and maintenance of storage space for the index data structures for the price, for raising the efficiency of data retrieval database data structure. Index can help us quickly locate the data without the need to search every time you are through each row in the database. Of course, the more the index is not established, the longer the better, because the index in addition to the space, to increase the follow-up database, delete, modify has additional operations to update the index. In general, small table with a full table scan faster, only large table using an index, and the index table Super largely ineffective, we may need to use a separate full-text indexing system; MySQL comes with the full-text index can only be used InnoDB, MyISAM, and can only be conducted in English full-text search, the general use of ES, Solr such a full-text index engine.

Index Type

From the realization of the index, we can be divided into clustered index and non-clustered index, also known as secondary indexes or secondary indexes, these two categories; from the practical application of the index, can be subdivided into general index, the only index , primary key index, joint index, foreign key indexes, full-text indexing these types.

InnoDB clustered index can be seen, because the leaf nodes of the B + tree which includes the complete data record. InnoDB data file itself is an index file, according to the table data file itself is a B + Tree index structure of the organization, this tree's leaf node data field holds a complete data record. This index is the primary key of key data tables, so InnoDB table data file itself is the main index. InnoDB auxiliary data field stores an index value is not the address of the corresponding primary key record. In other words, all the secondary index InnoDB are cited as the primary key data field.

The leaf nodes of the B + tree MyISAM embodiment only stores the address of the data, it is called a non-clustered index. MyISAM engine using B + Tree as an index structure, the leaf node data field is the address of the data record; in MyISAM, the main index and a secondary index (Secondary key) there is no difference in structure, only the primary index claim key unique , and assist the key index can be repeated.

In InnoDB, there are clustered sub-index and the ordinary index, the clustered index to build the primary key, the leaf node is stored in the rows of the main key corresponding primary key query can directly use a clustered index is positioned to where the recording. The general index is constructed according to declare this index when the column, the leaf node is stored in the value of the rows corresponding to the primary key, the query need to find the value corresponding to the primary key in the general index based on the general index, then go based on the primary key value Find a record on a clustered index, commonly known as back to the table. If we query a whole row record, it must go clustered on the index to find, but if we only need to query the primary key in accordance with the general index values, because these values in the general index already exists, it does not need to return to the table, this says the index cover, to a certain extent, can improve query performance.

Whether there are general index unique index and joint index two exceptions, and inserting a unique index modification time will check the value of the column corresponding to the index already exists, the index value of the combined two columns in the order stated time in constructing the index after stitching.

The smallest unit of storage data rows are not stored in the engine management, the index can only help us to locate a data page, the smallest unit of disk reads and writes every data page is also, and a page of data stored in multiple data lines, we need to understand the internal structure of the data page in order to know how to locate a storage engine row of data, you can refer to MySQL storage management https://url.wx-coder.cn/IF5HH series.

Selective Index

Column index and prefix length strings, are referenced selectivity (Selectivity) this indicator to determine: the ratio of selectivity is defined as the total number of records will not be repeated and the data index value, the higher the selectivity, the query efficiency index higher, such as for gender this parameter, index no sense.

Index Selectivity = Cardinality / #T
复制代码

Obviously the range of selectivity (0, 1], the higher the selectivity index greater the value, which is determined by the B + Tree properties. In the actual database, we can calculate the selectivity of a column by the following statement:

SELECT count(DISTINCT(title))/count(*) AS Selectivity FROM titles;
复制代码

Primary key

Inside the InnoDB, the primary key table fast data query optimization are arranged distributed, which is the fastest search speed, the logical order of the index key determines the physical order of the corresponding row in the table. Even if the table does not fit the column as a primary key, also recommended an automatic increase integer primary key (surrogate key), then the table increases when data is sequentially stored, and subsequent in another table referring to the foreign key query time It will be optimized.

If not explicitly define the primary key (Primary Key) when creating the table, the storage engine InnoDB select or create a primary key as follows:

First, if there is non-empty unique index (Unique NOT NULL) table, if there is, the column that is the primary key.
Does not meet the above conditions, InnoDB storage engine automatically create a 6-byte pointer size, users can not view or access.

You select the primary key

In a distributed ID https://url.wx-coder.cn/tQ5eH the article we discussed the selection policy ID distributed under the distributed scenario, while in the database, we also have such considerations. First, MySQL official primary key there is a clear suggestion to try the shorter the better, UUID 36 characters in length does not meet the requirements; if the primary key is a long string and built a lot of the general index, the index account for a large common cause physical space. And the primary key is the best order of increasing, or at InnoDB engine, UUID disorder may cause frequent changes in location data seriously affect performance.

ID incremented at the time of insertion ensures that adjacent two records in the same data block may, and continuity of the design of such a service order number may not increment the associated ID is good, resulting in a continuous plurality of data blocks may be inserted increase the number of disk read and write.

Uniqueness: time increment ID will easily be cracked violence, data migration time, especially in the form of merger of this operation, there will inevitably conflict. UUID can be guaranteed to be unique, completely avoid conflicts.
Key Length: increment length of the field is much smaller than the UUID, this will have a greater impact retrieval performance. When Innodb data search engine, the index is to find the master key, and then recording found primary key; so that in the primary key length is short, the read performance will be better.
Concurrency: the case of the self-energizing ID and highly concurrent, increasing competition from lock will reduce the throughput of the database. UUID UUID can be generated in the application layer, to improve the throughput of the database.
Index database: InnoDB table data stored in the order according to the master key, if the random IO occurs, it will frequently moving disk blocks when writing data. When the amount of data when writing short board will be very obvious. Self-energizing ID in the new data may be sequentially arranged in default, the performance is greatly improved; law between the UUID is not the primary key sequence.

Primary key and unique index

Primary key is a unique index, but the index is not necessarily the only primary key, a unique index may be null, but null can have only one primary key can not be null. For the single index, the list of all required data are not the same, but allow NULL values; joint index for a plurality of columns, the columns required combination is unique. Unique index itself may be used as an index, the actual data may also be used to generate constraints prevent the same add or modify data, thus ensuring data integrity.

For string types, the index can be specified prefix length (and for BLOB / TEXT prefix length parameter is required), in which table InnoDB longest prefix length is 767 bytes, and the parameter M is metered bytes. So long string, build B + Tree index waste is relatively large, this time with a manual simulation HASH index is a method, but this way strings can not use the prefix way flexible query (eg LIKE this type of operation).

Joint index

Separate index refers to the index established for a certain field on the table, the general index of choice to create an integer or a smaller fixed-length string will be more conducive to enhancing efficiency. It refers to a joint index plurality of index fields organized in a certain order. An index (name, city, gender)for example, the first name field in accordance with the order of the organization, when the value of the same field name (e.g., Bush), which according to the city field sequential organization, when the field value is the same city, its field organization according to gender. Since building the index by more than one column on a joint index, sometimes we need to be frequently queried field is applied inside the joint index, for example, often require age we can build a name and age of the joint index based on name lookup.

Common combinations include a WHERE clause conditions in combination with ORDER BY joint condition; refers to a so-called joint WHERE condition is equivalent to the WHERE condition, which is consistent with the joint index fields using fields (the order can be different).

ORDER BY means that if a joint ORDER BY behind after the field is a field where conditions index covering the joint, because the index is already in an orderly state, MySQL will read the data directly from the order index, then read on the disk after the data fetch data organized in this order, thereby reducing the operation disk to sort data. That is not covered ORDER BY queries for which there is a Creating sort index, were the most time-consuming is the sort of disk data; for covering ORDER BY queries that do not need to be sorted, but it is mainly reflected in the time-consuming disk the pulling process data.

Prefix index

MySQL prefix index can be divided into three categories: joint index prefix, like the string prefix and prefix.

United leftmost index prefix match (Leftmost Prefix)

Joint index prefix refers to the establishment of a multi-column index, you must use all or part of the indexed columns from left to right in order to fully use the joint index, for example: (col1, col2, col3)using (col1)、(col1, col2)、(col1, col2, col3)valid. In the query will always match the right until it encounters a range query (>,<,BETWEEN,LIKE)stops match, followed by the index of the column will not use the index to find the optimization.

In (name, city, interest)three fields combined index as an example, if the query is where name='Bush';then only need to locate the value of the B + tree name field where the first Bush, then sequentially scanned subsequent data until the first data that is not Bush It may be, the scanning process id of the data recorded in the index, obtaining the final result set based query id clustered indexes. Similarly for the query condition is where name='Bush' and city='Chicago';a query, MySQL can be positioned directly in accordance with the index to the index joint intermediate gray portion, and acquires data id of the index sheet, obtaining the final result set based query id clustered indexes.

From this we can draw attention to the point joint index prefix:

Across the field joint index not used, such as where name='Bush' and interest='baseball';, for the query, name field is the first field may be used jointly most index filtering data, but for the interest fields, which can not pass directly to the B + tree characteristic of the third field positioning the index data, such as baseball herein may be dispersed among the second and seventh data. Eventually, interest field is actually carried out covering index scan.
For non-equivalence conditions, such as>, <,! =, Etc., joint index prefix to the filter up to the index of the first field only using non-equivalence conditions, although it can not participate in subsequent fields in the index joint index filtration. Here for example where name='Bush' and city>'Chicago' and interest='baseball';, for the query may first be non Bush filtered data field name of a field index of the first sheet, and then positioning the second field according to a joint index Chicago location index sheet, due to its non-equivalence conditions here will MySQL sequential scanning down from Chicago positioned, since it is possible to interest fields dispersed in any position index of the third field, and thus can not participate in the third field of the index filtration.

Thus the order of columns B-Tree is very important, and the usage rule related column order. For practical applications, usually according to the specific needs of different columns to create an index and a different column order. Suppose Index Index(A,B,C):

# 使用索引
A>5 AND A<10 - 最左前缀匹配
A=5 AND B>6 - 最左前缀匹配
A=5 AND B=6 AND C=7 - 全列匹配
A=5 AND B IN (2,3) AND C>5 - 最左前缀匹配，填坑

# 不能使用索引
B>5 - 没有包含最左前缀
B=6 AND C=7 - 没有包含最左前缀

# 使用部分索引
A>5 AND B=2 - 使用索引 A 列
A=5 AND B>6 AND C=2 - 使用索引的 A 和 B 列
复制代码

Using an index to sort the results, in order to index and ORDER BY clauses of the same order, and all the columns in descending consistent with liter (ASC / DESC). If the query joins more than one table, only in the ORDER BY column reference is the first table can (need sequential JOIN).

# 使用索引排序
ORDER BY A - 最左前缀匹配
WHERE A=5 ORDER BY B,C - 最左前缀匹配
WHERE A=5 ORDER BY B DESC - 最左前缀匹配
WHERE A>5 ORDER BY A,B - 最左前缀匹配

# 不能使用索引排序
WHERE A=5 ORDER BY B DESC,C ASC - 升降序不一致
WHERE A=5 ORDER BY B,D - D 不在索引中
WHERE A=5 ORDER BY C - 没有包含最左前缀
WHERE A>5 ORDER BY B,C - 第一列是范围条件，无法使用 BC 排序
WHERE A=5 AND B IN(1, 2) ORDER BY C - B 也是范围条件，无法用 C 排序
复制代码

like prefixes

For like the prefix, which means that when used like a query, if the expression is used first_name like 'rMq%';then it can be used in the index of the first_name field. But for first_name like '%Chu%';that you can not use the index first_name. For like prefix, MySQL bottom is actually used a strategy to completion using an index, such as here first_name like 'rMq%';, MySQL which will complement the data into two: rMqAAAAA and rMqzzzzz, back up the whole length of the portion of the maximum length of the current field . When using the index query, MySQL on the use of indexed positioning of these two data, the final result is that the data set required intermediate portion of the two anchor points. The following is a schematic view of the use of prefixes like:

String prefix

String prefix index refers to the index before removing the string with only a few characters established. During the query, if a long field value, its cost will be very high indexing, and query efficiency is relatively low, string prefix index is to solve this problem exists. String prefix index mainly used in two ways:

Selective field prefix portion is relatively high;
Field overall selectivity is not so large (if the field is relatively large overall selectivity hash index can be used).

Established as such first_name prefix index field with a length of 4 can be seen, if the query is used where first_name='qWhNIZqxcbD';, it will first intercept MySQL first four characters of the equivalent condition, and then compares it with the string prefix index to locate the prefix "qWhN" index sheet, and then acquires the disk data corresponding to the index sheet, FIRST_NAME field of the last acquired data disk compared with the value of the equivalent query conditions, to obtain a result set.

One problem with prefix string is most important to note how to choose the length of the prefix, to select the appropriate length of time, the filtration of the prefix index and selectivity index almost equal to the entire field. Here we need to use the concept of selectivity on the field in front of explanation, that the selectivity of field after field grouping, the amount of data sets maximum amount of data the proportion of the total amount of data. When the prefix length selected here, it is understood, according to the prefix after the prefix packet selectivity ratio, the maximum amount of the total data size groups. Prefix length is calculated as shown in Table SQL formula:

select count(*) as cnt, first_name as perf from actor group by perf ORDER BY cnt desc limit 10;	-- 0
select count(*) as cnt, left(first_name, 2) as perf from actor group by perf ORDER BY cnt desc limit 10;	-- 2
select count(*) as cnt, left(first_name, 3) as perf from actor group by perf ORDER BY cnt desc limit 10;	-- 3
select count(*) as cnt, left(first_name, 4) as perf from actor group by perf ORDER BY cnt desc limit 10;	-- 4
复制代码

Other index

Covering index

It refers to a covering index for the removal of all fields involved in query index scan filter used to index the index added to the tail of the query used. Covering index scan advantage is that since all the fields used in the query are indexed in the same field, and thus only need to obtain relevant data in the index during a query, the corresponding data back to disk without the need for scanning, thus avoiding queries most time-consuming disk I / O read. For the following query:

select a, b, c from t where a='a' and b='b';
复制代码

If the query establish joint index (a, b, C), which is then used to cover the index scan, because for the query, the index may be used before two fields a and b according to a filter where the index of the condition, reading of the index sheet is directly filtered in the index a, b, c to the value of three fields, back to the table without scanning.

Samsung index

Samsung index refers to a query, the conditions for the establishment of three general conditions index, the index established for a particular query every one condition, says the index to get a star, when the index get three stars for this query indicates that the index is a Samsung index. Samsung index is the best index for a particular query, the establishment of Samsung index of the following conditions:

Remove all predicates equivalent column （WHERE COL=…）as the start index of the column;
The ORDER BY columns added to the index;
The query is added to the index in the remaining columns, the columns will be easy to get into the final update in order to reduce costs.

For example, the following query, the index (first_name, last_name, email)is a Samsung index:

SELECT first_name, last_name, email FROM user WHERE first_name = 'aa' ORDER BY last_name;
复制代码

Samsung index creation process can be found in the following rules:

Covering the equivalent predicate condition, such as FIRST_NAME, most of the index can filter data;
Covering the field order by sorting the result set can be avoided, such as last_name;
Covering the rest of the field to avoid reading the data back to disk, which uses covered index scans, such as email.

Index storage structure

When MySQL queries through the index will first locate the corresponding data page, and then detect whether the data pages in the buffer pool, if the direct return, if not go to a clustered index reads the corresponding data page and placed by disk IO buffer pool. A data page will contain multiple rows of data. Buffer pool by LRU algorithm for page management data, which is the most frequently used data pages at the front of the list, end of the line, when the pool is full of time will eliminate infrequently used data pages behind the tail. The new data from the disk to read the page and not on the head of the queue but in the middle position, the intermediate position parameter can be repaired. Buffer pool may also be provided a plurality of instances, the decision on which of the data buffer pool page according to the hash algorithm.

In MySQL storage structure of the article, we discussed the structure of MySQL to store data pages.

Memory Architecture | Memory Architecture

InnoDB memory has the following main components: a buffer pool (buffer pool), pool redo log (redo log buffer) and additional memory pool (additional memory pool), as shown below:

Which account for the largest block of memory buffer pool, each used to cache data, the data files by pages (16K) read into the buffer pool, according to least recently used algorithm (LRU) cache data retention. Pool buffered data types: data page, index page, insert buffer, adaptive hash index, lock information, data dictionary information, data and index pages which accounted for the majority of memory. Log buffer to redo log information first into this buffer, and then at a certain frequency (default 1s) will refresh it to the redo log files.

InnoDB background thread by a series related asynchronous processing operations, buffer pool while the means to reduce the difference in the CPU and disk speed. When the query will first positioned by the index to a corresponding data page, and whether the detected data page in the buffer pool, if the direct return, if not go to a clustered index corresponding to the read data into the page and through the disk IO buffer pool. A data page will contain multiple rows of data. Buffer pool by LRU algorithm for page management data, which is the most frequently used data pages at the front of the list, end of the line, when the pool is full of time will eliminate infrequently used data pages behind the tail. The new data from the disk to read the page and not on the head of the queue but in the middle position, the intermediate position parameter can be repaired. Buffer pool may also be provided a plurality of instances, the decision on which of the data buffer pool page according to the hash algorithm.

Storage Architecture | storage structure

Logical storage structure and Oracle InnoDB storage engine is substantially the same, all data is stored in a logical space, we call table space (tablespace). Table space is a segment (segment), area (extent), page (page) components. In some document pages sometimes referred to block (block), 1 extent = 64 pages, a logical memory structure InnoDB storage engine substantially as shown:

As the highest level of table space storage structure, all data is stored in the table space, by default, with a shared table space ibdata1, if you open a data innodb_file_per_table each table will be stored in a separate table space, that is, each tables will have a file,

Tablespace consists of individual segments, InnoDB storage engine by the index organization, and the leaf node index for recording data, stored in the data segment, the leaf node instead used to build the index stored in the index section. Zone is continuous pages, if any one region is 1MB, one zone can have multiple pages, each default is 16KB, so a default zone may comprise 64 consecutive pages, pages size can be provided by innodb_page_size, the page is stored in specific rows. His record eventually stored in a binary file.

From the physical point of view sense, InnoDB table (more specifically, it should be Redo file group) of the shared space table, the log file group, the table structure definition files. If innodb_file_per_table set to on, each table will independently produce a tablespace file, ending with IBD, data, index, table of internal data dictionary information is saved in the file space in a separate table. Table structure definition files ending in frm, this is nothing to do with the storage engine, the storage engine of any table structure definition files are the same, for the .frm file.

Process Architecture | process architecture

By default, InnoDB background thread has seven, of which four IO thread, 1 Ge Master thread, 1 Ge Lock monitor thread, a Error monitor thread. InnoDB is the main work of the Master in a separate thread to complete. Master thread has the highest priority, it is divided into the following cycle: the main loop (loop), the background loop (background loop), the refresh cycle (flush loop), cycle pause (suspend loop).

Wherein the main loop pseudo code is as follows:

void master_thread() (
    loop:
    for (int i =0; i <10; i++){
        do thing once per second
        sleep 1 second if necessary
    }
    do things once per ten seconds
    goto loop;
}
复制代码

Wherein one of the second operation comprises: refresh log buffer (always) and the combined insertion buffer (possible), the refresh up dirty data pages 100 (possible), if the user is not currently active, switched to the background loop (possible).
Wherein once every 10 seconds of operation comprising: inserting a buffer pool up to 5 (always), the refresh log buffer (always), 100 or the refresh dirty pages to disk 10 (always), generates a checkpoint (always ), Undo delete unwanted pages (always).
Background loop, if no user is currently active database or shut down, switches to the loop to do the following: undo delete unwanted pages (always), insert buffer merge 20 (always), jumps back to the main loop (always ), and constantly refresh 100 pages, until the qualifying jumps to flush loop (possible).
If the flush loop also having nothing to do, while switching to suspend loop, the master thread suspension.

Index and lock

MySQL provides us with row locks, table locks, page locks are three levels of locks, which cost a small table locks, lock fast; not deadlock; large locking strength, high probability of lock conflicts, the lowest degree of concurrency. Large overhead line lock, lock slow; there will be a deadlock; lock small size, low probability of lock conflicts, high concurrency; page locks and locking overhead rate between table and row locks; there will be a deadlock ; particle size between locking table and row locks, general concurrency. Each storage engine can have its own lock strategy, such as MyISAM engine supports only table-level locking, InnoDB engine and support in addition to table-level locking, but also supports row-level locking (default).

	Row lock	Table lock	Page locks
MyISAM		√
BDB		√	√
InnoDB	√	√

InnoDB row lock is achieved by the index to the index items on the lock, which is different from MySQL and Oracle, which is locked by a corresponding data line in the data block to achieve. InnoDB row lock to achieve this characteristic means: only through index conditions to retrieve the data, only InnoDB row-level locking, otherwise, would use InnoDB table lock, likewise, when the for updaterecord does not exist will lead to lock the entire table. When the table has a plurality of indexes, different transactions can lock with different index different rows, additionally, whether used primary key index, the general index or a unique index, InnoDB row locks are used to lock the data.

InnoDB locking procedure is more complicated, all the records are scanned into the lock, the lock gap will increase the scope of the query, and then locking procedure in accordance with the two-phase locking 2PL to achieve, which is to be locked, and then submit all the locks in things when released. The strategy will lock isolation level and database-related, in the case of default isolation level of repeatable read, the process will be locked and query whether the index contains, it is the primary key index or general index, if the index is unique and so on.

For example, select * from o_order where order_sn = '201912102322' for update;this SQL statement, under different circumstances its index locking strategy is also inconsistent:

order_sn is the primary key index, in this case the primary key index order_sn = 201912102322plus an exclusive lock on this record.
order_sn ordinary index and a unique index, will be on an index corresponding to ordinary recording plus exclusive lock on the record corresponding to the primary key index plus exclusive lock.
order_sn ordinary index, and is not a unique index, the index will be of the normal order_sn = 201912102322one or a plurality of record locking, and the lock of the record corresponding to the record these primary key index. Here in addition to adding a row lock, the lock will be added to the gap, to prevent other transactions from inserting order_sn = 201912102322records, but only if the index does not need gap locks, row lock on it.
not on order_sn index, innoDB will be a full table scan on the primary key index, and there is no plus table lock, but all records will add row-level exclusive lock, but in fact innoDB internal optimized when scanning to a row after the match will be found not to release the lock, of course, contrary to the principle of 2PL released when the transaction is committed. Here in addition to the records locked, but also on the gap between each two record locking, so the gap will eventually save all locks and order_sn = 201912102322row locks.
order_sn = 201912102322This is the case where the record does not exist, if the primary key index is order_sn will add a lock gap, and this gap is the first record in the primary key index 201 912 102 322 order_sn less than to greater than the first record of 201,912,102,322. Imagine if we do not lock gap, if everything else insert a order_sn = 201912102322record, due select for update is currently reading, even if things did not submit above that, if re-query the things one would read the magic happen.
Without an index, then the scanned all the records are locked and the gap, if the line does not match the lock will be released only lock gap. Recall that the results of the data page of the talk of the above, he also recorded a maximum and minimum record, Infimum and Supremum Record, both records will be used when the lock gap in Canada.