In-depth understanding of mysql index (2)

Continuing from the previous article, we briefly learned the essential problem of indexing: what is an index, and this article continues to learn the data structure of index

MySQL storage structure

The storage structure of mysql is divided into: table space, segment, cluster (area), page, row.

• Table Space

The table space can be regarded as the highest level of the logical structure of the InnoDB storage engine, and all data is stored in the table space. Divided into: system table space, exclusive table space, general table space, temporary table space, Undo table space.

• Segment

The table space is composed of various segments. Common segments include data segment, index segment, rollback segment, etc. Segment is a logical concept. An ibd file (independent table space file) will consist of many segments.
Creating an index will create two segments, one is the index segment: leaf node segment, and the other is the data segment: non-leaf node segment. The index segment manages the data of non-leaf nodes. The data segment manages the data of the leaf nodes. In other words, the number of segments in a table is the number of indexes multiplied by 2.

• Cluster (area) Extent

A segment (Segment) is composed of many clusters (also called zones), and the size of each zone is 1MB (64 consecutive pages).
Each segment has at least one cluster. The size of the space managed by a segment is unlimited and can be expanded continuously, but the smallest unit of expansion is the cluster.

• Page

In order to efficiently manage the physical space, the cluster is further subdivided to obtain pages. A cluster is a space composed of consecutive pages (Page), and there are 64 consecutive pages in a cluster. (1MB/16KB=64). These pages are physically and logically continuous.
Like most databases, InnoDB also has the concept of pages (also called blocks), and each page defaults to 16KB. Page is the smallest unit of InnoDB storage engine disk management, which is set by innodb_page_size.
A table space has up to 2^32 pages. By default, the size of a page is 16KB, which means that a table space can store up to 64TB of data.

Note: In the
file system, there is also the concept of pages.
The smallest unit of the operating system and memory is the Page. The memory page of the file system is usually 4K.

# 查看有多少个数据页
show variables like 'innodb_page_size';

When inserting data into the table, if a page has been written, a new leaf page is generated. If all pages of a cluster are used up, a new cluster will be allocated from the segment where the current page is located.
If the data is not continuous, inserting data into a page that is full will cause the leaf page to split:

• Row
  InnoDB storage engine is row-oriented (row-oriented), which means that data storage is stored in rows.

The problem of AVL tree storage index

First of all, the indexed data is placed on the hard disk. View the size of the data and index:

SELECT
	CONCAT( ROUND( SUM( DATA_LENGTH / 1024 / 1024 ), 2 ), 'MB' ) AS data_len,
	CONCAT( ROUND( SUM( INDEX_LENGTH / 1024 / 1024 ), 2 ), 'MB' ) AS index_len 
FROM
	information_schema.TABLES 
WHERE
	table_schema = 'tfree6' 
	AND table_name = 'horder';

When we use the tree structure to store the index, an IO will occur between accessing a node and the disk.
The smallest unit for InnoDB to operate a disk is a page (or a disk block), and the size is 16K (16384 bytes).

So every time I perform IO with the disk, a 16k data page is loaded.

If we only store a key value + data + reference for a node, such as a reshaped field, it may only use a dozen
or dozens of bytes, which is far less than the capacity of 16K, so visit a tree node and proceed A lot of space was wasted during an IO

Therefore, if each node stores too little data, to find the data we need from the index, more
nodes must be accessed , which means that there will be too many interactions with the disk.

If it is the era of mechanical hard drives, it takes about 10ms of addressing time to read data from the disk each time.
The more interactions, the more time consumed.

If there is a problem, there must be a solution:

Scenario -> Problem -> Solution to Problem -> Implementation of Solution

If I were a designer of mysql and I faced this problem, I would have the following
solutions:

1. During each IO interaction, store as many tree nodes as possible in the data page
2. Each tree node stores as many references (keywords) of child nodes as possible, so that we have more pointer trees and more branches (number of ways)

Because the more branches there are, the depth of the tree will decrease (the root node is 0).
In this way, our tree has changed from its original thin, tall and thin look to a short, squat, and chunky look?
At this time, our tree is no longer binary, but multi-fork, or multi-way.

AVL tree

B Tree From the

above picture, we can see that the same 17 elements are inserted. The AVL tree has a depth of 5 layers, while the BTree has only 4 layers. If the data is more, the comparison will become more and more obvious.

• Multi-channel balanced search tree (Balanced Tree)-B Tree

Features:

1. Like AVL trees, B-trees store key values, data addresses, and node references in branch nodes and leaf nodes.
2. The number of forks (the number of ways) is always one more than the number of keywords. For example, if each node stores two keywords, there will be three pointers to three child nodes.
   
   
   To find 1 here, only 3 IO operations are required, which is much more efficient than the AVL tree.

The number of subtrees owned by a node is called the number of " degree "
keywords: N
Degree (Degree): N + 1
Max Degree is set to 3 means: the maximum number of subtrees owned by a tree is 3. If a subtree has Three keywords. That means that the "degree" of this node is 4, which exceeds the set value of 3. At this time, this node must be split (note that it is not left-handed or right-handed in the AVL tree).

If the node is deleted, there will be a reverse merge operation. So this also explains: Why not create indexes on frequently updated fields or why not update primary keys, because splitting and merging keywords on index nodes consumes memory resources and requires a lot of adjustments to the tree structure.

• B+ tree (upgraded multi-channel balanced search tree)

Where is the B+ tree upgraded than the B tree? You can get a glimpse through the data structure diagram of the B+ tree

Features of B+Tree in MySQL:

1. The number of keywords is equal to the number of paths (degree)
2. Neither the root node nor the branch node of B+Tree stores data, only the leaf nodes store data. The searched keyword will not be returned directly, but the leaf node of the last layer will be reached.
3. Each leaf node of B+Tree adds a pointer to the adjacent leaf node, and its last data will point to the first data of the next leaf node, forming an ordered linked list structure
4. It retrieves data based on the interval of left closed and right open [).

Let's study how much data can be stored in the lower 3 levels of the B+ tree?

Assuming that a record is 1K, a leaf node (one page) can store 16 records. How many pointers can a non-leaf node store?
Assume that the index field is of type bigint and the length is 8 bytes. The pointer size is set to 6 bytes in the InnoDB source code, so a total of 14 bytes. A non-leaf node (one page) can store 16384/14=1170 such units (key value + pointer), which means there are 1170 pointers.
When the tree depth is 2, there are 1170^2 leaf nodes, and the data that can be stored is 1170 1170 16=21902400.
When searching for data, one page search represents one IO, that is, for a table of about 20 million, querying data requires up to 3 disk accesses.
Therefore, the depth of the B+ tree in InnoDB is generally 1-3 levels, which can satisfy tens of millions of data storage.

Advantages of B+Tree in MySQL:

1. Each node stores more keywords; more channels
2. The ability to scan the database and scan the table is stronger (if we want to scan the entire table of the table, we only need to traverse the leaf nodes, not the entire B+Tree to get all the data)
3. The disk read and write capability of B+Tree is stronger than that of B Tree (root node and branch node do not save data area, so one node can store more keywords, and one disk load (IO) has more keywords)
4. Stronger sorting ability (because there is a pointer to the next data area on the leaf node, the data forms a linked list)
5. Range query is faster: For example, if you want to query data from 22 to 60, when you find 22, you only need to traverse the nodes and pointers in order to access all the data nodes at once, which greatly improves the efficiency of range query ( There is no need to return to the upper parent node to repeat the traversal search).
6. The efficiency is more stable (B+Tree always gets the data at the leaf nodes, so the number of IOs is stable)

Related interview questions:

• Why doesn't mysql use red-black tree as index data structure?

Red-black trees are also BST trees (binary search/sort trees), but they are not strictly balanced.
Characteristics of red-black trees:

These constraints enforce the key properties of red-black trees:

The longest path from the root node to the leaf node (the red and black path) is not more than twice the shortest path (all black nodes).

Why not use red and black trees? So the answer is:
1. There are only two ways: too few degrees, resulting in increased IO times and wasted space utilization.
2. Not balanced

Note: The red-black tree is generally only used in the memory. For example, Java's TreeMap.

• InnoDB creates an index on the client. Can a hash index be used?

In Navicat's tool, there are two ways to create an index, Hash and B Tree.

We first analyze the characteristics of the hash index :

1. Retrieve data in the form of KV, that is, it will generate a hash code and pointer according to the index field, and the pointer points to the data.
2. Its time complexity is O(1), and the query speed is relatively fast. Because the data in the hash index is not stored in order, it cannot be used for sorting.
3. When querying data, the hash code is calculated based on the key value, so it can only support equivalent query (= IN), not range query (> <>= <= between and).
4. If there are many duplicate values ​​in the field, there will be a lot of hash conflicts (using the zipper method to solve), and the efficiency will be reduced.

Answer:
official website answer

InnoDB utilizes hash indexes internally for its Adaptive Hash Index feature is
directly translated as: InnoDB internally uses hash indexes to implement adaptive hash indexes.
The meaning of this sentence is that InnoDB only supports the explicit creation of B+Tree indexes. For some hot data pages, InnoDB will automatically create an adaptive Hash index, that is , create a Hash index based on the B+Tree index. This process is for the client It is uncontrollable and implicit.
We choose the index method in Navicat tool to be hash , but it creates a B+Tree index , which is not something we can manually control.
There is an area in the buffer pool that is Adaptive Hash Index, which is this.
This switch is ON by default:

show variables like 'innodb_adaptive_hash_index';
# 从存储引擎的运行信息中可以看到:
show engine innodb status\G
----------------------
# BUFFER POOL AND MEMORY 
# INSERT BUFFER AND ADAPTIVE HASH INDEX 

Another: B Tree and B+ Tree are also used in Oracle and SQLServer databases.
Next articleIn-
depth understanding of the B+Tree landing form of mysql index (3)