mysql B + tree index lookup using a binary search, also uses the jump table Redis binary search method, Kafka query message log also uses binary search method, binary search time complexity of O (logn);
Reference: jump redis index table principle underlying implementation talk Mysql indexes and tables --- redis jump jump table principle redis time complexity of O (logn) (Ali)
Reference: Kafka used to live how to achieve high concurrent memory - how to find data a consumer needs (Ali)
Reference: binary search method: Time various sorting algorithms complexity and space complexity (Ali)
In MySQL, there are four types of indexes, respectively: B-Tree index, Hash index Fulltext index ( MyISAM tables ) and R-Tree index, herein said is B-Tree index.
The principle behind the index have to look too important, Ali, two people have asked this index principle of mysql
mysql using B + Index:
B-tree: an ordered array of multiple balanced tree +;
B + tree: an ordered array list + balancing multiple tree;
A, Mysql index there are two structures: B + Tree index and Hash Index
(A) Inodb default storage engine B + Tree index
(B) MyISAM is the default storage engine Fulltext index;
(C) Memory storage engine the default Hash index;
Hash index
mysql, only have Memory (Memory table exists only in memory, power will disappear for a temporary table) storage engine display support Hash index is the default index type Memory table , although you can also use the Memory Table B + Tree index. Hash index data organized in a hash form, so when looking for something a record time, very fast. However, since the hash structure, each key corresponds to only one value, but is distributed hash. So it does not support the range of search and sorting functions.
B + Tree index
Mysql B + Tree is the most frequently used data structure of an index, an index type and Inodb Myisam storage engine mode . Hash relative index, B + Tree to find in a single record as fast as Hash index, but because it is more suitable for operations such as sorting, so it is more popular. After all, not only to operate the database of a single record.
With sequential access pointer B + Tree
B + Tree index data are all in the leaf nodes, and increases the sequential access pointer, each leaf node has a pointer pointing to a leaf node adjacent.
This is done to improve the efficiency range, for example, the query for all data records from the key 18 to 49, 18 when they are found, as long as the nodes and along the order of the pointers can be traversed in order to have access to all the data nodes, greatly improves the range query efficiency.
Greatly reduce disk I / O read
Designers clever use of the database system disk read-ahead principle, a node is set equal to the size of a page, so that each node requires one I / O can be fully loaded.
What is the index
Index (Index) is a database to help efficiently get the data structure of the data. The index is based on a database table is created, a table which contains the address value, and some of the columns corresponding to the record, and a data structure to store these values in. The most common is to use a hash table, B + tree as an index.
General applications, read and write in the ratio of about 10: 1, and rarely insert and update operations general performance problems in a production environment, we encounter the most, it is the most problematic, or some complex query operation, thus optimizing the query statement is clearly a top priority. Speaking to speed up queries, we have to mention indexed.
Why use an index
We know that the database query the database is one of the most important functions. The query speed course, the sooner the better. And when the increasing amount of data, the time it takes the query will follow growth. The index, you can speed up the query data. Because the index is an ordered arrangement.
For example, suppose we have a database table Employee, this table has three fields are: name, age, address. Suppose there are 1000 records in the table.
If you do not use the index, when we query the employees called "Jesus" of the time, that is, call:
select name,age,address from Employee where name = 'Jesus';At this time, the database has to this in the Employee table 1000 records a one to judge whether the field name "Jesus". This is called a full table scan.
When we create an index on the name field on the Employee table when we query the employees called "Jesus", and looks through the index to query the employee named "Jesus" because the index has been arranged in alphabetical order, So much faster when you want to find the name "Jesus" of the record, because the first letter of the name is "J" employees are aligned together. By this index, the table can be obtained in the corresponding record.
Illustrates the benefits of using an index
Suppose the index (index is a data structure) is a linked list structure. Each node is stored in a key field (in this case corresponds to the attribute name) and the address records in the database corresponding to the key field in the tables. These nodes are sorted according to the attribute name (i.e., sorted alphabetically). Therefore, when we execute find sql statement said above, called "Jesus", the database can be queried by the index, because the list is ordered in our property to find the first name "Jesus" node after, continue to the next looking for, when it comes to the name attribute is not "Jesus" node, then again there is no need to find, because the nodes are ordered according to the name attribute of ah. Assume that the first name = "Jesus" of the node is the first node 499, the last name = "Jesus" of the node is the first node 500, then only need to traverse 501 nodes on it. When they find the name field 501 nodes is not "Jesus", 499 nodes will not need to traverse the back. Through the index, we found a name for the "Jesus" of nodes, and by another property of the node (corresponding to the address of record of the key field in the database table), we can get to meet the conditions of the Employee table name = "Jesus" is recorded.
To reduce the number by using the index, the query is determined from 501 to 1000 times. Played accelerate query efficiency. But in fact the database index structure, not a linked list structure.
What database data structures used as an index
The index database is actually used not list structure, because efficiency is too low.
We know the list of query efficiency is O (n). As in the example above, traversing the 501 qualifying times recorded the first to find that it is very inefficient. And we know that the array + binary search efficiency is O (lgn), but the efficiency of insert and delete elements of the array elements is very low, so use the array as the index structure is not appropriate.
In addition, the choice of the structure of the database indexes when taking into account other issues. The index is present in the disk, when the index is very large, reaching several G when not loaded into memory at one time.
Taking into account the above two factors, the database index using a tree structure.
The names of various trees
There are so few trees:
B-Tree
B+-Tree
B*-TreeWe must first understand the three kinds of trees in the name of "-" is to separate the role played, not "cut" means.
Therefore, the correct translation should be B树，B+树，B*树. Instead B-树，B+树，B*树. So when you hear someone say "B minus tree" when, to understand that it refers to the B-Tree. I.e., B-trees and B- trees is the same trees.
Why the emphasis on top of this, it is because there is Bowen wrote: B tree is a binary tree, B- tree is a multi-way search tree.
However, B-trees and B- trees are by B-Tree. Quoted on Wikipedia:
B-tree
Not to be confused with Binary tree.
In other words, B-Tree is not Binart tree. B-Tree's Chinese name is balancing multiple search tree.
(B-tree related description below)
Balanced binary tree
The tree is the most important computer system data structures.
We know that the time complexity of binary tree lookup is O (log2N), and the depth of its search efficiency, and ordinary binary tree node arrangement may be due to internal problems degenerate into a linked list, so look for efficiency will be very low. Thus balanced binary tree is a better choice, because it is balanced, i.e., to maintain a minimum depth by rotating the adjusting structure. Find its time complexity is O (log2N).
But in fact, the database index structure is not an AVL tree or better red-black tree, although its query time complexity is low.
Why is not suitable as a balanced binary tree index
Said before the search time complexity of the balance of the tree is O (log2N), it has been very good, but still not suitable as an index structure. So there is definitely a more suitable as an index of the data structure. So this is more suitable as an index of the data structure, is it time to find it less complex? It is not. This approximation is also O (log2N) time complexity as a lookup data structure of an index.
Why not suitable as a balanced binary tree index it?
The index is present in the index file is present in the disk. Because the index is usually great, and therefore can not index all at once loaded into memory, so you can only read data from a disk page of the disk into memory. The read speed of the disk than in terms of read speed memory is the difference between several levels.
Note that we are talking about a balanced binary tree structure, referring to the balanced binary tree logical structure, its physical implementation is an array. Since then nodes on the logical structure may be similar to the difference in physical structure very far. Therefore, the data page of each disk read many of which are no longer applicable. Therefore, the process to find the disk read operation many times.
The structure is suitable as an index should perform disk IO operations as little as possible, because to perform disk IO operation is very time-consuming. Thus, as a balanced binary tree index structure is not suitable.
Suitable as a B-Tree index
Balanced binary tree is not suitable as an index. So what is suitable as an index --B tree.
Balanced binary tree could not take full advantage of disk read-ahead function, which is to take full advantage of the B-tree disk read-ahead function to a data structure created, that is to a B-tree index was only invented in.
Take a look at knowledge of "the principle of locality and disk read-ahead" in:
局部性原理与磁盘预读：
由于存储介质的特性，磁盘本身存取就比主存慢很多，再加上机械运动耗费，磁盘的存取速度往往是主存的几百分分之一，因此为了提高效率，要尽量减少磁盘I/O。为了达到这个目的，磁盘往往不是严格按需读取，而是每次都会预读，即使只需要一个字节，磁盘也会从这个位置开始，顺序向后读取一定长度的数据放入内存。这样做的理论依据是计算机科学中著名的局部性原理：
当一个数据被用到时，其附近的数据也通常会马上被使用。
程序运行期间所需要的数据通常比较集中。
由于磁盘顺序读取的效率很高（不需要寻道时间，只需很少的旋转时间），因此对于具有局部性的程序来说，预读可以提高I/O效率。
Clear meaning of the above. Disk read-ahead is the realization, on the theory that the principle of locality.
Why red-black tree did not take full advantage of disk-ahead, citing a blog post passage:
红黑树这种结构，h明显要深的多。由于逻辑上很近的节点（父子）物理上可能很远，无法利用局部性，所以红黑树的I/O渐进复杂度也为O(h)，效率明显比B-Tree差很多。
In other words, the use of red-black tree (balanced binary tree) structure, then each disk read-ahead Much of the data is no longer applicable data. Therefore, it fails to take a good disk read-ahead data provided. Since then a large depth (in terms of the more B-tree), so more disk IO operations performed.
Each node B-tree can store multiple keywords, it will node size to the size of the disk-page, full use of the disk read-ahead function. A node will read the entire disk each time a page is read. Which is why each node stores very multiple keywords, the depth of the tree will be very small. Furthermore disk reads operations to be performed will be very small, more of a lookup to read incoming data in memory.
Query B-tree, mainly in memory, and balanced binary tree query, occurred in the disk reads. Therefore, although the number of B-tree query query fewer than balanced binary tree, but compared to the speed of disk IO, memory can be relatively time-consuming negligible. Thus, B is more suitable as an index tree.
More suitable than the B-tree index tree structure --B +
Ratio B is more suitable as an index tree structure is a B + tree. MySQL is also used as a B + tree index. It is a variant of the B-tree, and therefore is based on the B-tree to improve. Why B + tree will be more outstanding than the B trees?
B-tree: an ordered array of multiple balanced tree +;
B + tree: an ordered array list + balancing multiple tree;
Keyword B + tree is all stored in a leaf node, the non-leaf nodes used for the index, and leaf node has a pointer to look at a leaf node. To do this optimization is to improve the performance range of access. It is this characteristic determines the B + tree is more suitable for storing external data.
Quoted passage:
走进搜索引擎的作者梁斌老师针对B树、B+树给出了他的意见（为了真实性，特引用其原话，未作任何改动）： “B+树还有一个最大的好处，方便扫库，B树必须用中序遍历的方法按序扫库，而B+树直接从叶子结点挨个扫一遍就完了，B+树支持range-query非常方便，而B树不支持。这是数据库选用B+树的最主要原因。
比如要查 5-10之间的，B+树一把到5这个标记，再一把到10，然后串起来就行了，B树就非常麻烦。B树的好处，就是成功查询特别有利，因为树的高度总体要比B+树矮。不成功的情况下，B树也比B+树稍稍占一点点便宜。
B树比如你的例子中查，17的话，一把就得到结果了，
有很多基于频率的搜索是选用B树，越频繁query的结点越往根上走，前提是需要对query做统计，而且要对key做一些变化。
另外B树也好B+树也好，根或者上面几层因为被反复query，所以这几块基本都在内存中，不会出现读磁盘IO，一般已启动的时候，就会主动换入内存。”
For example to compare.
B-tree:
For example, we want to find the keywords in the keyword range of 3-7, in the first to find a qualified digital 3, after completion of the first block where the access key, have to traverse the B-tree, get the next block, until it encounters a keyword does not meet the conditions. Ergodic process is more complicated.
B + tree (save data leaf node, all other nodes store the index):
By contrast, the scope of the query based on a number of simple B + tree. Since the leaf node has a pointer to the next leaf node, so the access block 1 to block 2, block 1 through block pointer pointing to 2. From block to block 3 2 can also pass a pointer.
He cited a blog post user comments passage:
数据库索引采用B+树的主要原因是B树在提高了磁盘IO性能的同时并没有解决元素遍历的效率低下的问题。正是为了解决这个问题，B+树应运而生。
B+树只要遍历叶子节点就可以实现整棵树的遍历。而且在数据库中基于范围的查询是非常频繁的，而B树不支持这样的操作（或者说效率太低）。As we mentioned above, in the database query range is based on very frequent, so the final choice of MySQL is B + tree index structure instead of a B-tree.
Second, the principle of indexing
A principle index
The purpose of the index is to improve query efficiency, and we used the catalog for books is a reason: to locate the chapter, and then navigate to a section in this chapter, and then find the pages. There are similar examples: dictionary, check train trips, airplane flights, etc.
Essentially: to filter through continuous narrow range of data you want to get the final results you want, while the random events become the order of events, that is to say, with this indexing mechanism, we can always use Find a way to lock the same data.
Database is the same, but obviously much more complex, because not only faced with the equivalent of a query, and the query range (>, <, between, in), fuzzy queries (like), and set the query (or), and so on. Database should choose how to deal with all kind of ways the problem? We recall the example of the dictionary, we can not put the data into segments and sub-queries it? The simplest if 1000 data, the first segment is divided into 1 to 100, 101 to 200 into the second segment, a third segment into 201-300 ...... article such check data 250, as long as the third stage to find it, all of a sudden go to 90% of invalid data in addition. But if it is a record 10 million, divided into paragraphs is better? Slightly algorithm based on the students think of the search tree, the average complexity is lgN, has good query performance. But here we have overlooked a critical issue, the complexity of each model is based on the same operating costs to consider. The database implementation is more complicated, on the one hand the data is stored on disks, on the other hand in order to improve performance, but also every part of the data can be read into memory to compute, because we know the cost of access to the disk is about one hundred thousand access memory around times, so simple search tree is difficult to meet the complex application scenarios.
Two disk IO and pre-reading
Considering the very high disk IO operation, the computer operating system to do some optimization, when the IO once, not just the current disk address of the data, but also the adjacent data is read into memory buffer , because the local pre-reading principle tells us that when a computer accesses data address when adjacent data will soon be accessed. Every time we read IO data call a (page). How much data with a specific operating system, generally for the 4k or 8k, that is, when we read the data in a fact only occur once IO, the data structure design theory for the index is very helpful.
Third, the index data structure
Any data structure is not created out of thin air, there will be its background and context, we now summarize, we need this data structure what can be done, it is actually very simple, that is: every time to find data the number of disk IO control in a small number of stages, preferably a constant magnitude. Then we wonder whether if a highly controllable multiple search trees to meet demand? In this way, b + tree came into being.
As shown above, the tree is b + a, b + tree definition can be found in B + tree , where only some of said key, which we call a blue block disk block, the block can be seen that each disk contains a few data items (shown in dark blue) and a pointer (shown in yellow), a magnetic disk comprising a block of data items 17 and 35 contain pointers P1, P2, P3, P1 represents a disk block is smaller than 17, P2 represents between 17 and 35 disk blocks, P3 that is greater than the disk block 35. Real data exists in the leaf node that is 3,5,9,10,13,15,28,29,36,60,75,79,90,99. Not only non-leaf nodes store the actual data, storing data items only guide the direction of the search, such as 17, 35 does not exist in the real data in the table.
lookup process ### b + tree
shown in FIG, 29, if you want to find the data item, then the first block will disk by the disk 1 is loaded into memory, IO occurs a case, in the memory by using the binary search is determined 29 and 17 35, the locking disk block P2 of the pointer 1, since the memory is very short time (as compared to a disk IO) is negligible, the disk block 3 is loaded into memory from the disk by the disk blocks P2 disk address pointer 1, the first occurrence secondary IO, 29 between 26 and 30, locking disk block pointer P2 3 through 8 pointer is loaded into memory disk blocks, the occurrence of the third IO, while memory do binary search to find 29, the end of the inquiry, a total of three times IO . The truth is, the layer 3 b + tree can represent millions of data, if millions of data to find only three IO, performance improvement would be great, if there is no index, each data item occurs once every IO then a total of millions of IO, obviously very, very high cost.
### b + tree nature
1 . index field to be as small as possible : by the above analysis, we know that the number of IO depends on the height h b + number, data of the current data table is assumed to N, the number of data items for each disk block is m, there ㏒ h = (m + 1) N, N when the data amount constant, the greater the m, the smaller H; m = size and the size of disk block size / data entries, disk blocks is also is the size of a data page is fixed, the smaller the space occupied if the data item, the more the number of data items, the lower the height of the tree. This is why each data item, or index fields to be as small as possible, such as int occupies 4 bytes, less than half bigint8 bytes. This is why the real requirements b + tree data into a leaf node rather than the inner nodes, once placed in the inner layer node, the data item will be a significant decline in disk blocks, resulting in increased tree. When the data item will be equal to a degenerate linear tables.
2. leftmost index matching characteristics (i.e., left to right match) : When the data item is a compound b + tree data structure, such as (name, age, sex), when, in accordance with the number b + left to right establishing search tree, such as (Zhang, 20, F) when such data to retrieve, b + tree name priority comparison determines the next search direction, if the same name and age Sex comparison in turn, and finally obtained data retrieved; but (20, F) no such data name to time, b + tree node which does not know the next check, because when building the search tree name is the first comparison factors must be according to the first name to know where to go next to search queries. For example, when (Zhang, F) to retrieve such data, b + tree name can be used to specify the search direction, but the lack of age next field, so only the name is equal to the seating of the data is found, then the matching sex F of the data, this is a very important property, namely the left-most matching characteristics of the index.
It is also often examined, for example, I define a joint index A, B, C, and if I just passed the A, B can take the index it? The answer is energy, because the principle of the far left (Baidu asked)
Add full text index (FULLTEXT) = mysql of myISAM search engine the default index type
MySQL is supported from version 3.23.23 full-text indexing and full-text search, FULLTEXT index for MyISAM tables only ; they can be used as CREATE TABLE statement from CHAR, VARCHAR or TEXT columns are part of creation, or subsequent use of ALTER TABLE or CREATE INDEX Added. //// For large data sets, enter your data without a FULLTEXT index table, and then create the index, its speed is more faster than the speed of existing FULLTEXT index data entry. But remember for large-capacity data tables, full-text indexes is a very time-consuming practice of consuming disk space.
The general index on the text field can only speed up the string (that is, the character at the beginning of the field contents) appears in front of the field content retrieval operation. If the field is stored in several, or even large blocks of text consisting of more than one word, no effect on the general index. Such searches often appear as LIKE% word% of the forms, which MySQL is very complicated for the large amount of data to be processed, the response time will be very long.
Such is the case of full-text index (full-text index) can flourish in places. Creating all the words in generating this type of index, MySQL will appear in the text is a list of query operation to retrieve the relevant data recorded in accordance with this list. That can create a full-text index along with the data table, if necessary in the future also can wait longer use the following command to add:
the ALTER TABLE table_name the ADD FULLTEXT (column1, column2)
With the full-text index, you can use the SELECT query command to retrieve those contained with one or more of the data records of a given word. Here is the basic syntax for this type of query command:
the SELECT * the FROM table_name
WHERE MATCH (column1, column2) AGAINST ( 'word1', 'word2', 'word3')
The above command will column1 and column2 field there word1, word2 and word3 data records of all check out.
Reference: Mysql Index Detailed and optimization (key and index difference)
Fourth, the index Precautions
1, do not abuse index
①, the index to improve query speed, but it will reduce the speed of updating the table, because the table is updated, mysql not only to update the data, save the data, but also to update the index, saving index
②, the index takes up disk space
2, includes a column index is not NULL values of
As long as there is a composite index containing NULL value, then this column for this index is invalid in line, so we do not let the default fields in the design database design value is NULL.
3, MySQL query with only an index
If the index where the words are used, then order by the column will not use the index
4，like
like '% aaa%' not using the index like "aaa%" can use the index
Second, select the data type index
Mysql supports many data types, select the appropriate data type for storing data has a great impact on performance.
(1) The smaller data type typically better: the smaller the type of data typically requires less disk, memory and cpu cache space, are faster.
(2) simple data types better: integer data than character, processing overhead is smaller, because of the relatively more complicated strings. In MySQL, applications built-in date and time data types, instead of the string storage time; IP address and storing integer data.
(3) try to avoid NULL: should develop as NOT NULL, unless you want to store NULL. In MySQL, the column contains a null value is difficult to query optimization, because they make the index, the index statistics, and comparison operations more complex.
Three, MySQL common indexes are: primary key index, the only index, the general index, full-text index, a composite index
1, INDEX (ordinary index): ALTER TABLE 'table_name' ADD INDEX index_name ( 'col')
The most basic index, without any restrictions
2, UNIQUE (unique index): ALTER TABLE 'table_name' ADD UNIQUE ( 'col')
And "general index" similar, is different: the value of the index columns must be unique, but allow free value.
3，PRIMARY KEY（主键索引）：ALTER TABLE 'table_name' ADD PRIMARY KEY('col')
Is a special unique index, does not allow nulls.
4, FULLTEXT (full-text indexing): ALTER TABLE 'table_name' ADD FULLTEXT ( 'col')
Only for MyISAM and InoDB, data for a larger, full-text indexing is time-consuming and space
Composite index: ALTER TABLE 'table_name' ADD INDEX index_name ( 'col1', 'col2', 'col3')
In order to further improve the efficiency of mysql can create a composite index, follow the "most left-prefix" principle. Creating composite index should be the most common (frequency) limitations do a column on the left, once decremented. Composite index used in the left-most field can be used in the index. Equivalent to the establishment of col1, col1col2, col1col2col3 three indexes
1. The data structure and algorithm principle behind MySQL index (awesome article, an article written in 2011, much ......)
2. From the B tree, B + tree, B * tree comes to R Tree (author of this article is a good powerful, visit his blog amounted to ten million)
3. Discussion on algorithms and data structures: ten balanced search tree B tree , and this blog, there are processes B and B + tree tree insert elements of a GIF, awesome, help in the understanding of B-tree and B + tree!
Reference: a step by step analysis of why the B + tree index structure suitable as
