The relationship between Server layer, engine layer, BufferPool and disk
Generally speaking, MySQL can be divided into two parts: the server layer and the storage engine layer.
1)Server layer: The Server layer includes connectors, query caches, analyzers, optimizers, executors, etc., covering MySQL Most core service functions, as well as all built-in functions (such as date, time, mathematical and encryption functions, etc.), and all cross-storage engine functions, such as stored procedures, triggers, views, etc., are implemented in this layer.
2)Engine layer: Below the Server layer are various storage engine layers, which are the core parts of MySQL and are responsible for data Storage and retrieval. Starting from MySQL version 5.5.5, MySQL sets InnoDB as the default storage engine, which uses BufferPool as a cache pool to cache data in the disk, thereby reducing IO operations on the disk.
3)BufferPool: BufferPool is a cache pool in the InnoDB engine, used to cache data on disk. It is located between the engine layer and the disk and acts as a bridge. By caching data in memory, you can reduce the number of disk accesses and improve database performance.
4)Disk: Disk is an external storage device of the computer, used to store large amounts of data. In MySQL, disks are mainly used to store database data files, log files, etc.
The Server layer and the engine layer are two relatively independent modules. To complete the work together, there will be a process of data interaction. Today we will use the Server layer to read data from the storage engine layer. This plays a key role. data interaction process.
Principle of interaction between server layer and engine layer
In the source code, each table in the database corresponds to an instance of the TABLE class. The instance has a record attribute. The record attribute is an array with 2 elements. The Server layer calls the method of the engine layer every time. When reading data, the address of the first element is passed to the engine layer in the form of table->record[0] (This address actually points to the data page stored in the BufferPool The address, that is, the Server layer indirectly accesses BufferPool by using table->record[0]). After the engine layer reads data from the disk or memory, it converts the data format of the engine layer into the data format of the Server layer, and then writes it into the memory space corresponding to this address , the Server layer can use this data to do various things (such as WHERE condition filtering, grouping, sorting, etc.).
The entire interaction process is so simple. The reason why this interaction process is so simple is because the server layer has done enough preparation work in the early stage to make this process look as simple as Baidu's search box.
To find out, let’s go back in time and trace the preparations.
Preparation
When creating a table, the Offset of each field in the record (that is, what we often call a row) will be calculated, as well as the maximum length of a record (including storage variables The number of bytes required for the length of the long field).
When we query a table for the first time, MySQL will read fields, indexes and other information from the frm file, as well as the field Offset just mentioned and the maximum length of a record.
Next, memory will be allocated for the record attribute of the TABLE class instance mentioned in Section 1 based on the maximum length of the record. The two elements of the record array, record[0] and record[1], occupy the same number of bytes as the record. The maximum length.
In the source code, each field corresponds to an instance of the Field subclass. There is a ptr attribute in the instance, which points to the memory address corresponding to each field in record[0]. For variable-length fields, the number of bytes occupied by the content length is also stored in the Field subclass instance.
After the storage engine reads a field of a record from disk or memory, it will determine the type of the field. If it is a fixed length Field, the field content is converted into the corresponding format and then written into the memory space pointed to by ptr. If it is a variable-length field, first write the content length into the memory space pointed to by ptr, and then write the field content after corresponding format conversion. The memory space after entering the content length.
Note: Fixed-length fields and variable-length fields refer to the characteristics of the data type. For details, please view MySQL data types.
Next, we will use an actual table as an example, and use a picture to show the memory layout of record[0] so that we can have an intuitive understanding.
Case Analysis
Example table:
CREATE TABLE t(`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`i1` int(10) unsigned DEFAULT '0',
`str1` varchar(32) DEFAULT '',
`str2` varchar(255) DEFAULT '',
`c1` char(11) DEFAULT '',
`e1` enum('北京','上海','广州','深圳') DEFAULT '北京',
`s1` set('吃','喝','玩','乐') DEFAULT '',
`bit1` bit(8) DEFAULT b'0',
`bit2` bit(17) DEFAULT b'0',
`blob1` blob,
`d1` decimal(10,2) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8;This is the memory layout of record[0]:
The sample table and memory layout diagram are available. Next, we will analyze the memory space corresponding to each field based on the diagram:
Field NULL value marks region
This area marks a specific record. When defining the table structure, there is no field that specifies NOT NULL. The actual content is NULL. If it is NULL, the corresponding position in this area will be set to 1. If it is not NULL, then in The corresponding position in this area will be set to 0, and the NULL mark of each field occupies 1 bit.
This field is at the beginning of record[0], so its Offset = 0. Since there are 10 fields in the example table that do not specify NOT NULL, a total of 10 bits are needed to store the NULL mark.. occupies 2 bytes in total
When the storage engine reads when fetching each field, if the field has a place in the field NULL value mark area, its corresponding position will be set to a value (0 or 1) .
id field
The type of the id field is int, a fixed-length field,occupies 4 bytes, Offset = field NULL value mark area occupies the number of bytes = 2, the ptr attribute points to Offset 2.
Storage Engine reads the content of the id field, and after converting the big and small endian storage modes (converting the data format of the engine layer to the data format of the Server layer), writes the content to the memory space pointed to by the ptr attribute.
Since in InnoDB, content is stored in big-endian mode (high-end content first, low-end mode last), and the server layer reads in little-endian mode, so the integer field content is written before writing to record[0]. Conversion between big and small endian storage modes.
i1 field
i1 field type is int, fixed-length field, occupies 4 bytes, Offset = id Offset(2) + id length ( 4) = 6, the ptr attribute points to Offset 6.
The storage engine reads the contents of the i1 field, and after converting the big and small endian storage modes, writes the contents into the memory space pointed to by the ptr attribute.
str1 field
The type of the str1 field is varchar, a variable-length field, Offset = i1 Offset(6) + i1 length(4) = 10, and the ptr attribute points to Offset 10.
When the str1 field is defined, it is specified to store 32 characters. The character set of the table is utf8. Each character will occupy up to 3 bytes (one Chinese character occupies 3 bytes) , 32 characters will occupy up to 96 bytes , 96 < 255, only 1 byte is enough The length of the content of str1 is stored, so the str1 len area occupies 1 byte. (Supplement: VARCHAR(n) stores variable-length strings (will not be padded with spaces), n represents the maximum number of characters (that is, the length of the string), and the value ranges from 0 to 65535. VARCHAR requires an additional 1 or 2 bytes to store the length of the string. The maximum length (the maximum length here refers to the maximum number of bytes occupied by n characters) is less than or equal to 255. An additional 1 byte is required, otherwise 2 words are required. section.)
The str1 field content is immediately after str1 len. Since str1 len occupies 1 byte, the offset of str1 content = 10 + 1 = 11.
Storage engine When reading the content of the str1 field, it will also read the content length of str1. It will first write the content length into the memory space pointed to by the ptr attribute, and then immediately Write the contents of str1 next to it.
str2 field
The type of the str2 field is also varchar, a variable-length field, Offset = str1 Offset(10) + str1 content length occupied bytes (1) + maximum content occupied bytes (96) = 107, the ptr attribute points to Offset 107.
str2 field is defined to store 255 characters. will occupy up to 255 * 3 = 765 bytes. requires 2 bytes. Stores the content length of str2, so the str2 len area occupies 2 bytes.
The str2 field content is stored immediately after str2 len. Since str2 len occupies 2 bytes, the Offset of str2 content = 107 + 2 = 109.
After the storage engine reads the str2 field content, it will first write the content length into the memory space pointed to by the ptr attribute, and then write the str2 content next to it.
c1 field
The type of c1 field is char, fixed-length field, Offset = str2 Offset(107) + str2 content length occupied bytes ( 2) + The maximum number of bytes occupied by the content (765) = 874, and the ptr attribute points to Offset 874.
When the c1 field is defined, it is specified to store 11 characters. will occupy up to 11 * 3 = 33 bytes.
After the storage engine reads the content of the c1 field, it writes the content into the memory space pointed to by the ptr attribute. If the actual content length of the c1 field is smaller than the maximum number of bytes of the field content, a certain number of spaces will be written next to the content just written.
For example: if the actual content length is 11 bytes and the maximum number of bytes of the field content is 33, then 22 spaces will be written after the actual content.
e1 field
e1 field type is enum, fixed length field, only 4 options, occupies 1 word Section, Offset = c1 Offset(874) + the maximum length of the content in bytes (33) = 907.
The enum type is stored as an integer in the storage engine. After the storage engine reads the content of the e1 field, it performs big-endian conversion on the content and writes the converted content into the internal space pointed to by the ptr attribute.
s1 field
s1 field type is set, fixed length field, only 4 options, occupies 1 word Section, Offset = e1 Offset(907) + e1 Length(1) = 908.
The set type is also stored as an integer in the storage engine. After the storage engine reads the content of the s1 field, it also needs to perform big-endian conversion on the content and write the converted content into the memory space pointed to by the ptr attribute.
The set field is implemented using enum and occupies up to 8 bytes, totaling 64 bits. Each option is represented by 1 bit, so a set field can have a total of 64 options. Please explain the required length of enum and set fields. If the number of options defined when creating the table is different, the length of the field may also be different (1 ~ 8 bytes), but the field length is already determined when the table is created. , so they are also fixed-length fields.
bit1 field
bit1 field type is bit, fixed length field, the length defined when creating the table represents bits, not the number of bytes, Offset = s1 Offset(908) + s1 length(1) = 909.
Bit(8) is specified when the bit1 field is defined, which means that the length of the field is 8 bits, which is 1 byte.
Bit type fields are stored as char in the storage engine. After the storage engine reads the contents of the bit1 field, it writes the contents into the memory space pointed to by the ptr attribute.
The char here refers to the char in C/C++, not the MySQL char type.
bit2 field
The type of the bit2 field is also bit, a fixed-length field. Bit(17) is defined when creating the table, occupying 3 bytes. Offset = bit1 Offset(909) + bit1 length(1) = 910.
For bit type fields, if the number of bits specified when creating the table is not an integer multiple of 8, the storage engine will add 0 in front when inserting data into the disk or memory, such as bit(17), which occupies 3 bytes, and the content is When 00010000010010011, 7 more zeros will be added in front to become 000000000010000010010011, and the content will still be like this when it is read out.
The reason why 2 bit fields are defined is to test bit type fields. When the number of bits defined is not an integer multiple of 8, will the extra bits be stored in the field value NULL mark area? Later, it was discovered that only MyISAM and NDB storage engines handle this. Bit fields in InnoDB are stored in char. When the number of bits is not an integer multiple of 8, the extra bits need to occupy 1 byte. For example: bit(17) needs to occupy 3 byte.
blob1 field
The type of blob1 field is blob, variable length field, Offset = bit2 Offset(910) + bit2 length(3) = 913.
Fields of type blob can store up to 2^16 = 65536 bytes = 64K.
After the storage engine reads the content of the blob1 field, it will allocate a memory space that can accommodate the content of the blob1 field and write the read content into the memory space. Then write the content length of the blob1 field into the memory space pointed to by the ptr attribute, occupying 2 bytes, and then write the first address of the memory space just allocated next to it, occupying 8 bytes.
Note: Only the first address of the content of the blob1 field is written to record[0], not the complete content of the blob1 field. In the example, only blob type fields are used. The actual blob types are divided into 4 types: tinyblob, blob, mediumblob, and longblob. The content lengths of these 4 types occupy 1 ~ 4 bytes respectively. In addition, one thing that needs to be explained is that tinytext, text, mediumtext, and longtext are also implemented using the corresponding blob types above, and the json type is implemented using the longblob type.
d1 field
The type of the d1 field is decimial, a fixed-length field, Offset = blob1 Offset(913) + the number of bytes occupied by the length of blob1 (2) + the number of bytes occupied by the first address of blob1 content (8) = 923.
Decimal type fields are stored in binary in the storage engine. When creating the table, the number of bytes required to store them is calculated.
After the storage engine reads the contents of the d1 field, it writes the contents into the memory space pointed to by the ptr attribute.