This article takes you to understand the InnoDB record structure of MySQL [Part 2]

foreword

The previous article said that InnoDB has Compact, Redundant, Dynamic, and Compressed4row formats. It mainly talks about Compactthe row format, which is composed of two major parts: additional information recorded and real data recorded. It is recommended to read the previous article and the previous article. If it is important Say [ through train ] three times, don't skip the chapter. Today we mainly understand the remaining three row formats and row data overflow

1. Redundant row format

The Redundant row format is a row format used before MySQL 5.0, which means it is very old, but I still want to mention it from the perspective of learning, and everyone can understand it. The Redundant row format As shown in the picture:

For better learning, we create a demo3table here and add two pieces of data

mysql> create table demo3( c1 varchar(10), c2 varchar(10) not null, c3 char(10), c4 varchar(10), c5 varchar(1024)) charset=ascii row_format=Redundant;
Query OK, 0 rows affected (0.02 sec)

mysql> insert into demo3 values('aaaaa','bbbb','ccc','dd','e');
Query OK, 1 row affected (0.00 sec)

mysql> insert into demo3 values('eeeee','ffff',null,null,'abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdeefghijklmnopqrstuvwxyz');
Query OK, 1 row affected (0.00 sec)

The records in the table now look like this

mysql> select * from demo3;
+-------+------+------+------+------------------------------------------------------------------------------------------------------------------------------------+
| c1    | c2   | c3   | c4   | c5                                                                                                                                 |
+-------+------+------+------+------------------------------------------------------------------------------------------------------------------------------------+
| aaaaa | bbbb | ccc  | dd   | e                                                                                                                                  |
| eeeee | ffff | NULL | NULL | abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz |
+-------+------+------+------+------------------------------------------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)

Let's take a look at the differences in the Redundant line format from various aspects.

1.1 Field length offset list

CompactThe beginning of the line format is a variable-length field length list, and Redundantthe beginning of the line format is a field length offset list, which is different from the variable-length field length list in two ways:

• There are no 变长two words, that is to say, the Redundant row format will store the length information of all columns (including hidden columns) in the record in reverse order to the field length offset list
• There are two more 偏移words, which means that the method of calculating the length is different from the Compact row format, which is not so intuitive. It uses the difference between adjacent numbers to calculate the length of each column.

For example, the first row of data, let's try to analyze it:

The length of the first column row_idis 6 bytes, that is, the length of 0x06
the second column trx_idis 6 bytes, a total of 12 bytes, that is, the length of 0x0c
the third column roll_pointeris 7 bytes, a total of 19 bytes, also That is, the length of 0x13
the fourth column C1is 5 bytes, a total of 24 bytes, that is, the length of 0x18
the fifth column C2is 4 bytes, a total of 28 bytes, that is, the length of 0x1c
the sixth column C3is 10 bytes, A total of 38 bytes, that is, the length of 0x26
the seventh column C4is 2 bytes, a total of 40 bytes, that is, the length of 0x28
the eighth column C5is 1 byte, a total of 41 bytes, that is0x29

The effect of reverse order discharge is: 29 28 26 1c 18 13 0c 06, fill in the data as shown in the figure

We directly open demo3.ibdthe file to view

1.2 Record header information

The record header information in Redundant row format consists of 6 bytes and 48 binary bits. The meanings of these binary bits are as follows:

name	size (unit: bit)	describe
Reserved bit 1	1	not used
Reserved 2	1	not used
delete_mask	1	Mark whether the record is deleted
min_rec_mask	1	This mark will be added to the smallest record in each non-leaf node of the B+ tree
n_owned	4	Indicates the number of records owned by the current record
heap_no	13	Indicates the position information of the current record in the record heap
n_field	10	Indicates the number of columns in the record
1byte_offs_flag	1	Whether the offset corresponding to each column in the tag field length offset list is represented by 1 byte or 2 bytes
next_record	16	Indicates the relative position of the next record

The first line of record header information is:00 00 10 11 00 ca

Let's convert the above record from hexadecimal to binary and try to analyze it

00000000 00000000 00010000 00010001 00000000 11001010

According to these binary data, the following information can be obtained by dividing the data according to the record header structure

Reserved bit 1: 0
Reserved bit 2: 0
delete_mask: 0
min_rec_mask: 0
n_owned: 0000
heap_no: 00000000 00010 (2)
n_field: 000 0001000 (8)
1byte_offs_flag: 1 (1)
next_record: 00000000 11001010 (202)

Compared with the record header information in the Compact line format, there are two differences:

• The Redundant row format has two more attributes: n_field and 1byte_offs_flag;
• The redundant row format does not have the record_type attribute.

1.4 How to choose the value of 1byte_offs_flag

We said earlier that the offset corresponding to each column can occupy 1 byte or 2 bytes for storage, so when to use 1 byte and when to use 2 bytes? In fact, it is judged according to the total size of the real data recorded in the Redundant row format:

• When the number of bytes occupied by the actual recorded data is less than or equal to 127 (hexadecimal 0x7f, binary 01111111), the offset corresponding to each column occupies 1 byte

• When the number of bytes occupied by the recorded real data is greater than 127 but not greater than 32767 (hexadecimal 0x7fff, binary 0111111111111111), the offset corresponding to each column occupies 2 bytes

• When the real data of the record is greater than 32767, the record at this time has been stored in the overflow page, and only the first 768 bytes and the overflow page address of 20 bytes are reserved in this page (of course, in these 20 bytes Some other information is also recorded). Because the field length offset list only needs to record the offset of each column in this page, so it is enough for each column to use 2 bytes to store the offset.

In order to know whether the offset of each column is represented by 1 byte or 2 bytes when parsing records, the Redundant row format deliberately places an attribute called 1byte_offs_flag in the record header information:

• When its value is 1, it indicates that 1 byte is used for storage
• When its value is 0, it indicates that 2 bytes are used for storage

1.5 Handling of NULL values ​​in the Redundant row format

Because the Redundant row format does not have a list of NULL values, the Redundant row format does some special processing at the offsets corresponding to each column in the field length offset list—the first bit of the offset value corresponding to the column As a basis for whether it is NULL, this bit may also be referred to as a NULL bit. That is to say, when parsing a column of a record, first check whether the NULL bit of the offset corresponding to the column is 1, if it is 1, then the value of the column is NULL, otherwise it is not NULL.

This also explains why as long as the real data recorded above is greater than 127 (hexadecimal 0x7f, binary 01111111), 2 bytes are used to represent the offset corresponding to a column, mainly the first bit Is the so-called NULL bit, used to mark whether the value of the column is NULL.

But there is another point to note, for a column with a value of NULL, whether the type of the column is a fixed-length type determines the actual storage method of the NULL value:

• If the field storing the NULL value is of fixed-length type, such as CHAR(M) data type, the NULL value will also occupy the real data part of the record, and the data corresponding to this field will be filled with 0x00 bytes.
• If the field storing the NULL value is of variable-length data type, no storage space will be occupied at the real data of the record.

In addition to the above points, the Redundant line format and the Compact line format are roughly the same

Let's try to analyze the second piece of data (the number of bytes occupied by the real data is greater than 127):

The length of the first column row_idis 6 bytes, that is, the length of 0x00 0x06
the second column trx_idis 6 bytes, a total of 12 bytes, that is, the length of 0x00 0x0c
the third column roll_pointeris 7 bytes, a total of 19 bytes, also That is, the length of 0x00 0x13
the fourth column C1is 5 bytes, a total of 24 bytes, that is, the length of 0x00 0x18
the fifth column C2is 4 bytes, a total of 28 bytes, that is, the length of 0x00 0x1c
the sixth column C3is 10 bytes, A total of 38 bytes, that is 0x00 0x26, but the value is NULL, 最高位标识1after that, the value of 0x80 0x26
the seventh column C4is NULL, a total of 38 bytes, that is, the length of 0x80 0x26
the eighth column C5is 130 bytes, a total of 168 characters Festival, that is0x00 0xa8

We directly open the demo3.ibd file to see if it is correct:

In addition to the above points, the Redundant line format and the Compact line format are roughly the same.

1.6 Storage format of CHAR(M) column

We know that the Compact row format is quite troublesome when storing data in CHAR(M) type columns. It is divided into variable-length character sets and fixed-length character sets, but it is very simple in the Redundant row format, regardless of the characters used in the column. What is the set, as long as the CHAR(M) type is used, the real data space occupied is the product of the maximum number of bytes required by the character set to represent a character and M. For example, the real data space occupied by a CHAR(10) type column using the utf8 character set is always 30 bytes, and the real data space occupied by a CHAR(10) type column using the gbk character set is always 20 bytes. It can be seen from this that the column of CHAR(M) type using the Redundant row format will not generate fragmentation.

2. Row data overflow

2.1 The maximum data that VARCHAR(M) can store

We know that a column of type VARCHAR(M) can occupy up to 65535 bytes. Among them, M represents the maximum number of characters stored in this type. If we use the ascii character set, one character represents one byte. Let's see if VARCHAR(65535) is available

mysql> create table demo4 (c varchar(65535)) charset=ascii row_format=compact;
ERROR 1118 (42000): Row size too large. The maximum row size for the used table type, not counting BLOBs, is 65535. This includes storage overhead, check the manual. You have to change some columns to TEXT or BLOBs

It can be seen from the error message that MySQL has a limit on the maximum storage space occupied by a record. Except for columns of BLOB or TEXT type, the bytes occupied by all other columns (excluding hidden columns and record header information) The total length cannot exceed 65535 bytes. So the MySQL server suggests that we change the storage type to TEXT or BLOB type. In addition to the data of the column itself, this 65535 bytes also includes some other data (storage overhead). For example, in order to store a VARCHAR(M) type column, we actually need to occupy 3 parts of storage space:

• real data
• The actual data occupies the length of the byte (may occupy two bits)
• NULL value identification, if the column does not have the NOT NULL attribute (one bit), then only 65532 bytes of data can be stored at most

mysql> create table demo4 (c varchar(65532)) charset=ascii row_format=compact;
Query OK, 0 rows affected (0.06 sec)

If the column of VARCHAR(M) type does not use the ascii character set, the maximum value of M depends on the maximum number of bytes required by the character set to represent a character. In the case where the value of the column is allowed to be NULL, the gbk character set indicates that a character requires at most 2 bytes, then under this character set, the maximum value of M is 32766 (that is: 65532/2), that is to say It can store up to 32766 characters; the utf8 character set means that a character needs up to 3 bytes, so under this character set, the maximum value of M is 21844, which means it can store up to 21844 (that is: 65532/3) character.

2.2 Overflow caused by too much data in the record

Let's do a test with asciia character set demo4table and insert a piece of data:

mysql> insert into demo4 values(repeat('a',65532));
Query OK, 1 row affected (0.02 sec)

Tip:
REPEAT('a', 65532) is a function call, which means to generate a string that repeats the character 'a' 65532 times

As mentioned earlier, the basic unit of interaction between disk and memory in MySQL is the page, that is to say, MySQL uses the page as the basic unit to manage storage space, and our records will be allocated to a certain page for storage. The size of a page is generally 16KB, which is 16384 bytes, and a VARCHAR(M)type of column can store up to 65532 bytes, which may cause an embarrassing situation where a page cannot store a record

In the Compact and Reduntant row formats, for a column that occupies a large storage space, only a part of the data of the column will be stored in the real data of the record, and the remaining data will be scattered and stored in several other pages, and then the real data of the record The data place uses 20 bytes to store the addresses pointing to these pages (of course, these 20 bytes also include the number of bytes occupied by the data scattered in other pages), so that the page where the remaining data is located can be found, such as As shown in the figure:

As can be seen from the figure, for Compact and Reduntant row formats, if there is a lot of data in a column, only the first 768 bytes of data in the column and a pointer to other columns will be stored in the real data of this record. The address of the page, and then store the remaining data in other pages. This process is also called page overflow, and those pages that store more than 768 bytes are also called overflow pages.

Finally, it should be noted that not only columns of VARCHAR(M) type, but also columns of other TEXT and BLOB types will overflow when storing a lot of data.

Three, the critical point of row overflow

What is the critical point where row overflow occurs? In other words, row overflow occurs when the column stores how many bytes of data?

MySQL stipulates that at least two rows of records should be stored in a page. In addition to storing our records, each page also needs to store some additional information. The extra messy information needs to add up to 132 bytes of space (now you only need to know this number. ok), the rest of the space can be used to store records.

The additional information required for each record is 27 bytes.

• 2 bytes are used to store the length of the actual data
• 1 byte is used to store whether the column is a NULL value
• 5 bytes of header information
• 6 bytes for the DB_ROW_ID column
• 6 bytes for the DB_TRX_ID column
• 7 bytes for the DB_ROLL_PTR column

Assuming that the number of data bytes stored in a column is n, MySQL stipulates that if the column does not overflow, it needs to satisfy the following formula:

132 + 2×(27 + n) < 16384

Solving this formula leads to the solution: n < 8099. That is to say, if the data stored in a column is less than 8099 bytes, then the column will not become an overflow column, otherwise the column will need to become an overflow column. However, the conclusion of 8099 bytes is only for a demo4table with only one column. If there are multiple columns in the table, the above formula and conclusion need to be changed, so the point is: you don’t need to pay attention to this critical What is the point, just know that if the data stored in a column of our record takes up a lot of bytes, the column may become an overflow column.

Four, DYNAMIC and COMPRESSED line format

The following will introduce the other two row formats, Dynamicand Compressedthe row format. The MySQL version I am using now is 8.0, and its default row format is Dynamic

mysql> show variables like '%innodb_default_row_format%';
+---------------------------+---------+
| Variable_name             | Value   |
+---------------------------+---------+
| innodb_default_row_format | dynamic |
+---------------------------+---------+
1 row in set (0.04 sec)

mysql> set persist innodb_default_row_format = 'dynamic';
Query OK, 0 rows affected (0.01 sec)

These two line formats are similar to the Compact line format, but they are a little different when dealing with row overflow data. They will not store the first 768 bytes of the real data of the field at the real data of the record, but store all the bytes They are all stored in other pages, and only the addresses of other pages are stored at the recorded real data.

The difference between the Compressed row format and Dynamic is that the Compressed row format uses a compression algorithm to compress the page to save space.

Summarize

These two articles have a lot of theoretical knowledge, haha, but fortunately, they are not used a lot in work. It is good for everyone to know these terms, because in the explanation of MVCC later, we will also use some professional terms in this article . InnoDB defines 4 row formats, and the record structure of each row format is different. At the same time, we have also learned about the storage methods of fixed-length and variable-length data types in the record structure and row overflow. Let's come conclude

• Page is the basic unit of disk and memory interaction in MySQL, and also the basic unit of MySQL to manage storage space.

• InnoDB currently defines 4 row formats:

  • COMPACT row format:
    
  • REDUNDANT row format:

  

  • DYNAMIC and COMPRESSED row formats:

    The Dynamic and Compressed row formats are similar to the COMPACT row format, and they are a bit different when dealing with row overflow data. They do not store the first 768 bytes of the field's real data at the real data of the record, but store all the bytes. To other pages, only store the address of other pages at the real data of the record

• COMPACT row format For columns of CHAR(M) type, when the column uses a fixed-length character set, the number of bytes occupied by the column will not be added to the variable-length field length list, and if the variable-length character set is used , the number of bytes occupied by the column will also be added to the variable-length field length list.

• In the case where the value of the column is allowed to be NULL, the maximum value of M under the gbk character set is 32766, and the maximum value of M under the utf8 character set is 21844. This is all in the case of only one field in the table. Be sure to remember that the total length of bytes occupied by all columns in a row (excluding hidden columns and record header information) cannot exceed 65535 bytes!

• When there is too much data in the record and cannot fit in the current page, the excess data will be stored in other pages. This phenomenon is called row overflow. In the Compact and Redundant row formats, for columns that occupy a large storage space, only a part of the data of the column will be stored in the recorded real data, and the remaining data will be scattered and stored in several other pages, and then the recorded real data 20 bytes are used to store addresses pointing to these pages.

• The critical point of row overflow: If the data stored in a column is less than 8099 bytes, then the column will not become an overflow column, otherwise the column needs to be an overflow column. However, the conclusion of 8099 bytes is only for a table with only one column. If there are multiple columns in the table, the conclusion will be inconsistent, so the point is: you don’t need to pay attention to what the critical point is, just know that if we When the data stored in a column of a record occupies a very high number of bytes, the column may become an overflow column.

• Columns of the CHAR(M) type of the variable-length character set require at least M bytes, while VARCHAR(M) does not have this requirement. For example, for a CHAR(10) column using the utf8 character set, the length of the data stored in the column ranges from 10 to 30 bytes. Even if we store an empty string in this column, it will take up 10 bytes. This is because the byte length of the value updated in the future is greater than the byte length of the original value and less than 10 bytes. The record is directly updated instead of reallocating a new record space in the storage space, causing the original record space to become so-called fragmentation

When I finished learning this part of knowledge, I admired the depth of the author's research. Although the knowledge points are many, difficult, and complicated, the author's way of combining pictures and texts allows us to understand it well.

So far, today's study is over, I hope you will become an indestructible self
~~~

You can’t connect the dots looking forward; you can only connect them looking backwards. So you have to trust that the dots will somehow connect in your future.You have to trust in something - your gut, destiny, life, karma, whatever. This approach has never let me down, and it has made all the difference in my life

If my content is helpful to you, please 点赞, 评论,, 收藏creation is not easy, everyone's support is the motivation for me to persevere