MySQL Innodb storage engine (1)

InnoDB On-Disk Structures physical structure (disk structure)

Introduction: The concept of table space originates from the Oracle database. The initial purpose is to be able to do a good job of storage expansion.

位置：The InnoDB Storage Engine —>InnoDB On-Disk Structures —>Tablespaces

1. Shared table space (system table space)
   ① Table space storage mode: ibdata1~ibdataN

5.5+ version of the default table space storage type.

ibdata1 storage: system data, logs, undo, temporary tables, user data (index segment + data segment).
frm storage: store the data dictionary information of each table

After reading the above storage structure, then the question arises. Copying the xxx.frm and xxx.ibd files in the Innodb table to other instances, does it mean that the data table is backed up?

The fact is not. Because there is no copy of the global data dictionary. The xxx.frm and xxx.ibd files are only part of the data. There are system-related data, log information, and temporary table information in the missing ibdataN. When copying is needed, the library will copy the entire stored directory to the corresponding directory.

Can I copy the user.frm, user.MYD and user.MYI files from the myisam table to other instances? Can they be used?

can. Because myisam does not have global data dictionary information (system data information). But the permissions must be correct for the restart to take effect.

② IbdataN shared table space changes in each version

The InnoDB data dictionary consists of internal system tables that contain metadata for tracking objects such as tables, indexes, and columns in the table. The metadata is actually located in the InnoDB system tablespace. Due to historical reasons, the data dictionary metadata overlaps to some extent with the information stored in the InnoDB table metadata file (.frm file—>desc city).

Official document link: https://dev.mysql.com/doc/refman/5.7/en/innodb-architecture.html

Version 5.5
shared table space store system data and user data

System related: (global) data dictionary information, Undo rollback information, DoubleWrite buffer information, temporary table information, change buffer

(Global) Data dictionary information
Table basic structure information, status, system parameters, attributes, etc. It is equivalent to the sum of the data dictionary of all tables backed up.

Undo rollback information
The log used to undo the operation is similar to the backup before modification. If you make a mistake, back it up.

DoubleWrite buffer information double writing mechanism
Temporary table information such as order by is sorted first, then duplicated , generated in a place, processed by the system, and discarded when used up

change buffer (insert buffer) Use the Change buffer function to temporarily buffer the updated data required by the auxiliary index. When the data of the new insert needs to be queried, the merge operation is performed in memory, and the auxiliary index is the latest at this time.
User data: table data rows, table index data

Version 5.6 The
shared tablespace only stores system data, separates user data (*.ibd files are separated), and manages independent tablespaces.

System related: (global) data dictionary information, Undo rollback information, DoubleWrite buffer information, temporary table information, change buffer

Version 5.7
Based on version 5.6, the temporary table is independent (ibtmp1), and Undo can also be set to be independent.
System related: (global) data dictionary information, Undo rollback information, DoubleWrite buffer information, change buffer

Version 8.0.19
On the basis of version 5.7, the UNDO rollback information is independent by default (undo_00x), and the data dictionary is no longer centrally stored in the shared table space, the .ibd file under the mysql library.
System related: DoubleWrite buffer information, change buffer

Version 8.0.20
Based on the previous version, independent DoubleWrite buffer information.
System related: change buffer

Official document 15.6.4 Double Write Buffer:
Before MySQL 8.0.20, the doublewrite buffer storage area is located in the InnoDB system table space.
Starting from MySQL 8.0.20, the doublewrite buffer storage area is located in the doublewrite file.

Independent storage function: Destroy a file without affecting the overall situation, ensuring data security

③Shared tablespace management When setting up a new environment, initialize: mysql --initalize-insecure
. The shared tablespace is not set during initialization. The default is one: ibdata1, ibdata1:12M:autoextend ibdata1 file, the default initial size is 12M, and it will be automatically expanded if it is not enough. By default, each expansion is 64M

-Query the name and size of the tablespace:

select @@innodb_data_file_path;

Query the default expansion size:

show variables like "%increm%"；
select @@innodb_autoextend_increment；64

How to expand the shared table space?

第一历程：查看表空间的大小
# ls -lh ibdata1
-rw-r----- 1 mysql mysql 12M May  7 10:04 ibdata1

第二历程：# vim /etc/my.cnf
innodb_data_file_path=ibdata1:12M;ibdata2:100M;ibdata3:100M:autoextend

第三历程：重启数据库
# systemctl restart mysqld

# ll -h ibdata*
-rw-r----- 1 mysql mysql  12M May  7 11:29 ibdata1
-rw-r----- 1 mysql mysql 100M May  7 11:29 ibdata2
-rw-r----- 1 mysql mysql 100M May  7 11:29 ibdata3

注意：ibdata1的大小是实际的大小，否则会报错

如果不一致：innodb_data_file_path=ibdata1:8M;ibdata2:100M;ibdata3:100M:autoextend

重启报错：# /etc/init.d/mysqld restart
Shutting down MySQL.... SUCCESS!
Starting MySQL. ERROR! The server quit without updating PID file (/server/3306/data/db01.pid).


错误日志信息：
# tail -30 /server/3306/data/db01.err
[ERROR] InnoDB: The innodb_system data file './ibdata1' is of a different size 768 pages (rounded down to MB) than the 640 pages specified in the .cnf file!

错误提示得出：ibdata1文件的实际大小：
# awk "BEGIN{print 768*16/1024}"      768页*16KB/1024=12M
12

设置大小：
# awk "BEGIN{print 640*16/1024}"
10

④ Set up shared tablespaces during initialization (production recommendations)
Version 5.7 recommends setting up 2~3 shared tablespaces with a size of 512M or 1G, and the last one is customized for automatic expansion

第一历程：修改配置文件
# vim /etc/my.cnf
innodb_data_file_path=ibdata1:1024M;ibdata2:1024M;ibdata3:1024M:autoextend

第二历程：初始化
# mysqld --initialize-insecure --user=mysql --basedir=/server/app/mysql --datadir=/server/3306/data

第三历程：重启
# systemctl restart mysqld
8.0 版本建议设置1个就ok,大小512M或者1G。

2. In the independent table space
   version 5.6, for user data, separate storage management, storage table data rows and indexes
   ① View the independent table space:

select @@innodb_file_per_table;

1 The table created by a separate storage management, the file corresponding to the disk: t1.frm and t1.ibd
0 The shared table space stores the created table, the file corresponding to the disk: t1.frm, no .ibd file

演示:
当前是独立的存储管理

mysql> create database t1;
mysql> create table t1(id int);

# ll /server/3306/data/t1/
-rw-r----- 1 mysql mysql    65 May  7 12:07 db.opt
-rw-r----- 1 mysql mysql  8556 May  7 12:07 t1.frm
-rw-r----- 1 mysql mysql 98304 May  7 12:07 t1.ibd

mysql> set global innodb_file_per_table=0;
mysql> create table t2(id int);
mysql> insert into t2 values (1);

# ll /server/3306/data/t1/
-rw-r----- 1 mysql mysql    65 May  7 12:07 db.opt
-rw-r----- 1 mysql mysql  8556 May  7 12:07 t1.frm
-rw-r----- 1 mysql mysql 98304 May  7 12:07 t1.ibd
-rw-r----- 1 mysql mysql  8556 May  7 12:34 t2.frm
此时存储的数据在ibdata中。

② Use independent table space for rapid data migration
Principle:

select @@innodb_file_per_table;
1                  --->      单独的存储管理

Data migration through .ibd files.
Prerequisite: the receiving end must have ibdata files and .frm files.
Source: 3306 /test/t100w ----> Target: 3307 /test/t100w

第一历程：锁定源端t100w表，只读
mysql> lock tables test.t100w write;
mysql> show create table t100w;
CREATE TABLE `t100w` (
`id` int(11) DEFAULT NULL,
`num` int(11) DEFAULT NULL,
`k1` char(2) DEFAULT NULL,
`k2` char(4) DEFAULT NULL,
`dt` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

第二历程：目标端创建test库和t100w的空表
mysql> create database test charset="utf8mb4";
mysql> use test
mysql> CREATE TABLE `t100w` (
`id` int(11) DEFAULT NULL,
`num` int(11) DEFAULT NULL,
`k1` char(2) DEFAULT NULL,
`k2` char(4) DEFAULT NULL,
`dt` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

第三历程：单独删除目标端空的表空间文件（保留t100w的frm,ibdata1中关于t100w的系统数据）
# ll /server/3307/data/test/
-rw-r----- 1 mysql mysql    67 May  7 12:46 db.opt
-rw-r----- 1 mysql mysql  8662 May  7 12:52 t100w.frm
-rw-r----- 1 mysql mysql 98304 May  7 12:52 t100w.ibd

mysql> alter table test.t100w discard tablespace;
# ll /server/3307/data/test/
-rw-r----- 1 mysql mysql   67 May  7 12:46 db.opt
-rw-r----- 1 mysql mysql 8662 May  7 12:52 t100w.frm

第四历程：拷贝源端ibd文件到目标端目录，并设置权限    如果不是单独的表空间管理，此时是没有.ibd文件的
# cp /server/3306/data/test/t100w.ibd /server/3307/data/test/
# chown  -R mysql.mysql /server/3307/

第五历程：导入表空间
mysql> alter table test.t100w import tablespace;
mysql> select count(*) from test.t100w;
+--------------+
| count(*)     |
+--------------+
|  1000000   |
+--------------+

第六历程：解锁源端数据表
mysql> unlock tables;

Well, today's content will be introduced here first, about the migration table space to restore data, Undo table space, temporary table space, etc., follow-up articles will go into details one by one.

For more exciting content, please follow WeChat public account