MySQL - table space defragmentation method

After multiple deletes, updates, and inserts are performed on the tables in the MySQL database, the table space will become fragmented. Regularly defragmenting the table space and eliminating fragmentation can improve the performance of accessing the table space.

Check table space fragmentation
The following experiment is used to verify the impact of table space defragmentation on performance. First, check the size of the table with 1 million records,

mysql> show table status like 'sbtest1'\G
*************************** 1. row ***************************
Name: sbtest1
Engine: MyISAM
Version: 10
Row_format: Fixed
Rows: 1000000
Avg_row_length: 729
Data_length: 729000000
Max_data_length: 205195258022068223
Index_length: 20457472
Data_free: 0
Auto_increment: 1000001
Create_time: 2021-05-31 18:54:22
Update_time: 2021-05-31 18:54:43
Check_time: 2021-05-31 18:55:05
Collation: utf8mb4_0900_ai_ci
Checksum: NULL
Create_options:
Comment:
1 row in set (0.00 sec)

mysql> system ls -l /var/lib/mysql/sbtest/sbtest1.*
-rw-r----- 1 mysql mysql 729000000 May 31 08:24 /var/lib/mysql/sbtest/sbtest1.MYD
-rw-r----- 1 mysql mysql 20457472 May 31 08:25 /var/lib/mysql/sbtest/sbtest1. MYI

The show table status command is consistent with the size of the data file seen from the OS layer, and the Data_free at this time is zero.

Delete two-thirds of the records in this table,

mysql> delete from sbtest1 where id%3<>0;
Query OK, 666667 rows affected (51.72 sec)

Recollect this Check the status of the table after obtaining the statistical information of the table,

mysql> show table status like 'sbtest1'\G
*************************** 1. row ***************************
Name: sbtest1
Engine: MyISAM
Version: 10
Row_format: Fixed
Rows: 333333
Avg_row_length: 729
Data_length: 729000000
Max_data_length: 205195258022068223
Index_length: 20457472
Data_free: 486000243
Auto_increment: 1000001
Create_time: 2021-05-31 18:54:22
Update_time: 2021-05-31 19:03:59
Check_time: 2021-05-31 18:55:05
Collation: utf8mb4_0900_ai_ci
Checksum: NULL
Create_options:
Comment:
1 row in set (0.01 sec)

mysql> select 486000243/729000000;
+---------------------+
| 486000243/729000000 |
+---------------------+
| 0.6667 |
+---------------------+
1 row in set (0.00 sec)

mysql> system ls -l /var/lib/mysql/sbtest/sbtest1.*
-rw-r----- 1 mysql mysql 729000000 May 31 08:33 /var/lib/mysql/sbtest/sbtest1.MYD
-rw-r----- 1 mysql mysql 20457472 May 31 08:34 /var/lib/mysql/sbtest/sbtest1. MYI

found that two-thirds of the records in this table have been deleted, but the size of the data file is still the same as before. Because deleted records are only marked for deletion, the storage space they occupy is not released.

Perform a full table scan to see the performance,

mysql> select count(*) from sbtest1 where c<>'aaa';
+----------+ | 333333 | Found this full table scan SQL It takes 0.82 seconds. You can see the same time by looking at last_statement_latency in the sys.session view. 1 row in set (0.82 sec) +----------+ +----------+
| count(*) |

Organize table space and improve performance

Organize the table space,

mysql> alter table sbtest1 force;
Query OK, 333333 rows affected (10.73 sec)
Records: 333333 Duplicates: 0 Warnings: 0

mysql> show table status like 'sbtest1'\G
*************************** 1. row ***************************
Name: sbtest1
Engine: MyISAM
Version: 10
Row_format: Fixed
Rows: 333333
Avg_row_length: 729
Data_length: 242999757
Max_data_length: 205195258022068223
Index_length: 6820864
Data_free: 0
Auto_increment: 1000001
Create_time: 2021-05-31 19:10:35
Update_time: 2021-05-31 19:10:41
Check_time: 2021-05-31 19:10:45
Collation: utf8mb4_0900_ai_ci
Checksum: NULL
Create_options:
Comment:
1 row in set (0.48 sec)

mysql> system ls -l /var/lib/mysql/sbtest/sbtest1.*
-rw-r----- 1 mysql mysql 242999757 May 31 08:40 /var/lib/mysql/sbtest/sbtest1.MYD
-rw-r----- 1 mysql mysql 6820864 May 31 08:40 /var/lib/mysql/sbtest/sbtest1. MYI

After sorting, the hard disk space occupied is only one-third of the original size, Data_free has become zero again, and the hard disk space of the deleted records has been released.

Execute the SQL statement of the full table scan again,

mysql> select count(*) from sbtest1 where c<>'aaa';
+----------+ | 333333 | It is found that the execution speed has also increased to About three times as much. The MyISAM table is used here for testing. If the InnoDB table is used, the speed improvement is not so obvious, because the InnoDB data will be cached in the InnoDB cache. MyISAM table data is not cached by MySQL, and the OS may cache it, so it is necessary to get accurate data. For the test results, use the following command to release the system cache before each test on the Linux system. 1 row in set (0.29 sec) +----------+ +----------+
| count(*) |

# echo 3 > /proc/sys/vm/drop_caches
Using alter table force to organize the table space has the same effect as the OPTIMIZE TABLE command. This command is suitable for InnoDB and MyISAM and ARCHIVE tables for three engines. However, for InnoDB tables, the OPTIMIZE TABLE command is not supported. You can use alter table sbtest1 engine=innodb instead, for example,

mysql> alter table sbtest2 engine=innodb;
Query OK, 0 rows affected (1 min 3.06 sec)
Records: 0 Duplicates: 0 Warnings: 0

Use mysqlcheck for batch table space optimization

#列出所有已经产生碎片的表
select table_schema, table_name, data_free, engine
from information_schema.tables
where table_schema not in ('sys', 'mysql', 'performance_schema', 'information_schema', 'test')
and data_free > 0;

The following command can find the maximum 10 tables in the table space with more than 10M of free space.

mysql> select table_name,round(data_length/1024/1024) as data_length_mb, round(data_free/1024/1024) as data_free_mb
from information_schema.tables where round(data_free/1024/1024) > 10 order by data_free_mb desc limit 10;
+------------+----------------+--------------+
| TABLE_NAME | data_length_mb | data_free_mb |
+------------+----------------+--------------+
| sbtest2 | 232 | 174 |
+------------+----------------+--------------+
1 row in set (0.02 sec)

Check the space occupied by the table and the number of records

select table_schema,table_name, concat(round((data_length+index_length)/1024/1024/1024,2),‘G’) as tablesize_gb, table_rows from information_schema.tables where table_schema=‘admin’ order by tablesize_gb desc limit 5;

You can use the -o option of MySQL's own tool mysqlcheck to perform table space optimization. This tool is suitable for batch processing in scripts and can be called by crontab in Linux or scheduled tasks in Windows.

An example of table space optimization for a single table is as follows:

$ mysqlcheck -o sbtest sbtest1
You can also use the following command to perform table space optimization on all tables in a database,

$ mysqlcheck -o sbtest
can also perform table space optimization on all databases in the entire instance,

$ mysqlcheck -o --all-databases
Instead of considering how to delete data and reclaim space, it is better to consider the data deletion strategy of the table at the beginning of the design, and according to the business requirements to store useful data.

According to the hot billing project production environment, the ibd file is abnormally large. I take the opportunity to sort out the knowledge points about table fragmentation and table space shrinkage.

1. Benefits of fragmentation cleaning
Reduce IO when accessing tables, improve mysql performance, release table space and reduce disk space usage

OPTIMIZE TABLE table_name; is useful for myisam tables and innodb. The system will automatically convert it to ALTER TABLE table_name ENGINE = Innodb; This is because the essence of optimize table is to alter table, so whether it is myisam engine or innodb engine, OPTIMIZE TABLE can be used to recycle it. Table space.

Note:
1) Type of table space management: shared table space ibdata1 (5.5 default) and independent table space *.ibd file (5.6 Default, that is, innodb_file_per_table=ON)
2). The table space of each table stores the table’s own data and indexes;
3), drop table Automatically reclaim table space. After deleting a large amount of data, you can reclaim space through alter table xx engine = innodb;

2. Some usages and descriptions of OPTIMIZE (marked in red are application scenarios) 4. Commands to check related fragments from information_schema.tables where table_schema not in ('information_schema', 'mysql') and data_free > 0 order by data_free_M desc ,rate desc; select table_schema db, table_name, engine,table_rows, round(data_free/1024/1024, 2 ) data_free_M, round((data_length+index_length)/1024/1024,2) length_M , round(data_free/(data_free + data_length+index_length),2) rate -- Query and calculate fragmentation rate from information_schema.tables where table_schema not in ('information_schema', 'mysql') and data_free > 0; select table_schema db, table_name, round(data_free/1024/1024, 2) data_free, engine,table_rows, round ((data_length+index_length)/1024/1024,2) length -- List all fragmented tables 4) By default, if you use OPTIMIZE TABLE directly on the data table of the InnoDB engine, the prompt message "Table does not support optimize, doing recreate + analyze instead" may be displayed. . At this time, we can use the mysqld --skip-new or mysqld --safe-mode command to restart MySQL so that other engines can support OPTIMIZE TABLE; 3) During the operation of OPTIMIZE TABLE, MySQL will lock the table; 2) OPTIMIZE TABLE only works on MyISAM, BDB and InnoDB tables, especially the MyISAM table has the most obvious effect . In addition, not all tables need to be defragmented. Generally, only tables containing the above variable-length text data types need to be defragmented; 1) MySQL officially recommends not to defragment frequently (hourly or daily) , generally depending on the actual situation, it only needs to be sorted once a week or a month; Notes: Leave blank space when deleting, try to use the blank space when inserting, but not all the blank space is inserted into the data after deletion , it can be considered that the spaces that are not used immediately are fragments; 3. The cause of fragmentation If you have deleted a large portion of the table, or if you have modified a table with variable-length rows (containing VARCHAR, BLOB or TEXT column table) has made a lot of changes, that is, delete or update, you should use OPTIMIZE TABLE. Deleted records are kept in the linked list, and subsequent INSERT operations will reuse the old record location. You can use OPTIMIZE TABLE to repurpose unused space and defragment data files.
OPTIMIZE TABLE tbl_name [, tbl_name] ...

--查看某张表的碎片率
mysql> show table status like 't_exception_log202005';
+-----------------------+--------+---------+------------+-------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+-----------------------+
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment |
+-----------------------+--------+---------+------------+-------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+-----------------------+
| t_exception_log202005 | InnoDB | 10 | Dynamic | 61360 | 18294 | 1122566144 | 0 | 0 | 6291456 | NULL | 2020-07-31 01:56:57 | NULL | NULL | utf8_general_ci | NULL | | 异常信息日志表 |
+-----------------------+--------+---------+------------+-------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+-----------------------+

查询结果中：
Data_length : 数据的大小
Index_length : 代表索引的数量
Data_free : 代表碎片数量（指占用page的大小）

5、本地模拟*.ibd文件缩小
5.1 创建新表frag_test
create table frag_test (id int auto_increment primary key, c1 varchar(64));
5.2 利用存储过程插入数据
mysql> delimiter $$
mysql> create procedure insert_frag_test(IN START INT(10),IN max_num INT(10))
-> begin
-> declare i int default 0;
-> set autocommit = 0;
-> repeat
-> set i = i + 1;
-> insert into frag_test(id,c1) values((START+i),"this is a test i");
-> until i = max_num
-> end repeat;
-> commit;
-> end $$
mysql> delimiter ;
mysql> call insert_frag_test(1, 2000000);
mysql> call insert_frag_test(2000002, 10000000);
ERROR 1534 (HY000): Writing one row to the row-based binary log failed

这是因为内存不足导致的binlog cache size不够不能写入binlog，导致语句无法执行
在配置文件中调整binlog_cache_size和max_binlog_cache_size参数的值，改大一点
查看参数：
mysql> show variables like '%binlog_cache_size%';
+-----------------------+-----------+
| Variable_name | Value |
+-----------------------+-----------+
| binlog_cache_size | 4194304 |
| max_binlog_cache_size | 536870912 |
+-----------------------+-----------+
mysql> set global binlog_cache_size=157810688;

5.3 查看ibd文件以及碎片量
[mysql@db1 test01]# du -sh *
4.0K db.opt
12K frag_test.frm
308M frag_test.ibd
12K t1.frm
96K t1.ibd
mysql> show table status like 'frag_test';
+-----------+--------+---------+------------+---------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+-----------------+----------+----------------+---------+
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment |
+-----------+--------+---------+------------+---------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+-----------------+----------+----------------+---------+
| frag_test | InnoDB | 10 | Dynamic | 7086834 | 41 | 295469056 | 0 | 0 | 5242880 | 7104392 | 2020-10-16 10:15:43 | 2020-10-16 10:49:22 | NULL | utf8_general_ci | NULL | | |
+-----------+--------+---------+------------+---------+----------------+-------------+-----------------+---------

5.4 删除数据，查看ibd文件大小以及碎片量
mysql> delete from frag_test where id =20;
mysql> delete from frag_test where id > 24 and id < 30;
mysql> delete from frag_test where id > 200 and id < 230;
mysql> delete from frag_test where id > 2220 and id < 2560;
mysql> delete from frag_test where id > 30000 and id < 50000;

mysql> show table status like 'frag_test';
+-----------+--------+---------+------------+---------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+-----------------+----------+----------------+---------+
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment |
+-----------+--------+---------+----------------+ ----------+----------------+-------------+--------- --------+--------------+-----------+-------------- --+---------------------+---------------------+--- ---------+------------------+----------+----------- -----+---------+
| frag_test | InnoDB | 10 | Dynamic | 7066457 | 41 | 295469056 | 0 | 227540992 | 7104392 | 2020- 10-16 10:15:43 | 2020-10-16 10:49:22 | NULL | utf8_general_ci | NULL | | |
+-----------+ --------+---------+------------+---------+-------- --------+-------------+-----------------+--------- -----+----------+----------------+--------------- ------+---------------------+------------+-------- ---------+----------+----------------+---------+ Found that both Data_length and Data_free have become smaller +-----------+--------+--- ------+------------+---------+----------------+--- ----------+------------------+--------------+------ -----+----------------+------------------------+----- --------+----------------+-----------------+---------- +----------------+---------+ | frag_test | InnoDB | 10 | Dynamic | 1993875 | 50 | 100253696 | 0 | 4194304 | 7104392 | 2020-10-16 14:00:30 | NULL | utf8_general_ci | NULL | | +-----------+--------+---------+-------- ----+---------+----------------+-------------+---- -------------+-------------+-----------+--------- -------+---------------------+-------------+------ ------+------------------+----------+-------------- --+---------+ | Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment | +-----------+--------+----------+------------+--- ------+----------------+-------------+------------ -----+--------------+----------+----------------+ ---------------------+-------------+---------------------+- ----------------+----------+----------------+----- ----+ mysql> show table status like 'frag_test'; 96K t1.ibd 12K t1.frm 104M frag_test.ibd 12K frag_test.frm 4.0K db.opt [mysql@db1 test01]# du -sh * View the ibd file, significantly reduced mysql> optimize table frag_test; 5.5 , perform optimization During the process of deleting data, I saw that the size of the frag_test.ibd file has not changed
1 row in set (0.00 sec) Current data volume mysql> select count(*) from frag_test; +----------+ | count(*) | +----------+ 1 row in set (0.29 sec) Delete all mysql> delete from frag_test; +----------+ | 1999002 |

全部删除后ibd文件依旧是104M
通过mysql> show table status like 'frag_test';
+-----------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+-----------------+----------+----------------+---------+
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment |
+-----------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+-----------------+----------+----------------+---------+
| frag_test | InnoDB | 10 | Dynamic | 0 | 0 | 16384 | 0 | 0 | 103809024 | 7104392 | 2020-10-16 14:00:30 | 2020-10-16 14:06:54 | NULL | utf8_general_ci | NULL | | |
+-----------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+-----------------+----------+----------------+---------+
1 row in set (0.00 sec)
发现Data_length变小和Data_free都有变大
执行mysql> optimize table frag_test;
查看ibd文件
[root@db1 test01]# du -sh *
4.0K db.opt
12K frag_test.frm
96K frag_test.ibd
12K t1.frm
96K t1.ibd

执行mysql> show table status like 'frag_test';
*************************** 1. row ***************************
Name: frag_test
Engine: InnoDB
Version: 10
Row_format: Dynamic
Rows: 0
Avg_row_length: 0
Data_length: 16384
Max_data_length: 0
Index_length: 0
Data_free: 0
Auto_increment: 7104392

结论：
1）碎片清理可以使用optimize table table_name,手动触发数据页合并；
2）optimize table执行过程中会锁表，会产生临时表，占用一定的空间，会影响主从延迟；

补充：
数据页合并有自动触发和手动触发;
手动触发：optimize table
自动触发：依赖于数据页合并临界值（MERGE_THRESHOLD）；
MySQL InnoDB 表数据页或者二级索引页（简称数据页或者索引页）的合并与分裂对 InnoDB 表整体性能影响很大；数据页的这类操作越多，对 InnoDB 表数据写入的影响越大。
MySQL 提供了一个数据页合并临界值（MERGE_THRESHOLD），在某些场景下，可以人为介入，减少数据页的合并与分裂。在 InnoDB 表里，每个数据页默认16K 大小，默认 MERGE_THRESHOLD 值为 50，取值范围从 1 到 50，默认值即是最大值。也就是当页面记录数占比小于 50% 时，MySQL 会把这页和相邻的页面进行合并，保证数据页的紧凑，避免太多浪费。

Notice:

1. When DELETE is executed, the pages will be merged when the threshold is reached, and the merged pages will be marked as free pages, which will be reserved for future use and will not be released. Therefore, after DELETE, ibd will not become smaller (truncate will become smaller). Using optimize is equivalent to table reconstruction, so ibd will become smaller.
2. For the Innodb engine, generally optimize requires metadata locks at the beginning and end stages, and DML operations can be performed in the middle stage.

MySQL - table space defragmentation method

Supongo que te gusta