Table of contents
parameter innodb_file_per_table
-
foreword
- The database takes up too much space. I deleted half of the data in the largest table. Why is the size of the table file still unchanged?
- This is related to space reclamation of database tables
- Here is a discussion on the most widely used InnoDB engine in MySQL
- An InnoDB table consists of two parts, namely: table structure definition and data
- Before MySQL 8.0, the table structure existed in a file with the suffix .frm
- The MySQL 8.0 version already allows the definition of the table structure to be placed in the system data table
- Because the table structure definition takes up very little space, the main discussion is the table data
- Next, I will explain why simply deleting table data cannot achieve the effect of table space recovery, and then introduce the correct way to reclaim space
-
parameter innodb_file_per_table
- Table data can exist either in a shared tablespace or as a separate file
- This behavior is controlled by the parameter innodb_file_per_table:
- 1- If this parameter is set to OFF, it means that the data of the table is placed in the system shared table space, that is, it is put together with the data dictionary
- 2-Setting this parameter to ON means that each InnoDB table data is stored in a file with the suffix .ibd
- Starting from MySQL 5.6.6, its default value is ON
- It is recommended to set this value to ON no matter which version of MySQL is used
- Because it is easier to manage a table stored as a file alone, and when you don’t need the table, the system will delete the file directly through the drop table command
- And if it is placed in the shared table space, even if the table is deleted, the space will not be recovered
- Therefore, it is recommended to set innodb_file_per_table to ON, and the following discussions are based on this setting.
- When deleting the entire table, you can use the drop table command to reclaim the table space
- However, more data deletion scenarios encountered are to delete certain rows. At this time, we encountered the problem at the beginning of the article: the data in the table was deleted, but the table space was not recycled
- To fully understand this problem, we must start with the data deletion process
-
Data deletion process
- Let's take a look at a schematic diagram of an index in InnoDB
- As mentioned in the previous introduction to the index, the data in InnoDB is organized in a B+ tree structure
- Suppose you want to delete the R4 record, the InnoDB engine will only mark the R4 record as deleted
- If you want to insert a record with an ID between 300 and 600 later, you may reuse this location
- However, the size of the disk file does not shrink
- Now that we know that InnoDB data is stored by page, what happens if we delete all records on a data page?
- The answer is that the entire data page can be reused
- However, the multiplexing of data pages is different from the multiplexing of records
- The multiplexing of records is limited to data that meets the range conditions
- For example, in the above example, after the record R4 is deleted, if a row with ID 400 is inserted, the space can be reused directly
- But if you insert a row with ID 800, you cannot reuse this position
- And when the entire page is removed from the B+ tree, it can be reused in any position
- Take the above picture as an example, if all records on data page page A are deleted, page A will be marked as reusable
- At this time, if you want to insert a record with ID=50 and need to use a new page, page A can be reused
- If the utilization rate of two adjacent data pages is very small, the system will combine the data on these two pages into one of the pages, and the other data page will be marked as reusable
- Furthermore, what if the data in the entire table is deleted with the delete command?
- As a result, all data pages are marked as reusable
- But on disk, the file doesn't get smaller
- Now I know that the delete command actually just marks the location of the record or the data page as "reusable", but the size of the disk file will not change
- In other words, the table space cannot be recovered through the delete command
- These can be reused, but the unused space looks like "empty"
- In fact, not only deleting data will cause holes, but inserting data will also cause holes.
- An index is compact if data is inserted in index-increasing order
- But if the data is inserted randomly, it may cause the data page of the index to split
- Assuming that page A in the above figure is already full, and another row of data needs to be inserted at this time, what will happen?
- It can be seen that since page A is full, when inserting another data with an ID of 550, you have to apply for a new page page B to save the data
- After the page split is completed, a hole is left at the end of page A (note: in fact, there may be more than one record position is a hole)
- In addition, updating the value on the index can be understood as deleting an old value and inserting a new value
- It is not difficult to understand that this will also cause voids
- In other words, tables that have undergone a large number of additions, deletions, and modifications may have holes
- Therefore, if these holes can be removed, the purpose of shrinking the table space can be achieved
- And rebuild the table, you can achieve this purpose
-
rebuild table
- Just imagine, if there is a table A now, and the space needs to be shrinked, what can be done to remove the holes in the table?
- You can create a new table B with the same structure as table A, and then read the data from table A line by line and insert them into table B in the order of increasing primary key ID
- Since table B is a newly created table, the hole in the primary key index of table A does not exist in table B
- Obviously, the primary key index of table B is more compact, and the utilization of data pages is higher
- If table B is used as a temporary table, after the operation of importing data from table A to table B is completed, replace A with table B. From the effect point of view, it will shrink the space of table A.
- Here you can use the alter table A engine=InnoDB command to rebuild the table
- Before MySQL 5.5, the execution process of this command is similar to that described above, the difference is that you don’t need to create the temporary table B, and MySQL will automatically complete the operations of dumping data, exchanging table names, and deleting old tables (Figure 1)
- Obviously, the step that takes the most time is the process of inserting data into the temporary table. If there is new data to be written to table A during this process, data will be lost
- Therefore, during the entire DDL process, there cannot be updates in table A
- In other words, this DDL is not Online
- The Online DDL introduced in MySQL 5.6 has optimized this operation process
- Briefly describe the process of rebuilding the table after the introduction of Online DDL: (Figure 2)
- 1-Create a temporary file and scan all data pages of the primary key of table A
- 2- Generate a B+ tree with the records of Table A in the data page and store it in a temporary file
- 3- During the process of generating temporary files, record all operations on A in a log file (rowlog), corresponding to the state of state2 in the figure
- 4- After the temporary file is generated, apply the operations in the log file to the temporary file to obtain a data file with the same logical data as Table A, which corresponds to the state of state3 in the figure
- 5- Replace the data file of table A with a temporary file
- It can be seen that the difference from the previous process is that due to the existence of the function of log file recording and replay operation, this scheme allows adding, deleting, and modifying operations on table A during the process of rebuilding the table
- This is where the name Online DDL comes from
- Before DDL, MDL write lock was required, so can it still be called Online DDL?
- Indeed, in the process shown above, the alter statement needs to obtain the MDL write lock when it starts, but this write lock degenerates into a read lock before actually copying the data
- Why degenerate? In order to implement Online, MDL read locks will not block addition, deletion, and modification operations
- So why not just unlock it directly?
- In order to protect yourself, prohibit other threads from doing DDL on this table at the same time
- For a large table, the most time-consuming process of Online DDL is the process of copying data to a temporary table. Additions, deletions, and modifications can be accepted during the execution of this step.
- Therefore, compared to the entire DDL process, the lock time is very short
- For business, it can be considered as Online
- It should be added that the reconstruction methods mentioned above will scan the original table data and build temporary files
- For very large tables, this operation consumes IO and CPU resources.
- Therefore, if it is an online service, the operation time must be carefully controlled
- If you want a safer operation, it is recommended to use GitHub's open source gh-ost to do it
-
Online and inplace
- Speaking of Online, I need to clarify the difference between it and another confusing concept inplace related to DDL.
- In Figure 1, the storage location where the data in Table A is exported is called tmp_table
- This is a temporary table created at the server layer
- In Figure 2, the data reconstructed from table A is placed in "tmp_file", which is a temporary file created internally by InnoDB
- The entire DDL process is done inside InnoDB
- For the server layer, it is an "in-place" operation without moving the data to the temporary table, which is the source of the name "inplace"
- So, now I ask you, if you have a 1TB table, and the current disk size is 1.2TB, can you do an inplace DDL?
- the answer is no
- Because, tmp_file also takes up temporary space
- The statement alter table t engine=InnoDB to rebuild the table actually implies:
- When ALGORITHM=copy is used, it means that the copy table is forced, and the corresponding process is the operation process in Figure 1
- But in this way, you may think, do inplace and Online mean the same thing?
- In fact, it is not, it just happens to be the case in the logic of rebuilding the table
- For example, if you want to add a full-text index to a field of an InnoDB table, the writing method is:
- This process is inplace, but it will block the addition, deletion and modification operations, which is not Online
- If we say what is the relationship between these two logics, it can be summarized as:
- 1- If the DDL process is Online, it must be inplace
- 2- The reverse is not necessarily the case, that is to say, the inplace DDL may not be Online
- As of MySQL 8.0, this is the case when adding full-text indexes (FULLTEXTindex) and spatial indexes (SPATIAL index)
- extend it again
- The difference between using optimizetable, analyze table and alter table to rebuild the table
- Let me briefly explain here
- Starting from MySQL 5.6, alter table t engine = InnoDB (that is, recreate) defaults to the process shown in Figure 2 above
- analyze table t is not actually rebuilding the table, but re-stating the index information of the table without modifying the data. MDL read lock is added in this process
- optimize table t 等于recreate+analyze