1 Introduction

Hello everyone, today I will share with you how mysql DDL is executed.

Generally speaking, MySQL is divided into DDL (definition) and DML (operation).

DDL: Data Definition Language, that is, the data definition language, the related definition operations are DDL, including: new, modified, deleted, etc.; related commands are: CREATE, ALTER, DROP, TRUNCATE truncate table content (development period, it is still quite common ), COMMENT adds a comment to the data dictionary.
DML: Data Manipulation Language, that is, the data manipulation language, that is, the operation of processing data in the database is DML, including: select, insert, update, delete, etc.; related commands are: SELECT, INSERT, UPDATE, DELETE, and LOCK TABLE, As well as the less commonly used CALL - invokes a PL/SQL or Java subroutine, EXPLAIN PLAN - parses and analyzes data access paths.

We can think that:

CREATE, ALTER, DROP, TRUNCATE, the definition related command is DDL;
SELECT, INSERT, UPDATE, DELETE, the command to manipulate data is DML;

DDL, DML difference:

DML operations can manually control the opening, committing and rolling back of transactions.
DDL operations are implicitly submitted and cannot be rolled back, so be careful!

We are familiar with a DML statement in daily development. Many developers are familiar with the execution process of sql, but how is DDL executed? Bar. In fact, it is not. If you know some pits that can avoid some ddl as much as possible, then let's take you to understand the way of DDL execution, it can be considered as a guide. If there are any mistakes, please correct me.

2 Overview

In the process of using MySQL, it is a common operation and maintenance operation to change the table structure according to the needs of the business, and these are called DDL operations. Common DDL operations include adding a new column to a table or adding an index to a column.

Our commonly used Yiwei platform provides two ways to execute DDL, including MySQL's native online DDL (online DDL) and a third-party tool pt-osc.

The following figure shows the performance comparison and description of the execution mode:

This article will briefly introduce and analyze Online DDL, which is the DDL execution tool, and pt-osc will introduce it later.

3 Introduction

The MySQL Online DDL function was officially introduced from version 5.6, and has been developed to version 8.0, which has undergone many adjustments and improvements. In fact, the INPLACE DDL method was added as early as MySQL 5.5, but due to implementation problems, INSERT, UPDATE, and DELETE operations will still be blocked, which is one of the reasons why the early version of MySQL has been complained for a long time.

Before MySQL 5.6, one of the most expensive database operations was executing DDL statements, especially ALTER statements, because when modifying a table, MySQL blocked the entire table read and write operations. For example, the specific process of DDLing Table A is as follows:

Create a new table B according to the definition of table A
Write lock on table A
Perform the operation specified by the DDL on table B
Copy data from A to B
Release A's write lock
delete table A
Rename table B to A

In the process of 2-4 above, if the data volume of table A is relatively large, the process of copying to table B will consume a lot of time and occupy additional storage space. In addition, both DDL and DML on Table A will be blocked and unable to serve because the DDL operation occupies the write lock on Table A.

In the case of huge tables, it can take hours to complete, which is bound to affect the application, so these operations need to be well planned to avoid performing these changes during peak hours. Doing DDL on large tables is a real nightmare for those who have to provide 24/7 service or have limited maintenance time.

Therefore, MySQL officials continue to enhance DDL statements. Since MySQL 5.6, more ALTER TABLE type operations have been supported to avoid data copying, and at the same time, DML operations are not blocked during the online DDL process, which is truly realized. Online DDL, that is, DML (insert, update, delete) is allowed to be executed without interrupting database services during the execution of DDL. However, not all DDL operations support online operations. In MySQL 5.7, some new features have been added on the basis of 5.6, such as: adding support for renaming indexes, supporting the increase and decrease of the length of numeric types, and supporting online increase of VARCHAR types. But the basic implementation logic and constraints have not changed much compared to 5.6.

4 Usage

ALTER TABLE tbl_name ADD PRIMARY KEY (column), ALGORITHM=INPLACE, LOCK=NONE;

The parameters ALGORITHM and LOCK can be specified in the ALTER statement to specify the algorithm mode of DDL execution and the lock control mode of DML during DDL, respectively.

ALGORITHM=INPLACE means that no table copy occurs during the execution of DDL, and DML is allowed to be executed concurrently during the process (INPLACE does not need to occupy a lot of disk I/O and CPU like COPY, which reduces the database load. At the same time, it reduces the use of buffer pool, To avoid performance problems caused by the large deletion of the original query cache in the buffer pool).
If ALGORITHM=COPY is set, DDL will be performed in the way before MySQL 5.6, using table copying, and all DML will be blocked during the process. In addition, you can also set ALGORITHEM=DAFAULT to let MySQL choose the execution mode based on the principle of ensuring DML concurrent operations as much as possible.
LOCK=NONE means that DML operations are not locked, and all DML operations are allowed in the DDL process. In addition, there are EXCLUSIVE (holds exclusive locks, blocks all requests, suitable for scenarios where DDL needs to be completed as soon as possible or the service library is idle), SHARED (allows SELECT, but blocks INSERT UPDATE DELETE, suitable for data warehouses, etc. that allow data writing input delay scenarios) and DEFAULT (according to the type of DDL, select the value of LOCK under the principle of ensuring maximum concurrency).

5 Two Algorithms

The first copy:

Create a new temporary table according to the original table definition;
Add a write lock to the original table (forbid DML, allow select);
Execute DDL on the temporary table created in step 1;
Copy the data in the original table to the temporary table;
Release the write lock of the original table;
Delete the original table and rename the temporary table to the original table.
As can be seen from the above, during the copy mode, the table needs to be locked and DML is prohibited, so it is not online. For example: delete the primary key, modify the column type, modify the character set, these operations will cause the row record format to change (the online cannot be achieved by full + incremental).

The second Inplace:

To make changes on the original table, there is no need to generate a temporary table or the process of data copying. According to whether the line record format, it can be divided into two categories:

rebuild: The table needs to be rebuilt (clustered indexes are reorganized). For example, optimize table, add index, add/delete column, modify column NULL/NOT NULL attribute, etc.;
no-rebuild: You don't need to rebuild the table, you only need to modify the metadata of the table, such as deleting the index, modifying the column name, modifying the default value of the column, modifying the self-value of the column, etc.

For the rebuild method to realize Online, it is realized by caching the DML during the DDL period, and after the DDL is completed, the DML is applied to the table. For example, to execute an alter table A engine=InnoDB; to rebuild the DDL of the table, the general process is as follows:

Create a temporary file to scan all data pages of the primary key of table A;
Generate a B+ tree with the records of table A in the data page and store it in a temporary file;
In the process of generating temporary files, all operations on A are recorded in a log file (row log);
After the temporary file is generated, the operations in the log file are applied to the temporary file, and a data file with the same logical data as Table A is obtained;
Replace the data file of Table A with the temporary file.

illustrate:

During the period of copying data to the new table, the MDL read lock is added on the original table (DML is allowed, DDL is prohibited);
Add MDL write lock to the original table during the application increment (disable DML and DDL);
The data reconstructed according to Table A is placed in tmp_file. This temporary file is created internally by InnoDB, and the entire DDL process is completed within InnoDB. For the server layer, the data is not moved to the temporary table, it is an in-place operation, which is the source of the "inplace" name.

When an error occurs or is killed, a DDL executed in the Inplace mode requires a certain rollback period. The longer the execution time, the longer the rollback time.

The DDL executed in the Copy mode needs to record the undo and redo logs in the process, and consumes the resources of the buffer pool at the same time. The efficiency is low, and the advantage is that it can be stopped quickly.

However, not all DDL operations can be executed in the way of INPLACE, and the specific support can be viewed in (Online DDL Operations).

The following are common DDL operations:

Official website support list:

6 Execution process

Online DDL mainly includes three stages, prepare stage, ddl execution stage, and commit stage. The following will mainly introduce the three stages of the ddl execution process.

1) Prepare phase: The initialization phase calculates the allowed concurrency in the DDL process according to the storage engine, user-specified operations, user-specified ALGORITHM, and LOCK. In this process, a shared metadata lock is acquired to protect the structure definition of the table.

Create new temporary frm file (not related to InnoDB).
Hold EXCLUSIVE-MDL lock, prohibit reading and writing.
According to the alter type, determine the execution method (copy, online-rebuild, online-norebuild). If it is Add Index, select online-norebuild, that is, INPLACE mode.
Update the memory object of the data dictionary.
Allocate a row_log object to log deltas (required only for rebuild types).
Generate new temporary ibd file (required only for rebuild type).
Commit the transaction and release the lock on the data dictionary.

Note: Row log is an exclusive structure, it is not redo log. It manages the storage of DML records in the form of blocks. The size of a block is controlled by the parameter innodb_sort_buffer_size, the default size is 1M, and two blocks will be applied for in the initialization phase.

2) DDL execution phase: the shared metadata lock during execution ensures that other DDLs will not be executed at the same time, but DML can be executed normally.

Downgrade the EXCLUSIVE-MDL lock to allow reading and writing (copy is not writable).
Scan the clustered index of old_table for each record rec.
Traverse the clustered index and secondary index of the new table and process them one by one.
Construct the corresponding index item according to rec
Insert the constructed index entry into the sort_buffer block sort.
Update the sort_buffer block to the new index.
Record the increments generated during the execution of ddl (required only for rebuild types)
Replay the operations in row_log to the new index (no-rebuild data is updated on the original table).
The dml operation append to the last block of row_log is generated between replaying row_log.

3) Commit stage: upgrade the shared metadata lock to exclusive metadata lock, prohibit DML, then delete the old table definition and submit the new table definition.

When the current Block is the last row_log, reading and writing are prohibited and the lock is upgraded to EXCLUSIVE-MDL.
Redo the last part of the increment in row_log.
Update the data dictionary table of innodb.
Commit the transaction (brush the redo log of the transaction).
Modify statistics.
Rename temporary idb file, frm file.
Change is complete.

The steps in the Online DDL process that take up the exclusive MDL execute very quickly, so there is almost no blocking of the DML statement.
However, other transactions can acquire the MDL before or while the DDL is executing. Due to the need to use exclusive MDL, it is necessary to wait until the other transaction holding the metadata lock is committed or rolled back before executing the above two places involving MDL.

7 Step on the pit

As mentioned earlier, MDL needs to be acquired during the execution of Online DDL. MDL (metadata lock) is a table-level lock introduced by MySQL 5.5, which is automatically added when accessing a table to ensure the correctness of reading and writing. When doing DML operations on a table, add MDL read locks; when doing DDL operations, add MDL write locks.

In order to ensure that DML can be executed concurrently in the process of executing DDL for large tables, Online DDL with ALGORITHM=INPLACE is used earlier, but there is still the risk of deadlock. The problem lies in the place where exclusive MDL is required in the online DDL process.

For example, Session 1 executes a SELECT operation in a transaction, and at this time gets the shared MDL. Since it is executed in a transaction, the shared MDL will only be released after the transaction ends.

# Session 1> START TRANSACTION;> SELECT * FROM tbl_name;# 正常执行

At this time, if Session 2 wants to perform DML operations, it only needs to obtain the shared MDL, and it can still be executed normally.

# Session 2> SELECT * FROM tbl_name;# 正常执行

But if Session 3 wants to perform DDL operation, it will block, because Session 1 has already occupied the shared MDL, and the execution of DDL needs to obtain the exclusive MDL first, so it cannot be executed normally.

# Session 3> ALTER TABLE tbl_name ADD COLUMN n INT;# 阻塞

With show processlist you can see that the ALTER operation is waiting for the MDL.

+----+-----------------+------------------+------+---------+------+---------------------------------+-----------------+
| Id | User            | Host             | db   | Command | Time | State                           | Info            |│----+-----------------+------------------+------+---------+------+---------------------------------+-----------------+
| 11 | root            | 172.17.0.1:53048 | demo | Query   |    3 | Waiting for table metadata lock | alter table ... |+----+-----------------+------------------+------+---------+------+---------------------------------+-----------------+

Since the acquisition of exclusive MDL takes precedence over shared MDL, subsequent attempts to acquire shared MDL will also be blocked.

# Session 4> SELECT * FROM tbl_name;# 阻塞

At this point, both DML and DDL will be blocked in the future until Session 1 is submitted or rolled back, and the shared MDL occupied by Session 1 is released, and subsequent operations can continue to be executed.

There are two main reasons for this problem:

The transaction in Session 1 was not committed in time, thus blocking the DDL of Session 3
Session 3 Online DDL blocks subsequent DML and DDL

For problem 1, some ORM frameworks encapsulate user statements into transaction execution by default. If the client program is interrupted and exited, and the transaction has not been committed or rolled back, the situation in Session 1 will occur. Then you can find out the thread corresponding to the unfinished transaction in infomation_schema.innodb_trx and force it to quit.

> SELECT * FROM information_schema.innodb_trx\G*************************** 1. row ***************************trx_id: 421564480355704trx_state: RUNNINGtrx_started: 2022-05-01 014:49:41trx_requested_lock_id: NULLtrx_wait_started: NULLtrx_weight: 0trx_mysql_thread_id: 9trx_query: NULLtrx_operation_state: NULLtrx_tables_in_use: 0trx_tables_locked: 0trx_lock_structs: 0trx_lock_memory_bytes: 1136trx_rows_locked: 0trx_rows_modified: 0trx_concurrency_tickets: 0trx_isolation_level: REPEATABLE READtrx_unique_checks: 1trx_foreign_key_checks: 1trx_last_foreign_key_error: NULLtrx_adaptive_hash_latched: 0trx_adaptive_hash_timeout: 0trx_is_read_only: 0trx_autocommit_non_locking: 0trx_schedule_weight: NULL1 row in set (0.0025 sec)

It can be seen that the trx_mysql_thread_id corresponding to the transaction being executed in Session 1 is 9, and then execute KILL 9 to interrupt the transaction in Session 1.
For problem 2, in the case of many queries, the blocked sessions will increase rapidly. In this case, the DDL operation can be interrupted first to prevent excessive impact on the service. You can also try to perform master-slave switching after modifying the table structure on the slave library or use third-party tools such as pt-osc.

8 Restrictions

Only works with InnoDB (syntactically it can be used with other storage engines like MyISAM, but MyISAM only allows algorithm=copy, same as the traditional approach);
No matter what kind of lock is used (NONE, shared or exclusive), it takes a short time at the beginning and end to lock the table (exclusive lock);
foreign_key_checks should be disabled to avoid table duplication when adding/removing foreign keys;
There are still some alter operations that need to copy or lock the table (the old method). For which table changes require table copying or table locking, please check the official website;
LOCK = NONE is not allowed in an alter table statement if there is an ON … CASCADE or ON … SET NULL constraint on the table;
Online DDL will be copied to the slave library (same as the master library, if LOCK = NONE, the slave library will not be locked), but the copy itself will be blocked, because the alter is executed in a single thread in the slave library, which will cause the master-slave library Latency issues.

Official reference: https://dev.mysql.com/doc/refman/5.7/en/innodb-online-ddl-limitations.html

9 Summary

This time, I will learn about the DDL, DML and differences of SQL with you, and also introduce the execution method of Online DDL.

Currently available DDL manipulation tools include pt-osc, gh-ost of github, and Online DDL, an online command to modify table structure provided by MySQL. Both pt-osc and gh-ost are implemented by copying the table, that is, creating an empty new table, and reading the records in the old table one by one and inserting them into the new table through select+insert. The difference is that the business during DDL is processed. DML operations on tables.

In MySQL 8.0, the official implementation of DDL has been redesigned. One of the biggest improvements is that DDL operations support atomic features. In addition, a new option is added to the ALGORITHM parameter of Online DDL: INSTANT, which only needs to modify the metadata in the data dictionary, without copying the data or rebuilding the table, and also without adding an exclusive MDL lock, and the original table data is not affected. The entire DDL process is completed almost instantaneously, and it will not block DML. However, the use of 8.0's INSTANT is small at present. I will introduce the 8.0's INSTANT in detail later.

In addition, the Yiwei platform also provides the execution method of pt-osc. I will share the execution method of pt-osc with you next time, so stay tuned!

Author: Liu Dengzhong

MySQL DDL execution mode - Online DDL introduction