How to Trace an XA Transaction Holding a Metadata Lock

APPLIES TO:

MySQL Server - Version 8.0 and later
Information in this document applies to any platform.

GOAL

Find out why a DDL statement hangs waiting for a metadata lock and how to free it.

SOLUTION

The issue is first noticed when an attempt is made to perform a DDL statement such as ALTER TABLE and it hangs with a metadata lock.
The SHOW PROCESS LIST will show details like:

All the processes are showing asleep (or other expected thread states) except for the ALTER TABLE which is waiting for a table metadata lock.
From this information, the schema and tablename are known, so tracking that table name is the best method.
Looking at the performance_schema metadata_locks table for information relating to the specific table shows:

mysql> select * from performance_schema.metadata_locks;
+-------------+--------------------+--------------+-------------+-----------------------+---------------------+---------------+-------------+--------------------+-----------------+----------------+
| OBJECT_TYPE | OBJECT_SCHEMA      | OBJECT_NAME | COLUMN_NAME | OBJECT_INSTANCE_BEGIN | LOCK_TYPE | LOCK_DURATION | LOCK_STATUS | SOURCE | OWNER_THREAD_ID | OWNER_EVENT_ID |
+-------------+--------------------+--------------+-------------+-----------------------+---------------------+---------------+-------------+--------------------+-----------------+----------------+
| TABLE       | mydb         | mytable      | NULL | 687573        | SHARED_WRITE        | TRANSACTION | GRANTED | xa.cc:278 | 1       | 3 |
| GLOBAL      | NULL         | NULL         | NULL | 1399048092256 | INTENTION_EXCLUSIVE | STATEMENT   | GRANTED | sql_base.cc:5398 | 113920 | 26 |
| BACKUP LOCK | NULL         | NULL         | NULL | 1399048631520 | INTENTION_EXCLUSIVE | TRANSACTION | GRANTED | sql_base.cc:5405 | 113920 | 26 |
| SCHEMA      | mydb               | NULL         | NULL | 1399048393424 | INTENTION_EXCLUSIVE | TRANSACTION | GRANTED | sql_base.cc:5385 | 113920 | 26 |
| TABLE       | mydb               | mytable      | NULL | 1399048528560 | SHARED_UPGRADABLE   | TRANSACTION | GRANTED | sql_parse.cc:6052 | 113920 | 26 |
| TABLESPACE | NULL               | mydb/mytable | NULL | 1399048969312 | INTENTION_EXCLUSIVE | TRANSACTION | GRANTED | lock.cc:795       | 113920 | 26 |
| SCHEMA      | mydb               | NULL         | NULL | 1399048090448 | INTENTION_EXCLUSIVE | STATEMENT | GRANTED | sql_table.cc:1117 | 113920 | 26 |
.....
| TABLE       | mydb               | #sql-33bf_13 | NULL | 1399048084640 | EXCLUSIVE           | STATEMENT   | GRANTED | sql_table.cc:16093 | 113920 | 26 |
| TABLE       | mydb               | mytable    | NULL | 1399048094192 | EXCLUSIVE           | TRANSACTION | PENDING | mdl.cc:3696       | 113920 | 26 |
| TABLE       | performance_schema | metadata_locks | NULL | 139903664510304 | SHARED_READ     | TRANSACTION | GRANTED | sql_parse.cc:6052 | 113921 | 33 |
+-------------+--------------------+--------------+-------+---------------+---------------------+---------------+-------------+--------------------+-----------------+----------------+
17 rows in set (0.00 sec)

It can be seen there are three main transactions that are happening.
The first is event id 3 which is a SHARED_WRITE lock on the table. The source field shows that this even is an XA transaction, and was probably there when the server was started, since the thread id and event id are below 10 (first lot of numbers assigned).

The second event id is 26 which has all the locks related to the transaction that is currently attempting the ALTER TABLE id.
These are based around schema, tablespace and table locks at the statement and/or transaction levels.
There can be any number of lines that relate to this, depending on the complexity of the query and the number of tables being affected.

The last event id of 33 is the SELECT to get the metadata locks information from the performance_schema database.

Note: Just because you don't use XA transactions does not mean they have not occurred in the underlying system.

So given the above, it suggests that there is an XA transaction that is holding the lock. Can this be confirmed in the InnoDB engine status reports?
Checking what transactions are showing in the SHOW ENGINE INNODB STATUS report shows:

---TRANSACTION 59125891137, ACTIVE (PREPARED) 341098 sec recovered trx
3 lock struct(s), heap size 1136, 1 row lock(s), undo log entries 2

This is the longest active query of 341098 seconds, but there are some features that are strange.
The first point is that there is no MySQL thread id which indicates that the thread is not exposed to the normal monitoring.
The status is 'recovered trx', however the process is not complete as there is still 2 undo log entries and the locks are still holding.
This is an example of what a lost XA transaction will look like in the InnoDB status report.

The next step is to recover the XA transaction ID which is done using:

mysql> XA RECOVER CONVERT XID;
+----------+--------------+--------------+------------------------------------------------------------------------------------------+
| formatID | gtrid_length | bqual_length | data |
+----------+--------------+--------------+------------------------------------------------------------------------------------------+
| 1 | 40 | 3 | 0x32356531656365632D393938622D343931372D623534352D6566666635353430613531663A313036313036 |
+----------+--------------+--------------+------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

The data contains information regarding the xid value that is needed.
To construct the xid value, construct the following variables:

mysql> SET @data = '32356531656365632D393938622D343931372D623534352D6566666635353430613531663A313036313036',
@gtrid_length = 40,
@bqual_length = 3,
@formatId =1;

Then run the following:

mysql> SELECT CONCAT('0x', SUBSTRING(@data, 1, @gtrid_length*2),',',IF(@bqual_length=0, '', CONCAT('0x', SUBSTRING(@data, @gtrid_length*2+1, @bqual_length*2))),',',@formatID) AS xrid;
+-----------------------------------------------------------------------------------------------+
| xrid |
+-----------------------------------------------------------------------------------------------+
| 0x32356531656365632D393938622D343931372D623534352D6566666635353430613531663A313036,0x313036,1 |
+-----------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

The output will be the xid recovery value that you can use in your XA command to ROLLBACK or COMMIT. For example:

mysql> XA ROLLBACK 0x32356531656365632D393938622D343931372D623534352D6566666635353430613531663A313036,0x313036,1;

REFERENCES

NOTE:2651095.1 - Why XA RECOVER CONVERT XID report a pending transaction but this transaction can´t be found in binlog ?
NOTE:1365549.1 - What Does Thread Status "Waiting for table metadata lock" in the MySQL Server Processlist Mean?
NOTE:1473704.1 - Metadata Locking -- Which Connection is Holding on to the Lock?
NOTE:1494002.1 - How to re-format XA RECOVER to use for XA ROLLBACK