Implementation of Mycat Distributed Transaction

Introduction: Mycat has become a powerful open source distributed database middleware product. Facing the massive data transaction processing of enterprise applications, it is currently the best open source solution. However, if you want to keep the data in multiple machines consistent, a more conventional solution is to introduce a "coordinator" to schedule the execution of all nodes uniformly. 
This article is selected from "Distributed Database Architecture and Enterprise Practice - Based on Mycat Middleware".

　　With the increasing amount of concurrency and data, and the business has been refined to the point that it can no longer be divided according to business, we have to use a distributed database to improve the performance of the system. In a distributed system, each node is relatively independent physically, and the data operations on each node can satisfy ACID. However, the execution of transactions of other nodes cannot be known between independent nodes. If you want to keep the data in multiple machines consistent, you must ensure that all data operations on all nodes are either successfully executed or not executed at all. More conventional The solution is to introduce a "coordinator" to uniformly schedule the execution of all nodes.

XA specification

　　The X/Open organization (now the Open Group) defines a distributed transaction processing model. The X/Open DTP model (1994) includes four parts: Application Program (AP), Transaction Manager (TM), Resource Manager (RM), and Communication Resource Manager (CRM). The Transaction Manager (TM) is the transaction middleware, the Resource Manager (RM) is the database, and the Communication Resource Manager (CRM) is the message middleware. Usually, the transaction processing within a database is regarded as a local transaction, while the object of distributed transaction processing is a global transaction. Global transaction means that in a distributed transaction processing environment, multiple databases may need to work together to complete a work, which is a global transaction. Several different databases may be updated in one transaction. At this time, the commit of a database's internal operations requires not only the success of its own operations, but also the success of operations of other databases related to the global transaction. If any operation on either database fails, all operations done by all databases participating in this transaction must be rolled back. XA is the interface specification (ie interface function) between the transaction middleware and the database defined by X/Open DTP. The transaction middleware uses it to notify the database transaction start, end, commit, rollback, etc. The XA interface function is defined by the database manufacturer. According to this idea, the two-phase commit protocol and the three-phase commit protocol are derived.

two-phase commit

　　The so-called two phases are the preparation phase and the commit phase. 
　　The preparation phase means that the transaction coordinator (transaction manager) sends a preparation message to each participant (resource manager), and each participant either directly returns a failure message (such as permission verification failure), or executes the transaction locally and writes the local The redo and undo logs are not committed, and the preparation phase can be further divided into the following three steps. 
　　(1) The coordinator node asks all participant nodes whether the commit operation (vote) can be performed, and starts to wait for the response of each participant node. 
　　(2) The participant node performs all transaction operations until the query is initiated, and writes the undo information and redo information to the log. 
　　(3) Each participant node responds to the query initiated by the coordinator node. If the participant node's transaction operation actually succeeds, it returns an "agreed" message; if the participant node's transaction operation actually fails, it returns an "abort" message. 
　　The commit phase means that if the coordinator receives the participant's failure message or timeout, it directly sends a Rollback message to each participant, otherwise a Commit message is sent, and the participant executes the commit or return according to the coordinator's instructions. The roll operation releases the lock resources used by all transactions during processing. 
　　The disadvantages of two-phase commit are as follows. 
　　(1) Synchronous blocking problem. All participating nodes are transaction blocking during execution. When participants occupy public resources, other third-party nodes have to be in a blocking state when accessing public resources. 
　　(2) Single point of failure. Due to the importance of the coordinator, once the coordinator fails, the participants will continue to block. 
　　(3) The data is inconsistent. In the second stage of the two-stage submission, when a local network exception occurs after the coordinator sends the commit request to the participant or the coordinator fails during the process of sending the commit request, only Some participants have received the commit request, and some participants will perform the commit operation after receiving the commit request, and other machines that have not received the commit request cannot perform the transaction commit, so the entire distributed system will appear data inconsistency. 
　　Due to defects such as synchronization blocking, single-point problem, data inconsistency, downtime, etc., the two-phase commit has been improved on the basis of the two-phase commit, and the three-phase commit has been proposed.

three-phase commit

　　Three-phase commit (3PC), also known as three-phase commit protocol (Three-phase commit protocol), is an improved version of two-phase commit (2PC). The three-phase commit divides the preparation phase of the two-phase commit into two, so that the three-phase commit has three phases: CanCommit, PreCommit, and DoCommit. 
　　(1) CanCommit phase: The CanCommit phase of three-phase submission is actually very similar to the preparation phase of two-phase submission. The coordinator sends a commit request to the participant, and the participant returns a Yes response if the participant can submit, otherwise returns a No response. 
　　(2) PreCommit stage: The coordinator decides whether the PreCommit operation of the transaction can be recorded according to the reaction of the participants. Depending on the response, there are the following two possibilities.

• If the coordinator gets a Yes response from all participants, the transaction is executed.
• If any participant sends a No response to the coordinator, or the coordinator does not receive a response from the participant after waiting for a timeout, the execution of the transaction is interrupted.

(3) DoCommit stage: This stage performs real transaction submission, which can also be divided into two execution situations: execution submission and interrupt transaction.

　　The process of performing a commit is as follows.

• 协调者接收到参与者发送的ACK响应后，将从预提交状态进入提交状态，并向所有参与者发送doCommit请求。
• 事务提交参与者接收到doCommit请求之后，执行正式的事务提交，并在完成事务提交之后释放所有的事务资源。
• 事务提交完之后，向协调者发送ACK响应。
• 协调者接收到所有参与者的ACK响应之后，完成事务。中断事务的过程如下。
• 协调者向所有参与者发送abort请求。
• 参与者接收到 abort 请求之后，利用其在第 2 个阶段记录的 undo 信息来执行事务的回滚操作，并在完成回滚之后释放所有的事务资源。
• 参与者完成事务回滚之后，向协调者发送 ACK 消息。
• 协调者接收到参与者反馈的 ACK 消息之后，执行事务的中断。

Mycat 中分布式事务的实现

　　Mycat在1.6版本以后已经完全支持 XA 分布式强事务类型了，先通过一个简单的示例来了解Mycat中XA的用法。 
　　用户应用侧（AP）的使用流程如下： 
　　（1）set autocommit=0 
　　在应用层需要设置事务不能自动提交； 
　　（2）set xa=on 
　　在 SQL 中设置 XA 为开启状态； 
　　（3）执行 SQL 
　　　insert into travelrecord(id,name) values(1,’N’),(6000000,’A’),(321,’D’),(13400000,’C’),(59,’E’); 
　　（4）commit 或者 rollback 
　　对事务进行提交（提交成功或者回滚异常）。 
　　完整的流程图如图所示。 
 
　　Mycat 内部实现侧的实现流程如下： 
　　（1）set autocommit=0 
　　将 MysqlConnection 中的 autocommit 设置为 false； 
　　（2）set xa=on 
　　在Mycat中开启 XA 事务管理器，用 MycatServer.getInstance().genXATXID()生成 XID，用XA START XID 命令进行 XA 事务开始标记，继续拼装 SQL 业务（Mycat 会将上面的 insert 数据分片到不同的节点上），拼装 XA END XID，XA PREPARE XID 最后进行 1pc 提交并记录日志到 tm.log 中，如果 1pc 阶段有异常，则直接回滚事务 XA ROLLBACK xid。 
　　（3）在多节点 MySQL 中全部进行 2pc 提交（XA COMMIT），提交成功后，事务结束；如果有异常，则对事务进行重新提交或者回滚。 
　　Mycat 中的 XA 分布式事务的异常处理流程如下： 
　　（1）一阶段 commit 异常：如果 1pc 提交任意一个 mysql 节点无法提交或者异常，则全部节点的事务进行回滚，抛出异常给应用侧事务回滚。 
　　（2）Mycat Crash Recovery 
　　Mycat 崩溃以后，根据 tm.log 事务日志再进行重启恢复，mycat 启动后执行事务日志查找各个节点中已经 prepared 的 XA 事务，进行 commit 或者 rollback。

1. 相关类说明

　　通过用户应用侧发送 set xa = on ; SQL 开启 Mycat 内部 XA 事务管理器的功能，事务管理器将对 MySQL 数据库进行 XA 方式的事务管理，具体事务管理功能的实现代码如下：

• MySQLConnection：数据库连接。
• NonBlockingSession：用户连接 Session。
• MultiNodeCoordinator：协调者。
• CommitNodeHandler：分片提交处理。
• RollbackNodeHandler：分片回滚处理。

2. 代码解析

　　XA 事务启动的源码如下：

public class MySQLConnection extends BackendAIOConnection {
    //设置开启事务
    private void getAutocommitCommand(StringBuilder sb, boolean autoCommit) {
        if (autoCommit) {
            sb.append("SET autocommit=1;");
        } else {
            sb.append("SET autocommit=0;");
        }
    }
    public void execute(RouteResultsetNode rrn, ServerConnection sc,boolean autocommit) throws UnsupportedEncodingException {
        if(!modifiedSQLExecuted && rrn.isModifySQL()) {
            modifiedSQLExecuted = true;
        }
        //获取当前事务 ID
        String xaTXID = sc.getSession2().getXaTXID();
        synAndDoExecute(xaTXID, rrn, sc.getCharsetIndex(), sc.getTxIsolation(),autocommit);
    }
……
……//省略此处代码，建议读者参考 GitHub 仓库的 MyCAT-Server 项目的 MySQLConnection.java源码
}

　　用户应用侧设置手动提交以后，Mycat 会在当前连接中加入

　　SET autocommit=0;

　　将该语句加入到 StringBuffer 中，等待提交到数据库。 
　　用户连接 Session 的源码如下：

public class NonBlockingSession implements Session {
    ……
……//省略此处代码，建议读者参考 GitHub 仓库的 MyCAT-Server 项目的 NonBlockingSession.java 源码
}
SET XA = ON ;语句分析

　　用户应用侧发送该语句到 Mycat 中，由 SQL 语句解析器解析后交由 SetHandle 进行处理c.getSession2().setXATXEnabled (true); 
　　调用 NonBlockSession 中的 setXATXEnable d 方法设置 XA 开关启动，并生成 XID,代码如下：

public void setXATXEnabled(boolean xaTXEnabled) {
    LOGGER.info("XA Transaction enabled ,con " + this.getSource());
    if (xaTXEnabled && this.xaTXID == null) {
        xaTXID = genXATXID();
    }
}

　　另外，NonBlockSession 会接收来自于用户应用侧的 commit, 调用 commit 方法进行处理事务提交的逻辑。 
　　在 commit()方法中，首先会 check 节点个数，一个节点和多个节点分为不同的处理过程，这里只讲下多个节点的处理方法 checkDistriTransaxAndExecute(); 
　　该方法会对多个节点的事务进行提交。 
　　协调者的源码如下：

public class MultiNodeCoordinator implements ResponseHandler {
    ……
……//省略此处代码，建议读者参考 GitHub 仓库 MyCAT-Server 项目的 MultiNodeCoordinator.java 源码
}

　　在 NonBlockSession 的 checkDistriTransaxAndExecute()方法中， NonBlockSession 会话类会调用专门进行多节点协同的 MultiNodeCoordinator 类进行具体的处理，在 MultiNodeCoordinator类中，executeBatchNodeCmd 方法加入 XA 1PC 提交的处理，代码片段如下：

for (RouteResultsetNode rrn : session.getTargetKeys()) {
    ……
    if (mysqlCon.getXaStatus() == TxState.TX_STARTED_STATE){
        //recovery Log
        participantLogEntry[started] = new
        ParticipantLogEntry(xaTxId,conn.getHost(),0,conn.getSchema(),((MySQLConnection) conn).getXaStatus());
        String[] cmds = new String[]{"XA END " + xaTxId,"XA PREPARE " + xaTxId};
        if (LOGGER.isDebugEnabled()) {
            LOGGER.debug("Start execute the batch cmd : "+ cmds[0] + ";" +cmds[1]+","+"current connection:"+conn.getHost()+":"+conn.getPort());
        }
    mysqlCon.execBatchCmd(cmds);
    }
……
}

　　在 MultiNodeCoordinator 类的 okResponse 方法中，则进行 2pc 的事务提交

MySQLConnection mysqlCon = (MySQLConnection) conn;
switch (mysqlCon.getXaStatus()){
    case TxState.TX_STARTED_STATE:
    if (mysqlCon.batchCmdFinished()){
        String xaTxId = session.getXaTXID();
        String cmd = "XA COMMIT " + xaTxId;
        if (LOGGER.isDebugEnabled()) {
            LOGGER.debug("Start execute the cmd :"+cmd+",current host:"+mysqlCon.getHost()+":"+mysqlCon.getPort());
        }
        //recovery log
        CoordinatorLogEntry coordinatorLogEntry =inMemoryRepository.get(xaTxId);
        for(int i=0; i<coordinatorLogEntry.participants.length;i++){
            LOGGER.debug("[In MemoryCoordinatorLogEntry]"+coordinatorLogEntry.participants[i]);
            if(coordinatorLogEntry.participants[i].resourceName.equals(conn.getSchema())){
                coordinatorLogEntry.participants[i].txState =TxState.TX_PREPARED_STATE;
            }
        }
        inMemoryRepository.put(session.getXaTXID(),coordinatorLogEntry);
        fileRepository.writeCheckpoint(inMemoryRepository.getAllCoordinatorLogEntries());
        //send commit
        mysqlCon.setXaStatus(TxState.TX_PREPARED_STATE);
        mysqlCon.execCmd(cmd);
    }
    return;
……
}

　　分片事务提交处理的源码如下：

public class CommitNodeHandler implements ResponseHandler {
    //结束 XA
    public void commit(BackendConnection conn) {
        ……
……//省略此处代码，建议读者参考 GitHub 仓库 MyCAT-Server 项目的 CommitNodeHandler.java源码
    }
    //提交 XA
    @Override
    public void okResponse(byte[] ok, BackendConnection conn) {
        ……
……//省略此处代码，建议读者参考 GitHub 仓库的 MyCAT-Server 项目的 CommitNodeHandler.java 源码
}

　　在 Mycat 中同样支持单节点 MySQL 数据库的 XA 事务处理，在 CommitNodeHandler 类中就是对单节点的 XA 二阶段处理，处理方式与 MultiNodeCoordinator 类同，通过 commit 方法进行 1pc 的提交，而通过 okResponse 的方法进行 2pc 阶段的事务提交。 
　　分片事务回滚处理的源码如下：

public class RollbackNodeHandler extends MultiNodeHandler {
    ……
……//省略此处代码，建议读者参考 GitHub 仓库的 MyCAT-Server 项目的 RollbackNodeHandler.java 源码
}

　　在 RollbackNodeHandler 的 rollback 方法中加入了对 XA 事务的 rollback 处理，用户应用侧发起的 rollback 会在这个方法中进行处理。

for (final RouteResultsetNode node : session.getTargetKeys()) {
    ……
    //support the XA rollback
    MySQLConnection mysqlCon = (MySQLConnection) conn;
    if(session.getXaTXID()!=null) {
        String xaTxId = session.getXaTXID();
        mysqlCon.execCmd("XA END " + xaTxId + ";");
        mysqlCon.execCmd("XA ROLLBACK " + xaTxId + ";");
    }else {
    conn.rollback();
    }
……
}

　　同样，该方法会对所有的 MySQL 数据库节点发起 xa rollback 指令。 
　　本文选自《分布式数据库架构及企业实践——基于Mycat中间件》，点此链接可在博文视点官网查看。 
　　　　　　　　　　　　　　　 
　　想及时获得更多精彩文章，可在微信中搜索“博文视点”或者扫描下方二维码并关注。