Architecture design: based on message middleware, graphic flexible transaction consistency

1. Best Effort Notice

TCC segment submission is suitable for business scenarios that require high consistency and real-time performance in a distributed architecture. In actual business, there are also services with relatively low real-time performance, such as common SMS notifications, client messages, and operating system updates. At this time, in order to reduce the complexity and pressure of the core process, the best effort notification method can be adopted to realize the management of flexible transactions.

For example, in a common third-party payment business, a message notification is generated after the processing of the local business and the payment terminal business is completed. The basic process is as follows:

• After the local business pre-processing is completed;
• Request a third-party payment service;
• The payment operation is successful and a message is sent to the account;
• Payment services call back local business;
• Local business generates system notification messages;

There are some basic features in the message scenario of the above process. After the core business processing is completed, a message notification is sent to allow failure, and within a specified time period or after a specified number of retries, message loss is allowed to exist, that is, the unreliability of the message.

In the actual payment system, when the daily reconciliation check is started, the current flow will be verified. If it is found that there is an unfinished process in the payment flow, there will be status compensation, and subsequent processing can be continued. This method is in the reconciliation Very commonly used in.

2. Reliable news

Distributed transactions are based on the realization of the final consistency of reliable messages. Since it is a reliable message, MQ must support transaction management to ensure business consistency.

1. RocketMQ transaction message

RocketMQ began to support distributed transaction messages in version 4.3, using the idea of ​​2PC to implement the commit transaction message, and at the same time adding a compensation logic to handle the two-phase timeout or failure message, as shown in the following figure:

The figure above illustrates the general scheme of transaction messages, which is divided into two processes: normal transaction message sending and submission, and transaction message compensation process.

1.1 Send and submit

(1) Send a message (half message, that is, sent but not consumed);

(2) The server responds to the message writing result;

(3) Execute local transactions according to the sent results. If the write fails, the half message is not visible to the business at this time, and the local logic is not executed;

(4) Execute Commit or Rollback according to the local transaction status (Commit operation generates a message index, and the message is visible to consumers)

1.1 Compensation process

(1) For transaction messages without Commit/Rollback (messages in the pending state), initiate a "back check" from the server;

(2) Producer receives the check-back message and checks the status of the local transaction corresponding to the check-back message;

(3) Re Commit or Rollback according to the local transaction status;

Among them, the compensation phase is used to resolve the timeout or failure of the message Commit or Rollback.

1.3 Design principle

In the main flow of RocketMQ transaction messages, how the first-stage messages are invisible to users. Among them, the biggest feature of transaction messages relative to ordinary messages is that the messages sent in one stage are invisible to users. So, how to write a message but not visible to the user? The RocketMQ transaction message method is: if the message is a half message, the subject of the original message and the message consumption queue are backed up, and then the subject is changed to RMQ_SYS_TRANS_HALF_TOPIC. Since the consumer group has not subscribed to the topic, the consumer cannot consume half-type messages, and RocketMQ will start a timed task, pull the message from the Topic RMQ_SYS_TRANS_HALF_TOPIC for consumption, and obtain a service provider according to the producer group to send back the check transaction Status request, according to the transaction status to decide whether to submit or roll back the message.

2. Final consistency

Based on the characteristics of the reliability of RocketMQ transaction messages, the final consistency of transactions under certain types of business can be achieved. Message sending consistency means that the business action that generates the message is consistent with the message sending, that is, if the business operation is successful, then the asynchronous message generated by this business operation must be sent, otherwise the business fails and rolls back, and the message is also discarded .

The process is basically as follows:

 

• Send half transaction message and cannot be consumed;
• The logic processing of the local business code is completed;
• Send a confirmation message, indicating that the message can be consumed;
• If the message producer is abnormal, cancel the overall action;

This process is mainly for the message producer. In actual development, the consumer of the message is also difficult to process. To ensure the final consistency, you will inevitably face a problem. The consumer is abnormal, the message is constantly retried, and there may be some business If the processing is successful and part of the business processing fails, then the idempotence problem of the service interface must be solved.

Three, idempotent interface

1. Introduction to idempotence

The characteristic of an idempotent operation in programming is that the effect of any number of executions is the same as that of one execution. In other words, requesting a certain resource once and multiple times will have the same effect.

In the complex asynchronous process, pay special attention to the problem of failed retry. Usually, in the payment process, every time the interface is requested, for each step of data update operation, a state query process will be preceded to determine the next data update. Whether it should be executed.

2. Idempotent interface

In the system service interface request, any explicit interface response, such as failure or success, will be easy to handle in the business process, but for example, if the request is timed out in the payment scenario, how to judge the result status of the service: client request timeout, local service timeout, request Payment timeout, payment callback timeout, client response timeout, etc., or the continuous retry mechanism based on MQ, in some business abnormal conditions, if no success is returned, the message will always be retried.

This requires the design of process-based state management. Especially under the message retry mechanism, heavy transaction control is rarely used on the retryed business interface again. After some services are executed, only one state needs to be judged, and the next message is retryed. Just try to skip, and only need to compensate the unprocessed business. Under the retry mechanism, the message will be retried until all of the business is not successfully executed until it is finally completed.