Achieving transaction consistency and reliability is an important challenge in distributed systems. This article will introduce in detail how to use RocketMQ's semi-message mechanism to implement distributed transactions, and provide specific code examples and best practices.
1 Introduction
In distributed systems, transaction processing is a complex and critical task. Traditional ACID transactions are difficult to operate across multiple services and databases. RocketMQ is a distributed message middleware. Through its semi-message mechanism, we can achieve the consistency and reliability of distributed transactions.
2. Overview of RocketMQ half-message
2.1 Definition of half-message
The reason why using ordinary messages and order transactions cannot guarantee consistency is essentially that ordinary messages cannot have the capabilities of committing, rolling back, and unified coordination like stand-alone database transactions. The RocketMQ-based distributed transaction message function supports two-phase commit capabilities on the basis of ordinary messages. Bind the two-phase commit with the local transaction to achieve the consistency of the global commit results.
Transaction message sending is divided into two phases. In the first stage, a semi-transactional message will be sent. The semi-transactional message refers to the message that cannot be delivered temporarily. The producer has successfully sent the message to the Broker, but the Broker has not received the second confirmation of the message from the producer. At this time, the The message is marked as "temporarily undeliverable". If the message is sent successfully, the local transaction will be executed, and according to whether the local transaction is executed successfully or not, the semi-transactional message status (commit or rollback) will be sent to the Broker. Only the commit status of the semi-transactional message will be sent to downstream delivery. If the second confirmation of a certain transaction message is lost due to network interruption, producer application restart, etc., the Broker side will scan and find that a certain message has been in the "semi-transactional message" for a long time, and it needs to actively ask the message producer about it. The final state of the message (Commit or Rollback). In this way, it is finally guaranteed that the local transaction is successfully executed, and the downstream can receive the message. If the local transaction fails, the downstream cannot receive the message. In general, the consistency of upstream and downstream data is guaranteed.
2.2 How half-messages work
The transaction message sending steps are as follows:
- Producers send semi-transactional messages to RocketMQ Broker.
- After the RocketMQ Broker successfully persists the message, it returns an Ack to the producer to confirm that the message has been sent successfully. At this time, the message cannot be delivered temporarily, and it is a semi-transactional message.
- The producer starts executing local transaction logic.
- The producer submits the second confirmation result (Commit or Rollback) to the server according to the execution result of the local transaction. After the server receives the confirmation result, the processing logic is as follows:
- The result of the second confirmation is Commit: the server marks the half-transaction message as deliverable and delivers it to the consumer.
- The result of the second confirmation is Rollback: the server will roll back the transaction and will not deliver the half-transaction message to the consumer.
- In the special case of network disconnection or producer application restart, if the server does not receive the second confirmation result submitted by the sender, or the second confirmation result received by the server is Unknown, after a fixed period of time, the service The terminal will initiate a message checkback to the message producer, that is, any producer instance in the producer cluster.
- It should be noted that the server will only try the specified number of times according to the parameters, and the transaction will be forced to roll back after the number of times is exceeded. Therefore, the timeliness of the review of pending transactions is very critical, and it needs to be set according to the actual risk of the business:::
The transaction message review steps are as follows: 7. After the producer receives the message review, it needs to check the final result of the local transaction execution of the corresponding message. 8. The producer resubmits the second confirmation according to the final state of the local transaction obtained through inspection, and the server still processes the half-transaction message according to step 4.
3. Sample Code and Best Practices
3.1 RocketMQ transaction producer configuration
First, configure the relevant parameters of the RocketMQ transaction producer, including nameserver address, producer group, transaction listener, etc.
TransactionMQProducer producer = new TransactionMQProducer("transaction_producer_group");
producer.setNamesrvAddr("localhost:9876");
producer.setTransactionListener(new TransactionListenerImpl());
producer.start();3.2 Sending transaction messages
When sending a transaction message, you need to use the sendMessageInTransaction method of TransactionSendResult and specify a class that implements the TransactionListener interface.
Message message = new Message("transaction_topic", "transaction_tag", "Transaction Message".getBytes());
TransactionSendResult sendResult = producer.sendMessageInTransaction(message, null);3.3 Implementation of transaction listener
In the class that implements the TransactionListener interface, you need to write local transaction logic and message checkback logic.
public class TransactionListenerImpl implements TransactionListener {
@Override
public LocalTransactionState executeLocalTransaction(Message msg, Object arg) {
try {
// 执行本地事务逻辑
// 在这里编写执行本地事务的代码，包括数据库操作、服务调用等。
// 本地事务成功，返回 COMMIT_MESSAGE
return LocalTransactionState.COMMIT_MESSAGE;
} catch (Exception e) {
// 本地事务失败，返回 ROLLBACK_MESSAGE
return LocalTransactionState.ROLLBACK_MESSAGE;
}
}
@Override
public LocalTransactionState checkLocalTransaction(MessageExt msg) {
// 消息回查逻辑
// 在这里编写消息回查的代码，根据本地事务的状态返回 COMMIT_MESSAGE、ROLLBACK_MESSAGE 或 UNKNOW。
}
}3.4 Consumer message confirmation
On the consumer side, after receiving the message, it needs to confirm according to the state of the local transaction.
public class MessageListenerImpl implements MessageListenerConcurrently {
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
for (MessageExt msg : msgs) {
// 处理消息
// 在这里编写处理消息的代码，包括业务逻辑的执行等。
// 根据本地事务状态确认消息
if (transactionState == LocalTransactionState.COMMIT_MESSAGE) {
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
} else if (transactionState == LocalTransactionState.ROLLBACK_MESSAGE) {
return ConsumeConcurrentlyStatus.RECONSUME_LATER;
}
}
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
}4. Precautions
When using RocketMQ's half-message to implement distributed transactions, you need to pay attention to the following points:
- Set an appropriate checkback interval and times: According to business requirements and system performance, set a reasonable message checkback interval and times to ensure the eventual consistency of transactions.
- Dealing with message duplication: Due to the existence of the message review mechanism, message duplication may occur. On the consumer side, it is necessary to consider how to handle duplicate messages to avoid impacting the business.
- Guarantee message idempotence: During message processing, message idempotence needs to be guaranteed to prevent repeated processing of successful messages.
- Monitoring and alarming: Establish a suitable monitoring and alarming mechanism to detect and deal with abnormal situations in time to ensure the stability and reliability of the system.
5. Summary
This article introduces in detail how to use RocketMQ's semi-message mechanism to implement distributed transactions. Reliable processing and consistency of messages can be guaranteed by sending half-messages, executing local transaction logic, checking back messages, and acknowledging messages. At the same time, specific code examples and best practices are provided to help readers better understand and apply RocketMQ's half-message mechanism.
Using RocketMQ's semi-message to implement distributed transactions can effectively solve the challenges of traditional transactions in distributed systems and improve the reliability and consistency of the system. In practical applications, it is necessary to make appropriate adjustments and optimizations according to specific scenarios to meet the needs of the system.
=================================
If the article is helpful to you, please don't forget to add attention and like it!