table of Contents
1. An overview of the eventual consistency of reliable messages
Second, the solution [local message table solution]
Third, the solution [RocketMQ transaction message program]
Four, RocketMQ realizes reliable message final consistent transaction
1. An overview of the eventual consistency of reliable messages
Reliable message eventual consistency scheme means that when the transaction initiator completes the local transaction and sends a message, the transaction participant (message consumer) must be able to receive the message and process the transaction successfully. This scheme emphasizes that as long as the message is sent to the transaction participant The final affairs of the parties must be consistent . This program is completed by using message middleware, as shown in the following figure:
The transaction initiator (message producer) sends the message to the message middleware, the transaction participant receives the message from the message middleware, between the transaction initiator and the message middleware, and between the transaction participant (message consumer) and the message middleware All communication is through the network , and the uncertainty of network communication will cause distributed transaction problems. Therefore, the eventual consistency of reliable messages must solve the following problems:
[1] The atomicity of local transactions and message sending : the transaction initiator must send the message after the local transaction is successfully executed, otherwise the message is discarded. That is, to achieve the atomicity of local transactions and message sending , either both succeed or both fail. The atomicity of local transactions and message sending is the key issue for realizing the eventual consistency of reliable messages . Let's try this kind of operation first, send the message first, and then operate the database : in this case, the consistency of the database operation and the message sent cannot be guaranteed, because the message may be sent successfully and the database operation failed.
begin transaction；
//1.发送MQ
//2.数据库操作
commit transation;The second solution is to perform database operations first, and then send messages : in this case, there seems to be no problem. If the MQ message fails to be sent, an exception will be thrown, causing the database transaction to roll back. but:
- If it is a timeout exception , the database is rolled back, but the MQ has actually been sent normally, which will also cause inconsistencies.
- If the message is sent successfully, the JVM suddenly hangs when the transaction is finally committed, and the transaction is not successfully committed, resulting in inconsistent data between the two systems.
begin transaction；
//1.数据库操作
//2.发送MQ
commit transation;[2] Reliability of message received by transaction participants : The transaction participants must be able to receive messages from the message queue. If the message reception fails, they can receive messages repeatedly .
[3] The problem of repeated message consumption: due to the existence of step 2, if a consumer node times out but the consumption is successful, the message middleware will repeatedly deliver the message at this time, resulting in repeated consumption of the message. To solve the problem of repeated consumption of messages, it is necessary to realize the idempotence of the method of the transaction participants .
Second, the solution [local message table solution ]
The local message table solution was originally proposed by eBay. The core of this solution is to ensure the consistency of data service operations and messages through local transactions , and then send the message to the message middleware through a timed task, and wait until the message is successfully sent to the consumer. Delete the message. The following is an example of registering to send points to illustrate: the following example has two microservice interactions, user service and point service, user service is responsible for adding users, and point service is responsible for adding points. 
[ Interaction process is as follows ] : [1] User registration: User service adds new users in local affairs and adds "point message log". (The user table and message table are guaranteed to be consistent through local transactions.) Below is the pseudo code. In this case, the local database operation and the stored integral message log are in the same transaction, and the local database operation and the recording message log operation are atomic.
begin transaction；
//1.新增用户
//2.存储积分消息日志
commit transation;[ 2 ] Scheduled task scan log: How to ensure that messages are sent to the message queue? After the first step, the message has been written to the message log table, and an independent thread can be started to scan the messages in the message log table regularly and send it to the message middleware. The message log is deleted after the message middleware reports that the sending is successful, otherwise Wait for the scheduled task to try again in the next cycle.
[3] Consumer news: How to ensure that consumers will be able to consume news? Here you can use MQ's ack ( message confirmation ) mechanism. Consumers monitor MQ. If the consumer receives the message and the business processing is completed, it sends an ack (message confirmation) to MQ, which means that the consumer has completed the normal consumption of the message, and MQ Will no longer push messages to consumers, otherwise consumers will continue to retry to send messages to consumers. The point service receives the "add points" message and starts to increase points. After the points increase successfully, the message middleware will respond with an ack, otherwise the message middleware will deliver the message repeatedly. Since messages will be delivered repeatedly, the "adding points" function of the point service needs to be idempotent .
Third, the solution [RocketMQ transaction message program ]
RocketMQ is a distributed messaging middleware from Alibaba. It was open sourced in 2012 and officially became the top Apache project in 2017. It is understood that all messaging products of Alibaba Group, including messaging products on Alibaba Cloud and acquired subsidiaries, run on RocketMQ, and RocketMQ has achieved eye-catching performance during the Double Eleven promotion in recent years. Apache RocketMQ version 4.3 or later officially supports transaction messages , providing convenient support for distributed transaction implementation. RocketMQ transaction messages designed primarily to solve atoms problem message with the local transactional execution Producer end , RocketMQ design bidirectional communication capability broker with the producer side , so that the broker born as a transaction coordinator exists ; stored mechanism RocketMQ itself provides Provides persistence capabilities for transaction messages ; RocketMQ's high-availability mechanism and reliable message design ensure that transactional messages can still ensure the final consistency of the transaction when the system is abnormal. After RocketMQ 4.3, a complete transaction message is implemented, which is actually an encapsulation of the local message table . The local message table is moved inside the MQ to solve the atomic problem of message sending and local transaction execution on the Producer side.
[The execution process is as follows ] : In order to facilitate understanding, we also use the example of registering to send points to describe the entire process. Producer is the sender of MQ. In this case, it is user service and is responsible for adding new users. The MQ subscriber is the message consumer. In this example, it is the point service and is responsible for adding points.
[1] Producer sends transaction message: Producer (MQ sender) sends transaction message to MQ Server, MQ Server marks the message status as Prepared (prepared state), note that this message consumer (MQ subscriber) cannot consume at this time Arrived. In this example, the Producer sends the "add points message" to the MQ Server.
[2] MQ Server responds to the message sent successfully: MQ Server receives the message sent to by the Producer, and responds with a successful response. Indicates that MQ has received the message .
[3] Producer executes local transactions: Producer executes business code logic and is controlled by local database transactions. In this example, the Producer performs the add user operation.
[4] Message delivery: If the Producer's local transaction is successfully executed, it will automatically send a commit message to MQServer. After receiving the commit message , MQ Server will mark the "Increase Points Message" status as consumable, At this time, the MQ subscriber (point service) consumes messages normally. If the Producer's local transaction execution fails, it will automatically send a rollback message to MQ Server, and MQ Server will delete the "Increase Points Message" after receiving the rollback message . MQ subscribers (point service) consume messages, and if the consumption is successful , they will respond to MQ with ack , otherwise they will receive messages repeatedly. Here ack automatically responds by default, that is, automatically responds to ack when the program executes normally.
[5] Transaction review: If the execution side hangs or times out during the execution of the local transaction on the Producer side, MQ Server will continuously query other Producers in the same group to obtain the transaction execution status. This process is called transaction review . MQ Server will decide whether to deliver the message based on the result of the transaction review. The above main process has been implemented by RocketMQ. On the user side, the user needs to implement local transaction execution and local transaction back-check methods, so only the execution status of the local transaction (maintain the local transaction status table) is enough . RoacketMQ provides RocketMQLocalTransactionListener interface:
public interface RocketMQLocalTransactionListener {
/**发送prepare消息成功此方法被回调，该方法用于执行本地事务
* @param msg 回传的消息，利用transactionId即可获取到该消息的唯一Id
* @param arg 调用send方法时传递的参数，当send时候若有额外的参数可以传递到send方法中，这里能获取到
* @return 返回事务状态，COMMIT：提交 ROLLBACK：回滚 UNKNOW：回调
*/
RocketMQLocalTransactionState executeLocalTransaction(Message msg, Object arg);
/**@param msg 通过获取transactionId来判断这条消息的本地事务执行状态
* @return 返回事务状态，COMMIT：提交 ROLLBACK：回滚 UNKNOW：回调
*/
RocketMQLocalTransactionState checkLocalTransaction(Message msg);
}[6] Send transaction messages: The following is the API provided by RocketMQ for sending transaction messages:
TransactionMQProducer producer = new TransactionMQProducer("ProducerGroup");
producer.setNamesrvAddr("127.0.0.1:9876");
producer.start();
//设置TransactionListener实现
producer.setTransactionListener(transactionListener）；
//发送事务消息
SendResult sendResult = producer.sendMessageInTransaction(msg, null);Fourth, RocketMQ realizes the ultimate consistent transaction of reliable messages
[ Business description ] Through RocketMQ middleware, it realizes the final consistent distributed transaction of reliable messages and simulates the transfer transaction process of two accounts. The two accounts are in different banks (Zhang San in bank1 and Li Si in bank2). Bank1 and bank2 are two microservices. The transaction process is that Zhang San transfers a specified amount to Li Si. The above transaction steps, Zhang San deducting the amount and sending a transfer message to bank2, the two operations must be an integral transaction.
[ Program components ]:
Database: MySQL-5.7.25
Including two databases bank1 and bank2.
JDK：64位 jdk1.8.0_201
rocketmq server: RocketMQ-4.5.0
rocketmq client: RocketMQ-Spring-Boot-starter.2.0.2-RELEASE
Microservice framework: spring-boot-2.1.3, spring-cloud-Greenwich.RELEASE
The relationship between microservices and databases:
dtx/dtx-txmsg-demo/dtx-txmsg-demo-bank1 Bank 1, operate Zhang San account, connect to database bank1
dtx/dtx-txmsg-demo/dtx-txmsg-demo-bank2 Bank 2, operate Li Si account, connect to database bank2
[ Core code ] : The program technical structure is as follows: 
[The interaction process is as follows ] :
[1] Bank1 sends a transfer message to MQ Server;
[2] Bank1 performs local transactions and
deducts the amount; [3] Bank2 receives messages, performs local transactions, and adds the amount;
[ Database ] : Add de_duplication in bank1 and bank2 databases , Transaction record table (de-duplication table), used for transaction idempotent control .
DROP TABLE IF EXISTS `de_duplication`;
CREATE TABLE `de_duplication` (
`tx_no` varchar(64) COLLATE utf8_bin NOT NULL,
`create_time` datetime(0) NULL DEFAULT NULL,
PRIMARY KEY (`tx_no`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8 COLLATE = utf8_bin ROW_FORMAT = Dynamic;[ Version dependency ] : The version of rocketmq-spring-boot-starter is specified in the parent project
<dependency>
<groupId>org.apache.rocketmq</groupId>
<artifactId>rocketmq-spring-boot-starter</artifactId>
<version>2.0.2</version>
</dependency>[ Configuration rocketMQ ] : Configure the address of the rocketMQ nameServer and the production group in application-local.propertis.
rocketmq.producer.group = producer_bank2
rocketmq.name-server = 127.0.0.1:9876[ Zhang San service layer code ] :
import com.alibaba.fastjson.JSONObject;
import lombok.extern.slf4j.Slf4j;
import org.apache.rocketmq.spring.core.RocketMQTemplate;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.messaging.Message;
import org.springframework.messaging.support.MessageBuilder;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
/**
* @author Administrator
* @version 1.0
**/
@Service
@Slf4j
public class AccountInfoServiceImpl implements AccountInfoService {
@Autowired
AccountInfoDao accountInfoDao;
@Autowired
RocketMQTemplate rocketMQTemplate;
//向mq发送转账消息
@Override
public void sendUpdateAccountBalance(AccountChangeEvent accountChangeEvent) {
//将accountChangeEvent转成json
JSONObject jsonObject = new JSONObject();
jsonObject.put("accountChange", accountChangeEvent);
String jsonString = jsonObject.toJSONString();
//生成message类型
Message<String> message = MessageBuilder.withPayload(jsonString).build();
//发送一条事务消息
/**
* String txProducerGroup 生产组
* String destination topic，
* Message<?> message, 消息内容
* Object arg 参数
*/
rocketMQTemplate.sendMessageInTransaction("producer_group_txmsg_bank1", "topic_txmsg", message, null);
}
//更新账户，扣减金额
@Override
@Transactional
public void doUpdateAccountBalance(AccountChangeEvent accountChangeEvent) {
//幂等判断,txNo是在Ctroller中生成的 UUID，全局唯一
if (accountInfoDao.isExistTx(accountChangeEvent.getTxNo()) > 0) {
return;
}
//扣减金额
accountInfoDao.updateAccountBalance(accountChangeEvent.getAccountNo(), accountChangeEvent.getAmount() * -1);
//添加事务日志
accountInfoDao.addTx(accountChangeEvent.getTxNo());
if (accountChangeEvent.getAmount() == 3) {
throw new RuntimeException("人为制造异常");
}
}
}[ Zhang San RocketMQLocalTransactionListener ] : Write the RocketMQLocalTransactionListener interface implementation class to implement the two methods of executing local transactions and transaction review.
import com.alibaba.fastjson.JSONObject;
import lombok.extern.slf4j.Slf4j;
import org.apache.rocketmq.spring.annotation.RocketMQTransactionListener;
import org.apache.rocketmq.spring.core.RocketMQLocalTransactionListener;
import org.apache.rocketmq.spring.core.RocketMQLocalTransactionState;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.messaging.Message;
import org.springframework.stereotype.Component;
import org.springframework.transaction.annotation.Transactional;
/**
* @author Administrator
* @version 1.0
**/
@Component
@Slf4j
//生产者组与发送消息时定义组相同
@RocketMQTransactionListener(txProducerGroup = "producer_group_txmsg_bank1")
public class ProducerTxmsgListener implements RocketMQLocalTransactionListener {
@Autowired
AccountInfoService accountInfoService;
@Autowired
AccountInfoDao accountInfoDao;
//事务消息发送后的回调方法，当消息发送给mq成功，此方法被回调
@Override
@Transactional
public RocketMQLocalTransactionState executeLocalTransaction(Message message, Object o) {
try {
//解析message，转成AccountChangeEvent
String messageString = new String((byte[]) message.getPayload());
JSONObject jsonObject = JSONObject.parseObject(messageString);
String accountChangeString = jsonObject.getString("accountChange");
//将accountChange（json）转成AccountChangeEvent
AccountChangeEvent accountChangeEvent = JSONObject.parseObject(accountChangeString, AccountChangeEvent.class);
//执行本地事务，扣减金额
accountInfoService.doUpdateAccountBalance(accountChangeEvent);
//当返回RocketMQLocalTransactionState.COMMIT，自动向mq发送commit消息，mq将消息的状态改为可消费
return RocketMQLocalTransactionState.COMMIT;
} catch (Exception e) {
e.printStackTrace();
return RocketMQLocalTransactionState.ROLLBACK;
}
}
//事务状态回查，查询是否扣减金额
@Override
public RocketMQLocalTransactionState checkLocalTransaction(Message message) {
//解析message，转成AccountChangeEvent
String messageString = new String((byte[]) message.getPayload());
JSONObject jsonObject = JSONObject.parseObject(messageString);
String accountChangeString = jsonObject.getString("accountChange");
//将accountChange（json）转成AccountChangeEvent
AccountChangeEvent accountChangeEvent = JSONObject.parseObject(accountChangeString, AccountChangeEvent.class);
//事务id
String txNo = accountChangeEvent.getTxNo();
int existTx = accountInfoDao.isExistTx(txNo);
if (existTx > 0) {
return RocketMQLocalTransactionState.COMMIT;
} else {
return RocketMQLocalTransactionState.UNKNOWN;
}
}
}[ Li Si service layer code ] :
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
/**
* @author Administrator
* @version 1.0
**/
@Service
@Slf4j
public class AccountInfoServiceImpl implements AccountInfoService {
@Autowired
AccountInfoDao accountInfoDao;
//更新账户，增加金额
@Override
@Transactional
public void addAccountInfoBalance(AccountChangeEvent accountChangeEvent) {
log.info("bank2更新本地账号，账号：{},金额：{}", accountChangeEvent.getAccountNo(), accountChangeEvent.getAmount());
if (accountInfoDao.isExistTx(accountChangeEvent.getTxNo()) > 0) {
return;
}
//增加金额
accountInfoDao.updateAccountBalance(accountChangeEvent.getAccountNo(), accountChangeEvent.getAmount());
//添加事务记录，用于幂等
accountInfoDao.addTx(accountChangeEvent.getTxNo());
if (accountChangeEvent.getAmount() == 4) {
throw new RuntimeException("人为制造异常");
}
}
}[MQ monitor class]: monitor the target topic by implementing the RocketMQListener interface
import com.alibaba.fastjson.JSONObject;
import lombok.extern.slf4j.Slf4j;
import org.apache.rocketmq.spring.annotation.RocketMQMessageListener;
import org.apache.rocketmq.spring.core.RocketMQListener;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
/**
* @author Administrator
* @version 1.0
**/
@Component
@Slf4j
@RocketMQMessageListener(consumerGroup = "consumer_group_txmsg_bank2", topic = "topic_txmsg")
public class TxmsgConsumer implements RocketMQListener<String> {
@Autowired
AccountInfoService accountInfoService;
//接收消息
@Override
public void onMessage(String message) {
log.info("开始消费消息:{}", message);
//解析消息
JSONObject jsonObject = JSONObject.parseObject(message);
String accountChangeString = jsonObject.getString("accountChange");
//转成AccountChangeEvent
AccountChangeEvent accountChangeEvent = JSONObject.parseObject(accountChangeString, AccountChangeEvent.class);
//设置账号为李四的
accountChangeEvent.setAccountNo("2");
//更新本地账户，增加金额
accountInfoService.addAccountInfoBalance(accountChangeEvent);
}
}Five, summary
The ultimate consistency of reliable messages is to ensure the consistency of messages from the producer to the consumer through the message middleware. In this case, RocketMQ is used as the message middleware. RocketMQ mainly solves two functions:
[1] Local transactions and message sending Atomic issues ;
[2] The reliability of the message received by the transaction participants ; the
eventual consistency of reliable messages is suitable for scenarios with long execution cycles and low real-time requirements . After the introduction of the message mechanism, the synchronous transaction operation becomes the asynchronous operation based on the message execution, which avoids the influence of the synchronous blocking operation in the distributed transaction, and realizes the decoupling of the two services.
●The strongest Tomcat8 performance optimization in history
● Why can Alibaba resist 10 billion in 90 seconds? --The evolution of server-side high-concurrency distributed architecture
● B2B e-commerce platform--ChinaPay UnionPay electronic payment function
● Learn Zookeeper distributed lock, let interviewers look at you with admiration
● SpringCloud e-commerce spike microservice-Redisson distributed lock solution
Check out more good articles, enter the official account--please me--excellent in the past
A deep and soulful public account 0.0