1. The origin of the transaction message
1. Case
Quoting the official shopping case:
When Xiao Ming purchases a 100 yuan item, the account deducts 100 yuan and he needs to ensure that the downstream points system adds 100 points to Xiao Ming's account. The account system and the points system are two independent systems, one should be reduced by 100 yuan, and the other should be increased by 100 points. As shown below:
2. The problem
-
The account service deduction was successful, and the notification point system was also successful, but it failed when the points were increased, and the data was inconsistent.
-
The account service deduction was successful, but the notification point system failed, so the points will not increase, and the data is inconsistent.
3. Scheme
RocketMQ's solution to the first problem is: if the consumption fails, it will automatically retry. If the consumption fails after a few retries, then this situation needs to be solved manually, such as putting it in the dead letter queue and then Manually check the cause and deal with it.
RocketMQ's solution to the second problem is: if your deduction succeeds, but fails when writing a message to mq, then RocketMQ will roll back the message, and at this time we can also roll back our deduction operation.
Second, the principle of transaction messages
1. Principle diagram
2. Detailed process
1. The Producer sends a half message (Half Message) to the broker.
I really want to complain about why it is called a half-message, which is difficult to understand. In fact, this is a prepare message, a pre-sent message.
-
After the Half Message is sent successfully, the local transaction will be executed.
-
If the local transaction is executed successfully, it returns commit, if the execution fails, it returns rollback. (This is determined by the developer himself in the callback method of the transaction message commit or rollback)
Producer sends the commit of the previous step or rollback to the broker. There are two situations here:
1. If the broker receives a commit/rollback message:
-
If a commit is received, the broker believes that the entire transaction is okay and the execution is successful. Then it will send a message to the Consumer end for consumption.
-
If a rollback is received, the broker thinks that the local transaction execution has failed, and the broker will delete the Half Message and not send it to the Consumer.
2. If the broker does not receive the message (if the execution of the local transaction suddenly goes down, the execution result of the equivalent local transaction returns unknow, it will be treated as if the broker did not receive the confirmation message.):
-
The broker will periodically review the execution results of the local transaction: if the review result is that the local transaction has been executed, it returns commit, if not, it returns rollback.
-
The results of the Producer's backcheck are sent to the Broker. If the broker receives a commit, the broker considers the entire transaction to be executed successfully. If it is a rollback, the broker considers the local transaction to fail, and the broker deletes the Half Message and does not send it to the consumer. If the broker does not receive the result of the back-check (or if it finds unknow), the broker will repeat the back-check regularly to ensure that the final transaction result is found. Both the time interval and the number of repeated back-checks can be configured.
Three, transaction message realization process
1. Implementation process
To put it simply, the transaction message is a listener with a callback function. In the callback function, we perform business logic operations, such as giving the account -100 yuan, and then sending a message to the mq of the points. At this time, if the account -100 succeeds, And if the message is successfully sent to mq, the message status is set to commit. At this time, the broker will send the half message to the real topic. At the beginning, it was sent to the semi-message queue, and it did not exist in the real topic queue. Only after confirming the commit will it be transferred.
2. Remedial plan
If the transaction fails to respond immediately due to interruption or other network reasons, RocketMQ treats it as UNKNOW. RocketMQ transaction messages also provide a remedy: query the transaction status of the transaction messages regularly. This is also a callback function. Compensation can be done in it. The compensation logic developer writes it himself. If it succeeds, it returns to commit and it's done.
Four, transaction message code example
1. Code
package com.chentongwei.mq.rocketmq;
import org.apache.rocketmq.client.producer.LocalTransactionState;
import org.apache.rocketmq.client.producer.TransactionListener;
import org.apache.rocketmq.client.producer.TransactionMQProducer;
import org.apache.rocketmq.client.producer.TransactionSendResult;
import org.apache.rocketmq.common.message.Message;
import org.apache.rocketmq.common.message.MessageExt;
import java.util.Date;
/**
* Description:
*
* @author TongWei.Chen 2020-06-21 11:32:58
*/
public class ProducerTransaction2 {
public static void main(String[] args) throws Exception {
TransactionMQProducer producer = new TransactionMQProducer("my-transaction-producer");
producer.setNamesrvAddr("124.57.180.156:9876");
// 回调
producer.setTransactionListener(new TransactionListener() {
@Override
public LocalTransactionState executeLocalTransaction(Message message, Object arg) {
LocalTransactionState state = null;
//msg-4返回COMMIT_MESSAGE
if(message.getKeys().equals("msg-1")){
state = LocalTransactionState.COMMIT_MESSAGE;
}
//msg-5返回ROLLBACK_MESSAGE
else if(message.getKeys().equals("msg-2")){
state = LocalTransactionState.ROLLBACK_MESSAGE;
}else{
//这里返回unknown的目的是模拟执行本地事务突然宕机的情况（或者本地执行成功发送确认消息失败的场景）
state = LocalTransactionState.UNKNOW;
}
System.out.println(message.getKeys() + ",state:" + state);
return state;
}
/**
* 事务消息的回查方法
*/
@Override
public LocalTransactionState checkLocalTransaction(MessageExt messageExt) {
if (null != messageExt.getKeys()) {
switch (messageExt.getKeys()) {
case "msg-3":
System.out.println("msg-3 unknow");
return LocalTransactionState.UNKNOW;
case "msg-4":
System.out.println("msg-4 COMMIT_MESSAGE");
return LocalTransactionState.COMMIT_MESSAGE;
case "msg-5":
//查询到本地事务执行失败，需要回滚消息。
System.out.println("msg-5 ROLLBACK_MESSAGE");
return LocalTransactionState.ROLLBACK_MESSAGE;
}
}
return LocalTransactionState.COMMIT_MESSAGE;
}
});
producer.start();
//模拟发送5条消息
for (int i = 1; i < 6; i++) {
try {
Message msg = new Message("transactionTopic", null, "msg-" + i, ("测试，这是事务消息！ " + i).getBytes());
producer.sendMessageInTransaction(msg, null);
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
2. Results
msg-1,state:COMMIT_MESSAGE
msg-2,state:ROLLBACK_MESSAGE
msg-3,state:UNKNOW
msg-4,state:UNKNOW
msg-5,state:UNKNOW
msg-3 unknow
msg-3 unknow
msg-5 ROLLBACK_MESSAGE
msg-4 COMMIT_MESSAGE
msg-3 unknow
msg-3 unknow
msg-3 unknow
msg-3 unknow
3. Control console
4. Result analysis
-
Only msg-1 and msg-4 were successfully sent. msg-4 is ahead of msg-1 because msg-1 succeeded first, and msg-4 succeeded only after back-checking. Comes in reverse chronological order.
-
First output five results, corresponding to five messages
msg-1,state:COMMIT_MESSAGE
msg-2,state:ROLLBACK_MESSAGE
msg-3,state:UNKNOW
msg-4,state:UNKNOW
msg-5,state:UNKNOW
-
Then entered the review, msg-3 was still unknow, msg-5 rolled back, and msg-4 submitted the transaction. So at this time msg-4 can be seen in the control console.
-
After a while, I checked msg-3 again and found that it was still unknow, so I kept checking back.
The time interval and the number of back-checks are configurable. The default is that if the check-back fails 15 times, the message will be discarded.
Five, questions
Question: Are Spring transactions and regular distributed transactions not good? Is Rocketmq's affairs superfluous?
MQ is used for decoupling. Previously, distributed transactions directly operated the account system and points system. But the two of them are the existence of strong coupling. If an mq is inserted in the middle, the account system will send a message to the mq after the account system is operated. At this time, as long as the sending is guaranteed to be successful, it will be submitted, and the sending will be rolled back if the sending fails. How to guarantee this step depends on the transaction. And there are quite a lot of people using RocketMQ for distributed transactions.
Six, sequence message explanation
1 Overview
RocketMQ messages are stored in the Topic queue, and the queue itself is a FIFO (First Int First Out) queue. So a single queue can guarantee order.
But the problem is that there are N queues for a topic. The benefits of the author's design in this way are also obvious. It naturally supports the characteristics of clustering and load balancing, and evenly distributes massive amounts of data to each queue. You send 10 messages to the same topic. , These 10 messages will be automatically scattered in all the queues under the topic, so when consuming, it is not necessarily which queue is consumed first and which queue is consumed later, which leads to disordered consumption.
2. Graphic
3. Analyze again
A Producer sends four messages m1, m2, m3, and m4 to the topic. The topic has four queues. Due to its own load balancing strategy, one message is stored on each of the four queues. The m1 stored on queue1, the m2 stored on queue2, the m3 stored on queue3, and the m4 stored on queue4. Consumer consumes multi-threaded consumption, so he cannot guarantee which queue or message is consumed first, such as the order of sending It is m1, m2, m3, m4, but because the consumer is consumed by multiple threads inside the consumer, it may consume m4 on the queue4 queue first, and then m1, which leads to disorder.
Seven, sequential message solution
1. Option One
Quite simply, the key to the problem is that multiple queues have messages, and when I consume, I don't know which queue has the latest message. Then the idea is there. If you want to ensure orderliness when sending messages, send them to a queue for me, and then when consuming, because only that queue has messages and the queue is FIFO, first in, first out, So normal consumption is over.
very perfect. And RocketMQ also provides us with the api (MessageQueueSelector) for selecting queue when sending messages. Go directly to the code.
2. Code One
2.1. Producer
import java.util.List;
import org.apache.rocketmq.client.producer.DefaultMQProducer;
import org.apache.rocketmq.client.producer.MessageQueueSelector;
import org.apache.rocketmq.common.message.Message;
import org.apache.rocketmq.common.message.MessageQueue;
/**
* 消息发送者
*/
public class Producer5 {
public static void main(String[] args)throws Exception {
DefaultMQProducer producer = new DefaultMQProducer("my-order-producer");
producer.setNamesrvAddr("124.57.180.156:9876");
producer.start();
for (int i = 0; i < 5; i++) {
Message message = new Message("orderTopic", ("hello!" + i).getBytes());
producer.send(
// 要发的那条消息
message,
// queue 选择器 ，向 topic中的哪个queue去写消息
new MessageQueueSelector() {
// 手动 选择一个queue
@Override
public MessageQueue select(
// 当前topic 里面包含的所有queue
List<MessageQueue> mqs,
// 具体要发的那条消息
Message msg,
// 对应到 send（） 里的 args，也就是2000前面的那个0
// 实际业务中可以把0换成实际业务系统的主键，比如订单号啥的，然后这里做hash进行选择queue等。能做的事情很多，我这里做演示就用第一个queue，所以不用arg。
Object arg) {
// 向固定的一个queue里写消息，比如这里就是向第一个queue里写消息
MessageQueue queue = mqs.get(0);
// 选好的queue
return queue;
}
},
// 自定义参数：0
// 2000代表2000毫秒超时时间
0, 2000);
}
}
}
2.2. Consumers
import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
import org.apache.rocketmq.client.consumer.listener.*;
import org.apache.rocketmq.common.message.MessageExt;
import java.util.List;
/**
* Description:
*
* @author TongWei.Chen 2020-06-22 11:17:47
*/
public class ConsumerOrder {
public static void main(String[] args) throws Exception {
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("my-consumer");
consumer.setNamesrvAddr("124.57.180.156:9876");
consumer.subscribe("orderTopic", "*");
consumer.registerMessageListener(new MessageListenerOrderly() {
@Override
public ConsumeOrderlyStatus consumeMessage(List<MessageExt> msgs, ConsumeOrderlyContext context) {
for (MessageExt msg : msgs) {
System.out.println(new String(msg.getBody()) + " Thread:" + Thread.currentThread().getName() + " queueid:" + msg.getQueueId());
}
return ConsumeOrderlyStatus.SUCCESS;
}
});
consumer.start();
System.out.println("Consumer start...");
}
}
2.3, output results
Consumer start...
hello!0 Thread:ConsumeMessageThread_1 queueid:0
hello!1 Thread:ConsumeMessageThread_1 queueid:0
hello!2 Thread:ConsumeMessageThread_1 queueid:0
hello!3 Thread:ConsumeMessageThread_1 queueid:0
hello!4 Thread:ConsumeMessageThread_1 queueid:0
Very perfect, orderly output!
3. Situation two
For example, your new requirement: put all unpaid orders in queue1, paid orders in queue2, and abnormal payment orders in queue3. Then you must ensure that each queue is in order when you consume. , Can't consume queue1 and go directly to queue2. You have to consume queue one by one.
At this time, the idea is to use the custom parameter arg when sending a message. The message body must contain the payment status. If it is judged to be unpaid, select queue1, and so on. This ensures that each queue contains only messages of the same state. So consumers are currently consumed by multiple threads, which must be out of order. Three queues are consumed randomly. The solution is simpler, directly change the number of threads on the consumer side to 1, so that the queue is FIFO, and he consumes one by one. RocketMQ also provides us with such an api, the following two sentences:
// 最大线程数1
consumer.setConsumeThreadMax(1);
// 最小线程数
consumer.setConsumeThreadMin(1);