One, background
MQ components in the system architecture is essential for a weapon, you can reduce system design level coupling, high concurrency scenarios and can act as load shifting, from single application to the cluster deployment, to the current micro service architecture, MQ, with its excellent performance and high reliability, has been widely recognized.
With the increasing amount of data, the system pressure increases, this phenomenon began to appear: The database has been updated, but the message is not sent to or starting a message, but then failed to update the database, the results of research and development of children's shoes of various data recovery, which probability seed production problems of small, but very depressed people. In fact, this is the consistency of database transactions and MQ messages, simply, with the average transaction database to send the message MQ database transaction can not be directly tied to bundled, such as the above-mentioned scenarios both of these problems:
- Send MQ message database transaction commits;
- MQ messaging starting before committing database transactions.
Problem Scenario 1 is the database transaction might just submitted, the server is down, MQ message did not go through problems Scene 2 MQ message is sent out, but the database transaction commits failed, not any way an additional MQ message has been sent out , resulting in data is not updated, the downstream message has been received, the final transaction appear inconsistent.
Second, the transaction message is drawn
Our micro-service architecture shopping scenario as an example, referring to what the official RocketMQ example, user A initiates an order, pay 100 dollars after the operation is completed, can get 100 points, account services and member services are two separate micro-service module, have their own database, according to the likelihood of the problem mentioned above, there will be these situations:
- If the first charge, then a message, money may just buckle end, is down, the message did not go through the results did not increase integration.
- If the first message, and then charge, integration may increase, but the money did not deducted, give away 100 points to the people.
- Normal money deducted, the message is sent successfully, but the membership service instance consumer information problems, the result of integration did not increase.
This raises the consistency of the database transaction MQ message transaction, the transaction message rocketmq problem solved: to solve the problem of a local transaction atomicity performs message transmission. Here boundaries must understand, it is to ensure the production side MQ correctly send messages out, no more hair, not omission. But as the sending end has no normal consumption consumed (e.g., the above-mentioned third case, the normal money deducted, the message also made, but the integration does not cause problems downstream consumption), this anomaly scenario MQ message by consumer failure retry mechanism to ensure, not included in this discussion within range.
MQ components commonly used for this scene has its own implementation, such as ActiveMQ using the AMQP protocol (second-order submission) to ensure that the right message is sent, here we RocketMQ focus on learning.
Three, RocketMQ transaction message design ideas
According to CAP theory, RocketMQ way through asynchronous transactional messaging to ensure that, to ensure the consistency of the final transaction. Designed to draw on two-phase commit process theory, a flow chart is as follows:
- When the application module encounters a scene you want to send a message of affairs, prepare to send message to MQ.
- After the message is sent successfully prepare, the application module performs database transactions (local transactions).
- According to the results of a database transaction execution, and then return to Commit or Rollback MQ.
- If the terminal is sent to the Consumer Commit, MQ the message, if the Rollback, deleted directly prepare message.
- Results of the third step if no response, or a timeout, the timing task starts Recall transaction state (retries up to 15 times, more than the default discard the message), step 4 with the processing result.
- MQ mechanism to ensure the success of the MQ own consumption.
Four, RocketMQ transaction message implementation process
To RocketMQ 4.5.2 version, for example, have a special transaction messages a queue RMQ_SYS_TRANS_HALF_TOPIC, all prepare messages are first put into here, when the message is received Commit request, and then put the message stuffed into real Topic queue for Consumer consumption, while the RMQ_SYS_TRANS_OP_HALF_TOPIC plug a message. Simplified flowchart follows:
The above process, please allow me to divide responsibilities module:
- RocketMQ Client project in which we rely on imported jar package, RocketMQ Broker end server that is deployed, NameServer yet reflected.
- Application module pairs, the upstream end of the production for the transaction message, the downstream end of the consumer for the transaction message (transaction message is transparent to the consumer side, consistent with the normal message).
Because the interrupt transaction application modules, network or other reasons, can not result in an immediate response, RocketMQ as UNKNOW process, RocketMQ transaction message also provides a remedy: polling the message database transaction state of transaction
simple flowchart is as follows:
Fifth, source code analysis
According to the following flow chart, according to the analysis module by one explain the functions and processes substantially the idea.
Preparing the Environment
Before reading the source code and need to obtain the source code debugging RocketMQ on IDE, this part of your own inspection methods.- Application module (transaction message production side) core source code
to create a listener class that implements an interface TransactionListener, submission method implemented in a database transaction and the transaction back to check the status method simulation results.
/**
* @program: rocket
* @description: 调试事务消息示例代码
* @author: Huang
* @create: 2019-10-16
**/
public class SelfTransactionListener implements TransactionListener {
private AtomicInteger transactionIndex = new AtomicInteger(0);
private AtomicInteger checkTimes = new AtomicInteger(0);
private ConcurrentHashMap<String, Integer> localTrans = new ConcurrentHashMap<>();
/**
* 执行本地事务
*
* @param message
* @param o
* @return
*/
@Override
public LocalTransactionState executeLocalTransaction(Message message, Object o) {
String msgKey = message.getKeys();
System.out.println("start execute local transaction " + msgKey);
LocalTransactionState state;
if (msgKey.contains("1")) {
// 第一条消息让他通过
state = LocalTransactionState.COMMIT_MESSAGE;
} else if (msgKey.contains("2")) {
// 第二条消息模拟异常，明确回复回滚操作
state = LocalTransactionState.ROLLBACK_MESSAGE;
} else {
// 第三条消息无响应，让它调用回查事务方法
state = LocalTransactionState.UNKNOW;
// 给剩下3条消息，放1，2，3三种状态
localTrans.put(msgKey, transactionIndex.incrementAndGet());
}
System.out.println("executeLocalTransaction:" + message.getKeys() + ",execute state:" + state + ",current time：" + System.currentTimeMillis());
return state;
}
/**
* 回查本地事务结果
*
* @param messageExt
* @return
*/
@Override
public LocalTransactionState checkLocalTransaction(MessageExt messageExt) {
String msgKey = messageExt.getKeys();
System.out.println("start check local transaction " + msgKey);
Integer state = localTrans.get(msgKey);
switch (state) {
case 1:
System.out.println("check result unknown 回查次数" + checkTimes.incrementAndGet());
return LocalTransactionState.UNKNOW;
case 2:
System.out.println("check result commit message, 回查次数" + checkTimes.incrementAndGet());
return LocalTransactionState.COMMIT_MESSAGE;
case 3:
System.out.println("check result rollback message, 回查次数" + checkTimes.incrementAndGet());
return LocalTransactionState.ROLLBACK_MESSAGE;
default:
return LocalTransactionState.COMMIT_MESSAGE;
}
}
}Transaction message producer code sample, a total of five messages transmitted, comprising substantially all of the scenes, the sleep time is set for a time sufficient to ensure that when the back check transaction instance is still running, as follows:
/**
* @program: rocket
* @description: Rocketmq事务消息
* @author: Huang
* @create: 2019-10-16
**/
public class TransactionProducer {
public static void main(String[] args) {
try {
TransactionMQProducer producer = new TransactionMQProducer("transactionMQProducer");
producer.setNamesrvAddr("10.0.133.29:9876");
producer.setTransactionListener(new SelfTransactionListener());
producer.start();
for (int i = 1; i < 6; i++) {
Message message = new Message("TransactionTopic", "transactionTest","msg-" + i, ("Hello" + ":" + i).getBytes());
try {
SendResult result = producer.sendMessageInTransaction(message, "Hello" + ":" + i);
System.out.printf("Topic:%s send success, misId is:%s%n", message.getTopic(), result.getMsgId());
} catch (Exception e) {
e.printStackTrace();
}
}
Thread.sleep(Integer.MAX_VALUE);
producer.shutdown();
} catch (MQClientException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}- RocketMQ Client-side code, the code can be mainly divided into three logical sections: the first message is provided to prepare for the message, the server sends RocketMQ
SendResult sendResult = null;
MessageAccessor.putProperty(msg, MessageConst.PROPERTY_TRANSACTION_PREPARED, "true");
MessageAccessor.putProperty(msg, MessageConst.PROPERTY_PRODUCER_GROUP, this.defaultMQProducer.getProducerGroup());
try {
sendResult = this.send(msg);
} catch (Exception e) {
throw new MQClientException("send message Exception", e);
}The second paragraph: After the message is sent successfully, the calling application module database transaction method to get the transaction result (To save space, the code abridged)
switch (sendResult.getSendStatus()) {
case SEND_OK: {
try {
if (null != localTransactionExecuter) {
localTransactionState = localTransactionExecuter.executeLocalTransactionBranch(msg, arg);
} else if (transactionListener != null) {
log.debug("Used new transaction API");
localTransactionState = transactionListener.executeLocalTransaction(msg, arg);
}
if (null == localTransactionState) {
localTransactionState = LocalTransactionState.UNKNOW;
}
} catch (Throwable e) {
log.info("executeLocalTransactionBranch exception", e);
log.info(msg.toString());
localException = e;
}
}
break;
case FLUSH_DISK_TIMEOUT:
case FLUSH_SLAVE_TIMEOUT:
case SLAVE_NOT_AVAILABLE:
localTransactionState = LocalTransactionState.ROLLBACK_MESSAGE;
break;
default:
break;
}Third stage: the results sent to the transaction RocketMQ terminal end of the transaction, and the response result to the application module
try {
this.endTransaction(sendResult, localTransactionState, localException);
} catch (Exception e) {
log.warn("local transaction execute " + localTransactionState + ", but end broker transaction failed", e);
}- RocketMQ Broker end transaction commit / rollback operations (herein take part endTransaction)
code entry: org.apache.rocketmq.broker.processor.EndTransactionProcessor
OperationResult result = new OperationResult();
if (MessageSysFlag.TRANSACTION_COMMIT_TYPE == requestHeader.getCommitOrRollback()) {
result = this.brokerController.getTransactionalMessageService().commitMessage(requestHeader);
if (result.getResponseCode() == ResponseCode.SUCCESS) {
RemotingCommand res = checkPrepareMessage(result.getPrepareMessage(), requestHeader);
if (res.getCode() == ResponseCode.SUCCESS) {
// 修改消息的Topic为由RMQ_SYS_TRANS_HALF_TOPIC改为真实Topic
MessageExtBrokerInner msgInner = endMessageTransaction(result.getPrepareMessage());
msgInner.setSysFlag(MessageSysFlag.resetTransactionValue(msgInner.getSysFlag(), requestHeader.getCommitOrRollback()));
msgInner.setQueueOffset(requestHeader.getTranStateTableOffset());
msgInner.setPreparedTransactionOffset(requestHeader.getCommitLogOffset());
msgInner.setStoreTimestamp(result.getPrepareMessage().getStoreTimestamp());
// 将消息存储到真实Topic中，供Consumer消费
RemotingCommand sendResult = sendFinalMessage(msgInner);
if (sendResult.getCode() == ResponseCode.SUCCESS) {
// 将消息存储到RMQ_SYS_TRANS_OP_HALF_TOPIC，标记为删除状态，事务消息回查的定时任务中会做处理
this.brokerController.getTransactionalMessageService().deletePrepareMessage(result.getPrepareMessage());
}
return sendResult;
}
return res;
}
} else if (MessageSysFlag.TRANSACTION_ROLLBACK_TYPE == requestHeader.getCommitOrRollback()) {
result = this.brokerController.getTransactionalMessageService().rollbackMessage(requestHeader);
if (result.getResponseCode() == ResponseCode.SUCCESS) {
RemotingCommand res = checkPrepareMessage(result.getPrepareMessage(), requestHeader);
if (res.getCode() == ResponseCode.SUCCESS) {
this.brokerController.getTransactionalMessageService().deletePrepareMessage(result.getPrepareMessage());
}
return res;
}
}- RocketMQ Broker task of the timing section Recall database transaction
method entry: org.apache.rocketmq.broker.transaction.TransactionalMessageCheckService
@Override
protected void onWaitEnd() {
long timeout = brokerController.getBrokerConfig().getTransactionTimeOut();
// 超过15次的回查事务状态失败后，默认是丢弃此消息
int checkMax = brokerController.getBrokerConfig().getTransactionCheckMax();
long begin = System.currentTimeMillis();
log.info("Begin to check prepare message, begin time:{}", begin);
this.brokerController.getTransactionalMessageService().check(timeout, checkMax, this.brokerController.getTransactionalMessageCheckListener());
log.info("End to check prepare message, consumed time:{}", System.currentTimeMillis() - begin);
}Check back entrance transaction calls:
// 此段代码为TransactionalMessageServiceImpl类中的check方法
List<MessageExt> opMsg = pullResult.getMsgFoundList();
boolean isNeedCheck = (opMsg == null && valueOfCurrentMinusBorn > checkImmunityTime)
|| (opMsg != null && (opMsg.get(opMsg.size() - 1).getBornTimestamp() - startTime > transactionTimeout))
|| (valueOfCurrentMinusBorn <= -1
);
if (isNeedCheck) {
if (!putBackHalfMsgQueue(msgExt, i)) {
continue;
}
// 调用AbstractTransactionalMessageCheckListener的
listener.resolveHalfMsg(msgExt);
} else {
pullResult = fillOpRemoveMap(removeMap, opQueue, pullResult.getNextBeginOffset(), halfOffset, doneOpOffset);
log.info("The miss offset:{} in messageQueue:{} need to get more opMsg, result is:{}", i,
messageQueue, pullResult);
continue;
}
// 此方法在AbstractTransactionalMessageCheckListener类中
public void resolveHalfMsg(final MessageExt msgExt) {
executorService.execute(new Runnable() {
@Override
public void run() {
try {
sendCheckMessage(msgExt);
} catch (Exception e) {
LOGGER.error("Send check message error!", e);
}
}
});
}
// 此方法在AbstractTransactionalMessageCheckListener类中
public void sendCheckMessage(MessageExt msgExt) throws Exception {
CheckTransactionStateRequestHeader checkTransactionStateRequestHeader = new CheckTransactionStateRequestHeader();
checkTransactionStateRequestHeader.setCommitLogOffset(msgExt.getCommitLogOffset());
checkTransactionStateRequestHeader.setOffsetMsgId(msgExt.getMsgId());
checkTransactionStateRequestHeader.setMsgId(msgExt.getUserProperty(MessageConst.PROPERTY_UNIQ_CLIENT_MESSAGE_ID_KEYIDX));
checkTransactionStateRequestHeader.setTransactionId(checkTransactionStateRequestHeader.getMsgId());
checkTransactionStateRequestHeader.setTranStateTableOffset(msgExt.getQueueOffset());
msgExt.setTopic(msgExt.getUserProperty(MessageConst.PROPERTY_REAL_TOPIC));
msgExt.setQueueId(Integer.parseInt(msgExt.getUserProperty(MessageConst.PROPERTY_REAL_QUEUE_ID)));
msgExt.setStoreSize(0);
String groupId = msgExt.getProperty(MessageConst.PROPERTY_PRODUCER_GROUP);
Channel channel = brokerController.getProducerManager().getAvaliableChannel(groupId);
if (channel != null) {
// 通过Netty发送请求到RocketMQ Client端，执行checkTransactionState方法
brokerController.getBroker2Client().checkProducerTransactionState(groupId, channel, checkTransactionStateRequestHeader, msgExt);
} else {
LOGGER.warn("Check transaction failed, channel is null. groupId={}", groupId);
}
}RocketMQ Client server after receiving a request, the recall Recall database transaction method and the results of the transaction once again submit to RocketMQ Broker end
method entry: class method org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl
try {
if (transactionCheckListener != null) {
localTransactionState = transactionCheckListener.checkLocalTransactionState(message);
} else if (transactionListener != null) {
log.debug("Used new check API in transaction message");
localTransactionState = transactionListener.checkLocalTransaction(message);
} else {
log.warn("CheckTransactionState, pick transactionListener by group[{}] failed", group);
}
} catch (Throwable e) {
log.error("Broker call checkTransactionState, but checkLocalTransactionState exception", e);
exception = e;
}
this.processTransactionState(
localTransactionState,
group,
exception);Sixth, a supplementary question
Official website mentioned transaction message is not supported latency messaging and bulk message, I hand cheap tried it delayed messages, set up a transaction message DelayTimeLevel, the results of this message has been lost can not be removed from RMQ_SYS_TRANS_HALF_TOPIC, log application module discovery in repeated attempts to check back affairs, RMQ_SYS_TRANS_HALF_TOPIC Console interface on the list of queries news soon more than 2000 records, and why?
We return to the code level are analyzed as follows:
1. After setting up DelayTimeLevel, the data after the transaction commits (or check back after the database transaction is completed), the message is written to the target Topic, due to interference DelayTimeLevel, the target Topic will become SCHEDULE_TOPIC_XXXX, while REAL_TOPIC become RMQ_SYS_TRANS_HALF_TOPIC, real Topic has been lost in this session.
// RocketMQ Broker端接受事务提交后的处理
org.apache.rocketmq.broker.processor.EndTransactionProcessor类
OperationResult result = new OperationResult();
if (MessageSysFlag.TRANSACTION_COMMIT_TYPE == requestHeader.getCommitOrRollback()) {
// 这里调用CommitLog的putMessage方法
result = this.brokerController.getTransactionalMessageService().commitMessage(requestHeader);
if (result.getResponseCode() == ResponseCode.SUCCESS) {
RemotingCommand res = checkPrepareMessage(result.getPrepareMessage(), requestHeader);
if (res.getCode() == ResponseCode.SUCCESS) {
// 修改消息的Topic为由RMQ_SYS_TRANS_HALF_TOPIC改为真实Topic
MessageExtBrokerInner msgInner = endMessageTransaction(result.getPrepareMessage());
msgInner.setSysFlag(MessageSysFlag.resetTransactionValue(msgInner.getSysFlag(), requestHeader.getCommitOrRollback()));
msgInner.setQueueOffset(requestHeader.getTranStateTableOffset());
msgInner.setPreparedTransactionOffset(requestHeader.getCommitLogOffset());
msgInner.setStoreTimestamp(result.getPrepareMessage().getStoreTimestamp());
// 将消息存储到真实Topic中，此时Topic已经变成SCHEDULE_TOPIC_XXXX
RemotingCommand sendResult = sendFinalMessage(msgInner);
if (sendResult.getCode() == ResponseCode.SUCCESS) {
// 将消息存储到RMQ_SYS_TRANS_OP_HALF_TOPIC，标记为删除状态，事务消息回查的定时任务中会做处理
this.brokerController.getTransactionalMessageService().deletePrepareMessage(result.getPrepareMessage());
}
return sendResult;
}
return res;
}
}
// 此段代码在org.apache.rocketmq.store.CommitLog类的putMessage方法中
// 由于DelayTimeLevel的干扰，目标Topic将变成SCHEDULE_TOPIC_XXXX
final int tranType = MessageSysFlag.getTransactionValue(msg.getSysFlag());
if (tranType == MessageSysFlag.TRANSACTION_NOT_TYPE
|| tranType == MessageSysFlag.TRANSACTION_COMMIT_TYPE) {
// Delay Delivery
if (msg.getDelayTimeLevel() > 0) {
if (msg.getDelayTimeLevel() > this.defaultMessageStore.getScheduleMessageService().getMaxDelayLevel()) {
msg.setDelayTimeLevel(this.defaultMessageStore.getScheduleMessageService().getMaxDelayLevel());
}
topic = ScheduleMessageService.SCHEDULE_TOPIC;
queueId = ScheduleMessageService.delayLevel2QueueId(msg.getDelayTimeLevel());
// Backup real topic, queueId
MessageAccessor.putProperty(msg, MessageConst.PROPERTY_REAL_TOPIC, msg.getTopic());
MessageAccessor.putProperty(msg, MessageConst.PROPERTY_REAL_QUEUE_ID, String.valueOf(msg.getQueueId()));
msg.setPropertiesString(MessageDecoder.messageProperties2String(msg.getProperties()));
msg.setTopic(topic);
msg.setQueueId(queueId);
}
}Example, the print log as follows:
2019-10-17 14\:41\:05 INFO EndTransactionThread_4 - Transaction op message write successfully. messageId=0A00851D00002A9F0000000000000E09, queueId=0
msgExt:MessageExt [queueId=0, storeSize=335, queueOffset=5, sysFlag=8, bornTimestamp=1571293959305, bornHost=/10.0.133.29:54634, storeTimestamp=1571294460555,
storeHost=/10.0.133.29:10911, msgId=0A00851D00002A9F0000000000000E09, commitLogOffset=3593, bodyCRC=1849408413, reconsumeTimes=0, preparedTransactionOffset=0,
toString()=Message{topic='SCHEDULE_TOPIC_XXXX', flag=0, properties={REAL_TOPIC=RMQ_SYS_TRANS_HALF_TOPIC, TRANSACTION_CHECK_TIMES=3, KEYS=msg-test-3,
TRAN_MSG=true, UNIQ_KEY=0A00851D422C18B4AAC25584B0880000, WAIT=false, DELAY=1, PGROUP=transactionMQProducer, TAGS=transactionTest, REAL_QID=0},
body=[72, 101, 108, 108, 111, 84, 105, 109, 101, 58, 51], transactionId='null'}]2. delay the message is timed tasks triggered, I just set a delay of 1 second, scheduled tasks back into the news again RMQ_SYS_TRANS_HALF_TOPIC, notice that this time only RMQ_SYS_TRANS_HALF_TOPIC news, RMQ_SYS_TRANS_OP_HALF_TOPIC queue is not this message, the following code:
// 此段代码在org.apache.rocketmq.store.schedule.ScheduleMessageService类executeOnTimeup方法内
try {
// 消息重新回到RMQ_SYS_TRANS_HALF_TOPIC队列中
MessageExtBrokerInner msgInner = this.messageTimeup(msgExt);
PutMessageResult putMessageResult =
ScheduleMessageService.this.writeMessageStore
.putMessage(msgInner);
if (putMessageResult != null
&& putMessageResult.getPutMessageStatus() == PutMessageStatus.PUT_OK) {
continue;
} else {
log.error(
"ScheduleMessageService, a message time up, but reput it failed, topic: {} msgId {}",
msgExt.getTopic(), msgExt.getMsgId());
ScheduleMessageService.this.timer.schedule(
new DeliverDelayedMessageTimerTask(this.delayLevel,
nextOffset), DELAY_FOR_A_PERIOD);
ScheduleMessageService.this.updateOffset(this.delayLevel,
nextOffset);
return;
}
} catch (Exception e) {
log.error(
"ScheduleMessageService, messageTimeup execute error, drop it. msgExt="
+ msgExt + ", nextOffset=" + nextOffset + ",offsetPy="
+ offsetPy + ",sizePy=" + sizePy, e);
}3. The transaction message timed task starts, check RMQ_SYS_TRANS_HALF_TOPIC news, but RMQ_SYS_TRANS_OP_HALF_TOPIC no message, write the message in order to ensure order, and repopulated RMQ_SYS_TRANS_OP_HALF_TOPIC this message, and check back trigger a transaction operations. The sample code above Recall same call transaction entry:
// 此段代码为TransactionalMessageServiceImpl类中的check方法
List<MessageExt> opMsg = pullResult.getMsgFoundList();
boolean isNeedCheck = (opMsg == null && valueOfCurrentMinusBorn > checkImmunityTime)
|| (opMsg != null && (opMsg.get(opMsg.size() - 1).getBornTimestamp() - startTime > transactionTimeout))
|| (valueOfCurrentMinusBorn <= -1
);
if (isNeedCheck) {
if (!putBackHalfMsgQueue(msgExt, i)) {
continue;
}
listener.resolveHalfMsg(msgExt);
} else {
pullResult = fillOpRemoveMap(removeMap, opQueue, pullResult.getNextBeginOffset(), halfOffset, doneOpOffset);
log.info("The miss offset:{} in messageQueue:{} need to get more opMsg, result is:{}", i,
messageQueue, pullResult);
continue;
}This constitutes an infinite loop, until the attempt to discard this message 15 times before (the default maximum number of attempts is 15), that price is a little big. For optimization of this problem, has been submitted to RocketMQ PR community, the new version is released, the transaction message will shield DelayTimeLevel, this problem will not happen again.
Before the new version is released, our solution:
- R & D process clearly prohibited transaction message settings DelayTimeLevel.
Feeling at risk, after all, the new children's shoes, is not particularly aware of the tremor might add some of the features of this (as I did the first). - RocketMQ Client to make a simple package, such as a rocketmq-spring-boot-starter, not available to provide the transmission method provided in an inlet of the transaction message, the following example:
/**
* 事务消息发送
* 不支持延迟发送和批量发送
*/
public void sendMessageInTransaction(String topic, String tag, Object message, String requestId) throws Exception {
TransactionMQProducer producer = annotationScan.getProducer(topic + "_" + tag);
producer.sendMessageInTransaction(MessageBuilder.of(topic, tag, message, requestId).build(), message);
}It should be ideal, after all, put an end to the parameters set DelayTimeLevel from the source.
VII Conclusion
Benpian briefly addressed the limits of transaction messages scene and responsibilities, basic design ideas and processes in this RocketMQ learn from the author's artwork, then pick a brief explanation of the code, or your own dig pits process , in the article there are any incorrect or exhaustive guide place, please leave a message, thank you.
High focus on Java concurrency, distributed architecture, more dry goods share technology and experience, please pay attention to the public number: Java Architecture Community
