In a distributed system, in order to ensure data consistency must use distributed transactions. Distributed transaction implementation very many, mainly introduce today use RocketMQ transaction messages, realize distributed transaction.
End of the article there are eggs, watching the walk
Why transaction message?
Many students may not know what Affairs news is, it does not matter, give a real business scenario, let's take you look at the problems of ordinary messages.
Above business scenario, when users pay for success, will update the payment order, then send MQ message. The system will be charged by the pull message, save the calculated fees and fees to another database.
Since the calculation of this fee may be calculated off-line step, the process used here MQ decoupled payment and fee calculation.
The main process involves three steps:
- Update order data
- Send a message to MQ
- Pull message handling system
The steps mentioned above, any will fail, if we do not deal with, it will cause data inconsistency on both sides, will result in the following two situations:
- Order data updated, commission data not generated
- Fees generated data, order data has not been updated
This is real money involved, once a small calculation will result in capital loss , really can not afford to lose!
For the final step is concerned, it is relatively simple. If the consumer fails the message, as long as no submitted news confirmed, MQ server will automatically retry.
The biggest problem is that we can not guarantee the update operation and send a message consistency. Whether we use the data to update the order, then send a message, or to send a message, and then update the order data, in the presence of a success, a failure might.
As shown below, using the first message transmitted before updating the database.
After the above process messages sent successfully, then a local transaction commits. This process looks perfect, but imagine if the database fails while the transaction is committed, resulting in a transaction rollback.
But this time the message has been sent, can not be withdrawn. This leads to commission the system will immediately consume messages, calculate fees and update the database. This has resulted in the payment data is not updated, case handling system has generated inconsistent.
If we look at the process that trans is not like it?
We use the following pseudo-code representation:
// 开始事务
try {
// 1.执行数据库操作
// 2.提交事务
}catch (Exception e){
// 3.回滚事务
}
// 4.发送 mq 消息
Here, if the transaction is committed successfully, but mq message fails, it will lead to the payment data is not updated but the commission data generated inconsistencies.
Some students may think here, the message will be sent mq step of moving to a transaction, message sending fails, roll back the transaction, it is not perfect yet?
Pseudo-code as follows:
// 开始事务
try {
// 1.执行数据库操作
// 2.发送 mq 消息
// 3.提交事务
}catch (Exception e){
// 4.回滚事务
}
The above code looks really no problem, send the message fails, roll back the transaction.
The second step but in fact there may MQ message has been sent to the server, but not in a timely manner due to network problems MQ response message is received, leading messaging system considers the message failed to send a message.
This will lead to orders for transaction rollback, but the fee system was able to consume messages, and on both sides of the database inconsistent.
MQ familiar with the students might think, failed to send a message, you can retry ah.
Yes, we can increase the number of retries, resend the message. But here we need to pay attention, because the message is coupled in the transaction, will be stretched too much retry database transaction execution time, transaction time for too long, resulting in a transaction locks holding time becomes longer, affect the overall database throughput .
The actual business, the couple is not recommended to send messages in a database transaction.
Transaction message
RocketMQ transaction message is provided affairs function, you can achieve distributed transactions, thus ensuring the operation of the above matters with the message sent to either succeed or fail.
Use transaction message, the overall process is as follows:
First, we will send a half ( Half ) message to the MQ, open a notified transaction. Here half the message is not to say the message is incomplete, in fact, it contains all the full message content.
This half news is that the only difference with the ordinary news, before being filed, the news for consumers is not visible , consumers will not consume the news.
Once the message is sent half success, we can execute database transactions. Then according to the results of the transaction before deciding to commit or roll back the transaction message.
If the transaction is successfully submitted, a confirmation message will be sent to MQ, fee system can be successfully consumption to this message.
If the transaction is rolled back, the rollback will be sent a notification to MQ, MQ will then delete this message. For the fee system, we will not know the existence of this message.
This solves all either succeed or fail conformance requirements.
The actual process above is still a problem, if we commit / rollback transaction messages fails how to do?
For this problem, RocketMQ give a transaction pegging mechanism. We need need to register a callback interface for pegging the local state of affairs.
If no request is received RocketMQ committed or rolled back, will regularly go reverse lookup callback interface, then you can decide to roll back or commit the transaction based on anti-check the results.
RocketMQ transaction message flow as a whole is as follows:
Example transaction message code is as follows:
public class TransactionMQProducerExample {
public static void main(String[] args) throws MQClientException, InterruptedException, UnsupportedEncodingException {
TransactionMQProducer producer = new TransactionMQProducer("test_transaction_producer");
// 不定义将会使用默认的
ExecutorService executorService =
new ThreadPoolExecutor(2, 5, 100,
TimeUnit.SECONDS, new ArrayBlockingQueue<Runnable>(2000),
new ThreadFactory() {
@Override
public Thread newThread(Runnable r) {
Thread thread = new Thread(r);
thread.setName("client-transaction-msg-check-thread");
return thread;
}
});
producer.setExecutorService(executorService);
TransactionListener transactionListener = new TransactionListenerImpl();
producer.setTransactionListener(transactionListener);
// 改成自己的地址
producer.setNamesrvAddr("127.0.0.1:9876");
producer.start();
Order order = new Order("66666", "books");
Message msg =
new Message("transaction_tp",
JSON.toJSONString(order).getBytes(RemotingHelper.DEFAULT_CHARSET));
// 发送半消息
SendResult sendResult = producer.sendMessageInTransaction(msg, null);
System.out.println(sendResult.getSendStatus());
producer.shutdown();
}
public static class TransactionListenerImpl implements TransactionListener {
/**
* 半消息发送成功将会自动执行该逻辑
*
* @param msg
* @param arg
* @return
*/
@Override
public LocalTransactionState executeLocalTransaction(Message msg, Object arg) {
// 执行本地事务
Order order = null;
try {
order = JSON.parseObject(new String(msg.getBody(),
RemotingHelper.DEFAULT_CHARSET), Order.class);
boolean isSuccess = updateOrder(order);
if (isSuccess) {
// 本地事务执行成功，提交半消息
System.out.println("本地事务执行成功，提交事务事务消息");
return LocalTransactionState.COMMIT_MESSAGE;
} else {
// 本地事务执行成功，回滚半消息
System.out.println("本地事务执行失败，回滚事务消息");
return LocalTransactionState.ROLLBACK_MESSAGE;
}
} catch (Exception e) {
System.out.println("本地事务执行异常");
}
// 异常情况返回未知状态
return LocalTransactionState.UNKNOW;
}
/**
* 更新订单
* 这里模拟数据库更新，返回事务执行成功
*
* @param order
* @return
*/
private boolean updateOrder(Order order) throws InterruptedException {
TimeUnit.SECONDS.sleep(1);
return true;
}
/***
* 若提交/回滚事务消息失败，rocketmq 自动反查事务状态
* @param msg
* @return
*/
@Override
public LocalTransactionState checkLocalTransaction(MessageExt msg) {
try {
Order order = JSON.parseObject(new String(msg.getBody(),
RemotingHelper.DEFAULT_CHARSET), Order.class);
boolean isSuccess = queryOrder(order.getOrderId());
if (isSuccess) {
// 本地事务执行成功，提交半消息
return LocalTransactionState.COMMIT_MESSAGE;
} else {
// 本地事务执行成功，回滚半消息
return LocalTransactionState.ROLLBACK_MESSAGE;
}
} catch (Exception e) {
System.out.println("查询失败");
}
// 异常情况返回未知状态
return LocalTransactionState.UNKNOW;
}
/**
* 查询订单状态
* 模拟返回查询成功
*
* @param orderId
* @return
*/
private boolean queryOrder(String orderId) throws InterruptedException {
TimeUnit.SECONDS.sleep(1);
return true;
}
}
@Data
public static class Order {
private String orderId;
private String goods;
public Order(String orderId, String goods) {
this.orderId = orderId;
this.goods = goods;
}
}
}
The above code:
- We need to specify a producer for the single of
ProducerGroup - Need to inherit
TransactionListenerannotations callback interface,executeLocalTransactionthe method performs local affairs,checkLocalTranscationto perform check local affairs. - Returns the state of affairs in three ways:
- LocalTransactionState.UNKNOW intermediate state, RocketMQ will reverse lookup
- LocalTransactionState.COMMIT_MESSAGE commit the transaction, this news will follow this news consumption
- LocalTransactionState.ROLLBACK_MESSAGE, roll back the transaction, RocketMQ will delete this message
Note the use of transaction message points
The maximum number reverse lookup transaction message
Since a single anti-check message too many times, will result in the accumulation of semi message queue, affect performance. The default number of checks RocketMQ single message limit is 15 times.
We can modify brokerthe configuration file, add the following configuration:
# N 为最大检查次数
transactionCheckMax=N
After checking the number exceeds the maximum number, RocketMQ will discard the message and print the error log.
Discard the message you want to customize the behavior, needs to be modified RocketMQ broker side code, inheritance AbstractTransactionalMessageCheckListeneroverride resolveDiscardMsgmethods include custom logic.
Synchronized dual write mechanism
In order to ensure that the transaction message is not lost, and to ensure transaction integrity, you need to copy transaction message to the other nodes in the cluster, it is recommended to use a dual synchronous write mechanism.
Transaction anti-check time setting
We can set the following parameters, set the start reverse lookup transaction messages (counted from the transaction after the message has been successfully saved) after MQ server how long.
msg.putUserProperty(MessageConst.PROPERTY_CHECK_IMMUNITY_TIME_IN_SECONDS, "10");
Or we can broker.confset the following parameters:
# 单位为 ms,默认为 6 s
transactionTimeout=60000
Active transmitting side is greater than the priority of the configuration parameters provided brokerside configuration.
Further RocketMQ also configured to control a transactional message check interval:
## 默认为 60s
transactionCheckInterval=5000
If the custom configuration above, the transaction message check interval is 5 seconds, the transaction is a check time setting message 60 s.
This represents the broker checked once every 5s transaction message, if the transaction has a message to the MQ server time has not been exceeded 60s, this time will not be anti-check, until the time is greater than 60s.
Egg
Find transaction message data, we found there is a correlation error RocketMQ documents.
Documents Address: https://github.com/9526xu/rocketmq/blob/master/docs/cn/RocketMQ_Example.md
As two actually wrong , it should be amended as follows: AbstractTransactionalMessageCheckListenerand transactionTimeout.
issue Address: https://github.com/apache/rocketmq/issues/481
Easily modified a bit, submit PR . Haha, but also contribute to a force to open source projects.
Reference
- https://github.com/apache/rocketmq/issues/481
- https://github.com/9526xu/rocketmq/blob/master/docs/cn/RocketMQ_Example.md
- Time Geeks - message queue master class
One last word (seeking attention)
Always think before participating in open source projects is difficult, until recently involved in a series of two open source projects to modify, only to find that in fact is not so difficult to imagine. Since version changes, some of which are open source project documentation errors, and if we see, easily repair it, which is to contribute to a force to open source projects.
Caishuxueqian, it is inevitable there will be flaws, if you find the wrong place, please leave a message and pointed out to me, and I modify them.
Thanks again for your reading, I was downstairs small Heige , a tool has not yet bald ape, the next article we will meet ~
I welcome the attention of the public number: Interpreter program, and getting daily dry push. If you are interested in my topic content, you can focus on my blog: studyidea.cn