In a distributed system, in order to ensure data consistency must use distributed transactions. Distributed transaction implementation very many, mainly introduce today use RocketMQ transaction messages, realize distributed transaction.
End of the article there are eggs, watching the walk
Why transaction message?
Many students may not know what Affairs news is, it does not matter, give a real business scenario, let's take you look at the problems of ordinary messages.
Above business scenario, when users pay for success, will update the payment order, then send MQ message. The system will be charged by the pull message, save the calculated fees and fees to another database.
Since the calculation of this fee may be calculated off-line step, the process used here MQ decoupled payment and fee calculation.
The main process involves three steps:
- Update order data
- Send a message to MQ
- Pull message handling system
The steps mentioned above, any will fail, if we do not deal with, it will cause data inconsistency on both sides, will result in the following two situations:
- Order data updated, commission data not generated
- Fees generated data, order data has not been updated
This is real money involved, once a small calculation will result in capital loss , really can not afford to lose!
For the final step is concerned, it is relatively simple. If the consumer fails the message, as long as no submitted news confirmed, MQ server will automatically retry.
The biggest problem is that we can not guarantee the update operation and send a message consistency. Whether we use the data to update the order, then send a message, or to send a message, and then update the order data, in the presence of a success, a failure might.
As shown below, using the first message transmitted before updating the database.
After the above process messages sent successfully, then a local transaction commits. This process looks perfect, but imagine if the database fails while the transaction is committed, resulting in a transaction rollback.
But this time the message has been sent, can not be withdrawn. This leads to commission the system will immediately consume messages, calculate fees and update the database. This has resulted in the payment data is not updated, case handling system has generated inconsistent.
If we look at the process that trans is not like it?
We use the following pseudo-code representation:
// start transaction
try
{
// 1. perform database operations
2. // commit the transaction
}
catch
(Exception e){
// 3. roll back the transaction
}
4. Send message // mq
Here, if the transaction is committed successfully, but mq message fails, it will lead to the payment data is not updated but the commission data generated inconsistencies.
Some students may think here, the message will be sent mq step of moving to a transaction, message sending fails, roll back the transaction, it is not perfect yet?
Pseudo-code as follows:
// start transaction
try
{
// 1. perform database operations
// 2. Send a message mq
// 3. Submit Affairs
}
catch
(Exception e){
4. // roll back the transaction
}
The above code looks really no problem, send the message fails, roll back the transaction.
The second step but in fact there may MQ message has been sent to the server, but not in a timely manner due to network problems MQ response message is received, leading messaging system considers the message failed to send a message.
This will lead to orders for transaction rollback, but the fee system was able to consume messages, and on both sides of the database inconsistent.
MQ familiar with the students might think, failed to send a message, you can retry ah.
Yes, we can increase the number of retries, resend the message. But here we need to pay attention, because the message is coupled in the transaction, will be stretched too much retry database transaction execution time, transaction time for too long, resulting in a transaction locks holding time becomes longer, affect the overall database throughput .
The actual business, the couple is not recommended to send messages in a database transaction.
Transaction message
RocketMQ transaction message is provided affairs function, you can achieve distributed transactions, thus ensuring the operation of the above matters with the message sent to either succeed or fail.
Use transaction message, the overall process is as follows:
First, we will send a half ( Half ) message to the MQ, open a notified transaction. Here half the message is not to say the message is incomplete, in fact, it contains all the full message content.
This half news is that the only difference with the ordinary news, before being filed, the news for consumers is not visible , consumers will not consume the news.
Once the message is sent half success, we can execute database transactions. Then according to the results of the transaction before deciding to commit or roll back the transaction message.
If the transaction is successfully submitted, a confirmation message will be sent to MQ, fee system can be successfully consumption to this message.
If the transaction is rolled back, the rollback will be sent a notification to MQ, MQ will then delete this message. For the fee system, we will not know the existence of this message.
This solves all either succeed or fail conformance requirements.
The actual process above is still a problem, if we commit / rollback transaction messages fails how to do?
For this problem, RocketMQ give a transaction pegging mechanism. We need need to register a callback interface for pegging the local state of affairs.
If no request is received RocketMQ committed or rolled back, will regularly go reverse lookup callback interface, then you can decide to roll back or commit the transaction based on anti-check the results.
RocketMQ transaction message flow as a whole is as follows:
Example transaction message code is as follows:
public
class TransactionMQProducerExample { public static void main(String[] args) throws MQClientException, InterruptedException, UnsupportedEncodingException { TransactionMQProducer producer =
new
TransactionMQProducer(
"test_transaction_producer"
);
// definition will not use the default
ExecutorService executorService =
new
ThreadPoolExecutor(2, 5, 100, TimeUnit.SECONDS,
new
ArrayBlockingQueue<Runnable>(2000),
new
ThreadFactory() {
@Override
public Thread newThread(Runnable r) { Thread thread =
new
Thread(r); thread.setName(
"client-transaction-msg-check-thread"
);
return
thread; } }); producer.setExecutorService(executorService); TransactionListener transactionListener =
new
Transaction impl Listener (); producer.setTransactionListener (Transaction Listener);
// change their address
producer.setNamesrvAddr(
"127.0.0.1:9876"
); producer.start(); Order order =
new
Order(
"66666"
,
"books"
); Message msg =
new
Message(
"transaction_tp"
, JSON.toJSONString(order).getBytes(RemotingHelper.DEFAULT_CHARSET));
// send the message half
SendResult sendResult = producer.sendMessageInTransaction(msg,
null
); System.out.println(sendResult.getSendStatus()); producer.shutdown(); }
public
static
class TransactionListenerImpl implements TransactionListener {
/ * ** semi-successful message transmission will automatically execute the logic * * @param msg * @param arg * @return * /
@Override
public LocalTransactionState executeLocalTransaction(Message msg, Object arg) {
// execute local transactions
Order order =
null
;
try
{ order = JSON.parseObject(
new
String(msg.getBody(), RemotingHelper.DEFAULT_CHARSET), Order.class);
boolean
isSuccess = updateOrder(order);
if
(isSuccess) {
// local transaction is executed successfully, submit semi message
System.out.println(
"Local transaction is successful, the transaction is committed transactional messaging"
);
return
LocalTransactionState.COMMIT_MESSAGE; }
else
{
// local transaction is executed successfully, the message is rolled back half
System.out.println(
"Local transaction fails, roll back the transaction news"
);
return
LocalTransactionState.ROLLBACK_MESSAGE; } }
catch
(Exception e) { System.out.println(
"Local transaction execution exception"
); }
// anomalies returned an unknown state
return
LocalTransactionState.UNKNOW; }
/ ** * Update * Order here simulation database updates, return to perform a successful transaction * * @param order * @return * /
private boolean updateOrder(Order order) throws InterruptedException { TimeUnit.SECONDS.sleep(1);
return
true
; }
/ *** * If the commit / rollback transaction messages fails, rocketmq automatic anti-check transaction status * @param msg * @return * /
@Override
public LocalTransactionState checkLocalTransaction(MessageExt msg) {
try
{ Order order = JSON.parseObject(
new
String(msg.getBody(), RemotingHelper.DEFAULT_CHARSET), Order.class);
boolean
isSuccess = queryOrder(order.getOrderId());
if
(isSuccess) {
// local transaction is executed successfully, submit semi message
return
LocalTransactionState.COMMIT_MESSAGE; }
else
{
// local transaction is executed successfully, the message is rolled back half
return
LocalTransactionState.ROLLBACK_MESSAGE; } }
catch
(Exception e) { System.out.println(
"Query failed"
); }
// anomalies returned an unknown state
return
LocalTransactionState.UNKNOW; }
/ ** * Check Order Status * analog return the query was successful * * @param orderId * @return * /
private boolean queryOrder(String orderId) throws InterruptedException { TimeUnit.SECONDS.sleep(1);
return
true
; } }
@Data
public
static
class Order {
private
String orderId;
private
String goods; public Order(String orderId, String goods) {
this
.orderId = orderId;
this
.goods = goods; } }}
The above code:
- We need to specify a producer for the single of ProducerGroup
- Need to inherit TransactionListener notes callback interface, which executeLocalTransaction method executes local transactions, checkLocalTranscation to perform check local affairs.
- Returns the state of affairs in three ways:
- LocalTransactionState.UNKNOW intermediate state, RocketMQ will reverse lookup
- LocalTransactionState.COMMIT_MESSAGE commit the transaction, this news will follow this news consumption
- LocalTransactionState.ROLLBACK_MESSAGE, roll back the transaction, RocketMQ will delete this message
Note the use of transaction message points
The maximum number reverse lookup transaction message
Since a single anti-check message too many times, will result in the accumulation of semi message queue, affect performance. The default number of checks RocketMQ single message limit is 15 times.
We can modify the broker configuration file, add the following configuration:
# N is the maximum number of check transactionCheckMax = N
After checking the number exceeds the maximum number, RocketMQ will discard the message and print the error log.
Discard the message you want to customize the behavior, needs to be modified RocketMQ broker side code AbstractTransactionalMessageCheckListener override inherited methods resolveDiscardMsg added custom logic.
Synchronized dual write mechanism
In order to ensure that the transaction message is not lost, and to ensure transaction integrity, you need to copy transaction message to the other nodes in the cluster, it is recommended to use a dual synchronous write mechanism.
Transaction anti-check time setting
We can set the following parameters, set the start reverse lookup transaction messages (counted from the transaction after the message has been successfully saved) after MQ server how long.
msg.putUserProperty(MessageConst.PROPERTY_CHECK_IMMUNITY_TIME_IN_SECONDS,
"10"
);
Or we can set broker.conf the following parameters:
# Units of ms, default is 6 stransactionTimeout = 60000
Active transmitting side is greater than the priority level set configuration parameters broker side configuration.
Further RocketMQ also configured to control a transactional message check interval:
The default is ## 60stransactionCheckInterval = 5000
If the custom configuration above, the transaction message check interval is 5 seconds, the transaction is a check time setting message 60 s.
This represents the broker checked once every 5s transaction message, if the transaction has a message to the MQ server time has not been exceeded 60s, this time will not be anti-check, until the time is greater than 60s.
Egg
Find transaction message data, we found there is a correlation error RocketMQ documents.
Documents Address:
As two is actually wrong and should be revised to: AbstractTransactionalMessageCheckListener and transactionTimeout.
issue Address:
Easily modified a bit, submit PR . Haha, but also contribute to a force to open source projects.
Author: Silly bulk
link: https: //juejin.im/post/5e8d7180e51d4546ea2835c5
Source: Nuggets
copyright reserved by the authors. Commercial reprint please contact the author authorized, non-commercial reprint please indicate the source.
