Two Foreword
Ali Yunqi 2017 Conference " crack worldwide technical problems! GTS allows distributed transactions simple and efficient "in claiming that Ali proposed a crack worldwide problem the ultimate solution for the distributed transaction, whether the reliability, processing speed or are ahead of all the technology available in the market. It is regrettable that a project to not open source , and secondly, you must rely on Ali cloud of distributed database. After all, the guy to eat can not be easily seen by people there .
Even so, "the world's problems ...," an article on the transaction or inductive still pretty place: "a seemingly simple function, may need to call multiple internal" service "and the operation of multiple database or fragmented to achieve single technology and solutions have been unable to meet these complex scenarios. Thus, distributed system architecture in a distributed transaction is a challenge around the past.
What is a distributed transaction? Simply put, that is a big small operation operated by different composition, these small operations are distributed on different servers, distributed transactions need to ensure that these small operations either all succeed, or all fail . "
For chestnut:
You buy things on Taobao, you need to deduct money and merchandise inventory -1 bar. However, the charge and inventory owned by two services, both through a series of intermediate intermediate service layer network, gateway, host, etc., in case any one place is a problem, such as network jitter, excursion wait, can lead to inconsistencies such as debit successful, but not inventory-1, there will be oversold phenomenon, and this is distributed transaction need to be resolved
Three 2-phase commit (2PC, 3PC, etc.)
Stage 2 submission is distributed transaction traditional solutions advanced so far still widespread. When a transaction across multiple nodes, in order to maintain the transaction ACIDcharacteristics, need to be introduced as a coordinator result of the operation to unify control of all nodes (called participants) and finally whether or not to instruct the operating results of these nodes actually submitted (for example, the updated data is written to disk, and so on). Therefore, the idea of two-phase commit protocol algorithm can be summarized as follows: Participants will inform the coordinator of the success of the operation, and then decide based on the feedback information from the coordinator of all participants each participant whether to commit operation or abort the operation.
For example in order to meet
ABC D four to organize a conference, meeting time determined, it may assume that A coordinator, B Bingding participants.
Voting phase
- A message sent to B Bingding, week eight pm meeting if there is time;
- A reply time;
- B reply time;
- Propionic delay in reply, this time for this event, B and C are in blocking state, the algorithm can not continue;
- Propan reply time (or no time);
Commit Phase
- A result of the coordinator to the collected feedback Bingding B (feedback when, and how the result of feedback, depends on time and decisions of the prop in this example);
- B received;
- Propionic received;
- Ding received;
To lock all resources not only participants, but also to lock the resource coordinator, large overhead. Sentence summary is: 2PC inefficient, high concurrency very friendly.
Quote 《世界性难题...》a text of the original words "Foreign Affairs has distributed commercial products for decades and precipitation technology-based XA model, hardware and software under the same conditions, after opening a distributed transaction throughput is often an order of magnitude of decline."
In addition there are three-phase commit
We are interested may wish to study under
Four flexible Affairs
The so-called flexible transaction is relatively rigid mandatory terms of the transaction lock table. Flow into the next: Transaction Server A, if successfully implemented, then the transaction will first be submitted A, B also perform well if the transaction, the transaction B also be submitted, even if the entire transaction is completed. However, if the transaction fails B, B itself rollback transaction, then transaction A has been committed, it is necessary to perform a compensating operation, the operation performed by transaction A already filed for the inverse operation, before the return to the state A transaction unexecuted .
The disadvantage is too intrusive service, but also compensation operations, the lack of universality, not large-scale promotion.
Five news programs of eventual consistency solve RocketMQ
Currently queue-based messaging solutions have Ali RocketMQ, which implements the 半消息solution, somewhat similar to Paxos algorithm , the specific process is as follows
The first stage: application executing business and upstream transmission message MQ
- Application of the upstream transmission acknowledgment message to be reliable messaging system
- Reliable messaging system saves the message to be acknowledged and return
- Upstream applications perform local business
- Reliable Messaging application notifies the upstream acknowledgment and sends the message service has been performed.
Reliable messaging system to send a status modification message and the message is delivered to the status MQ middleware
The second stage: MQ message and listen for downstream applications execute business
Downstream applications monitor MQ message and execution of business, and the results of the consumer message notification service to reliable sources.
- MQ messaging applications monitor downstream components and obtain messages
- The MQ message body downstream application information processing local service
- Application of the downstream MQ
- A confirmation message is consumed
- Downstream application notification system to reliable sources message was successfully consumption, the message status changes to reliable sources as complete
RocketMQIt looks like is an advanced implementation, but the problem is 缺乏文档, whether it is in the Apache project page or pages on Ali, at most only tell you how to use, and what principle or guiding sorely lacking.
Of course, if you buy a special on Ali cloud RocketMQservices, it must have been another matter. But if you try to deploy and use in their service environment, we think necessary after considerable learning curve.毕竟是人家吃饭的家伙嘛
The final six RabbitMQ messaging solutions for the realization of consistency
RabbitMQFollowed AMQP规范by a message confirming mechanism to ensure that: as long as the message is sent, you can ensure that consumer spending to achieve the ultimate message consistency. And open source, the document also exceptionally rich, looks like a good carrier to achieve a distributed transaction
6.1 RabbitMQ message acknowledgment.
The whole transmission process is as follows rabbitmq
1. Producer send a message to the message service
2. If the message persistence to complete the landing, then a flag is returned to the producer. After the producer to get this confirmation can be assured of success, he said the message was finally sent to a messaging service. Otherwise, enter the exception handling process.
rabbitTemplate.setConfirmCallback((correlationData, ack, cause) -> {
if (!ack) {
//try to resend msg
} else {
//delete msg in db
}
});3. Messaging Service to send a message to consumers
4. Consumers receive and process messages, if the treatment is successful manual confirmation. When the news service to get this confirmation was assured, he said consumer finally completed. Otherwise, retransmission, or go to exception handling.
final Consumer consumer = new DefaultConsumer(channel) {
@Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
String message = new String(body, "UTF-8");
System.out.println(" [x] Received '" + message + "'");
try {
doWork(message);
} finally {
//确认收到消息
channel.basicAck(envelope.getDeliveryTag(), false);
}
}
};
6.2 Abnormal
Let's look at four kinds of abnormalities may send
1. Direct unreachable message service
Network broken, thrown, rolled back to direct business. If there is connection closedan error, directly increase the connectionnumber to
connectionFactory.setChannelCacheSize(100);
2. The message has reached the server, but when abnormal returns appear
rabbitmqIt provides confirmation ack mechanism can be used to confirm whether the message has returned. So before we can send in the db (or memory relational database) about the pre-existing message, if an exception is ack retransmit
/**confirmcallback用来确认消息是否有送达消息队列*/
rabbitTemplate.setConfirmCallback((correlationData, ack, cause) -> {
if (!ack) {
//try to resend msg
} else {
//delete msg in db
}
});
/**若消息找不到对应的Exchange会先触发returncallback */
rabbitTemplate.setReturnCallback((message, replyCode, replyText, tmpExchange, tmpRoutingKey) -> {
try {
Thread.sleep(Constants.ONE_SECOND);
} catch (InterruptedException e) {
e.printStackTrace();
}
log.info("send message failed: " + replyCode + " " + replyText);
rabbitTemplate.send(message);
});
3. After the message is delivered, the message service hung himself
If the message is persistent, it ack= trueis complete after the message persistence, that is written to disk and then later sent to ensure that the message already exists on the hard disk, in case of a hung message service, message service is the ability to recover and then resend the message
4. Non-Delivery consumers
After receiving the message service message, the message is in a state of "UNACK" until the client a confirmation message
channel.basicQos(1); // accept only one unack-ed message at a time (see below)
final Consumer consumer = new DefaultConsumer(channel) {
@Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
String message = new String(body, "UTF-8");
System.out.println(" [x] Received '" + message + "'");
try {
doWork(message);
} finally {
//确认收到消息
channel.basicAck(envelope.getDeliveryTag(), false);
}
}
};
boolean autoAck = false;
channel.basicConsume(TASK_QUEUE_NAME, autoAck, consumer);
5. Confirm the message is lost
Assuming that the confirmation message is lost when the message is returned, the message service will resend the message. Note that if you set up autoAck= false, but not any response channel.baskAckdid not answer channel.baskNack, it will lead to very serious error: the message queue will be blocked to live, so, in any case must answer
6. consumer business to handle exceptions
Message listener accepts and processes the message, assuming throw an exception, the first phase has been completed thing, if you want to configure rollback is too much trouble, even when doing a transaction compensation may also compensate for the failed transaction, so here you can do a repeat execution, such as guavathe retry, set up a time to cycle index, if still fails after n, e-mail, text messaging, employing meat to reveal all the details.
Seven summary
《世界性难题...》In an article for distributed transactions in several ways to achieve the image of induction
You go to work every day, to go through a 10 km two-lane road just to reach the company. This road is blocked, often we need two or three hours working hours does not guarantee that this is 2PC problem - slow.
Select a very winding, but very little traffic jam 30 kilometers long road, which is selected b. Working hours is guaranteed, but must get up early, pay sufficient time and gasoline. This is a question flexible transaction must be rolled back to a specific business, modular difficult
Select a bit around, 20 km long mountain road, bumpy, can go only suv, this is the final transaction message consistency. Introduced a new messaging middleware, it requires additional development costs. But Our development CoolMQ already components of the package, only need to send, receive, will be able to meet the requirements of the transaction. Currently there are lectures of the program , you can choose according to their needs.