Article directory
Preface
I. Introduction
RocketMQ can be understood as a special storage system. The special thing about this storage system is that data is generally only used once. In this case, it is very important to ensure that the message consumed once is not lost. This article will analyze how RocketMQ ensures that messages are not lost.
2. Under what circumstances will the message be lost?
Since the message goes from the producer to the broker and is finally consumed by the consumer, it has gone through at least three applications and two rpc calls. Since the rpc call will have a third situation besides success and failure, the message will be unreliable.
So, what means do we have to improve the reliability of news? This article will analyze the means to ensure that messages are not lost from the producer side, consumer side, and broker side respectively.
3. How to ensure that messages are not lost?
3.1. Producer side
The producer sends messages, either synchronously or asynchronously to the Broker.
If we have higher requirements for message reliability, we can choose to send it synchronously. On the RocketMQ client, synchronous sending comes with a retry mechanism. If the sending in synchronous mode fails, it will rotate to the next Broker.
Insert image description here
What should I do if the message fails to be sent even after retrying?
At this time, we need to consider the solution to the failure of sending - the business system implements it by itself. The business system can store the message and use mechanisms such as scheduled tasks to resend the message.
3.2. Broker side
As a Broker, his main responsibility is to store messages persistently and deliver the messages to the consumer at least once.
Since the message is stored on the disk, the persistence mechanism involves the disk flushing mechanism. RocketMQ supports synchronous disk flushing and asynchronous disk flushing mechanisms.
RocketMQ defaults to writing to the buffer when processing a request to send a message. It will not synchronize the disk immediately. It will refresh the disk by timing 5s.
SYNC_FLUSH, flush the disk synchronously, and then flush the disk synchronously. Returned to the client, timeout 5s
ASYNC_FLUSH, asynchronous disk flushing, refreshed every 200ms, high performance
The above mechanism can ensure storage reliability. In addition to ensuring the reliability of stored messages, the broker also needs to ensure that the message can be delivered to consumers for consumption once. How does the Broker ensure that the message will be delivered to the consumer?
The Broker side has designed a retry mechanism. If message consumption fails, the message will be written to the queue under the retry topic and will be retried up to 16 times to send to the consumer.
If the message is still not consumed successfully after 16 times, the Broker will write the message to the dead letter queue.
3.3. Consumer side
The message is delivered to the consumer. If the consumer fails to consume, it cannot return ack to the broker. Generally, the ack mechanism needs to be set to be submitted manually. If the consumer fails to consume the message, CONSUME_SUCCESS will not be returned. Returning RECONSUME_LATER indicates that the broker needs to deliver the message again.
What needs to be noted here is that since the broker has a retry mechanism to ensure that messages are not lost, it may lead to repeated delivery of messages. Therefore, the consumer needs to do idempotent processing, which is generally processed according to business rules.
4. Summary
The messaging system decouples different systems, which not only improves the high throughput and asynchronous performance of the system, but also brings challenges to system stability. Ensuring that message reliability is not lost is a very critical stability challenge. This article Consider corresponding solutions from the producer, broker, and consumer ends to deal with the means of not losing messages.