How does rocketmq ensure that the queue is completely consumed in sequence?

In fact, RocketMQ supports sequential consumption. But this order, not the global order, just the partition order. To global order only one partition. The reason why your scenario does not appear to be sequential is because when sending a message, the message is sent to the unreachable queue (partition) in a polling manner by default. As shown in the figure: When the consumer consumes, it will be allocated to multiple queues, and multiple queues are pulled and submitted for consumption at the same time. As shown in the figure: But in the same queue, RocketMQ can indeed guarantee FIFO. So to achieve sequential messages, how should it be implemented - to ensure that messages are delivered to the same queue. The sample code of the rocketmq message production side is as follows: According to this example, the order number is taken and a modulo operation is performed and then thrown into the selector. The selector ensures that the same modulo will be delivered to the same queue. That is: the same order number ---> have the same mode ---> have the same queue. It will end up like this:In this way, the same batch that you need to consume sequentially will definitely be delivered to the same queue, the same queue will definitely be delivered to the same consumption instance, and the same consumption instance must be pulled sequentially and submitted to the thread pool sequentially, as long as the consumer side is guaranteed Consume sequentially and you're done! How to guarantee sequential consumption? If you use MessageListenerOrderly, it comes with this implementation. If you use MessageListenerConcurrently, you need to change the thread pool to single-threaded mode. (It does not matter if the queue is allocated to others because of the triggering of rearrangement. Since the messages of the queue are always FIFO, at most the messages that have been consumed are only repeated, and the order in the queue can still be guaranteed) But there are indeed some abnormal scenarios that will lead to chaos sequence. If the master goes down, the number of write queues changes. If the modulo selector is still used, the front of a batch of order number messages will be hashed to q0, and the latter may be scattered to q1, so the order cannot be guaranteed. Unless you choose to sacrifice the failover feature, if the master hangs up, it cannot send the next batch of messages. From the consumer side, if you want to ensure that this batch of messages is M1 consumption and then M2 consumption, you can use the MessageListenerOrderly interface, but in this case, there will be the following problems: 1. When encountering a message that fails, it cannot be skipped, and the current queue consumption is suspended2 . The MessageListenerOrderly of the current version of RocketMQ cannot consume messages from slaves. For more analysis, please refer to: RocketMQ--Explanation of Roles and Terminology-Schrodinger's Outlet Pig RocketMQ--Detailed Explanation of Horizontal Scaling and Load Balancing-Schrodinger's Outlet Pig

 

 

https://www.zhihu.com/question/30195969

Guess you like

Origin http://10.200.1.11:23101/article/api/json?id=326709560&siteId=291194637