"In-depth understanding RocketMQ" - MQ message delivery mechanism

0. Introduction

RocketMQ message delivery is divided into two: one is 生产者to deliver the MQ Broker; Another is the MQ broker to 消费者 delivery (this 投递statement is set forth in terms of message passing, in fact, is the bottom layer 消费者from the MQ broker Pull the pulled). This article from the perspective of models to illustrate these two mechanisms.

1. RocketMQ message model

Model RocketMQ whole message is not complicated, as shown below:

A Topic(消息主题)possible corresponding to the plurality of actual message queue (MessgeQueue)
on the underlying implementation, in order to improve availability and flexibility of MQ, a Topic stored in the actual process, using multiple queue embodiment, the specific forms shown above. Each message queue should ensure that, in use, first in first out (FIFO, First In First Out) manner consumption.
So, based on this model, it will come out of two questions:

Topic producer when sending the same message, the message body should be placed where a message queue (the MessageQueue) in?
Consumers in the consumer news, which should pull message from the message queue?

2. Producers (Producer) message delivery strategy

2.1 Default Delivery Method: Queue队列Polling algorithm Delivery

By default, using the most simple polling algorithm, which has a good feature is to ensure that every Queue队列message delivery number as uniform as possible, the algorithm as shown below:

/**
*  根据 TopicPublishInfo Topic发布信息对象中维护的index，每次选择队列时，都会递增
*  然后根据 index % queueSize 进行取余，达到轮询的效果
*
*/
public MessageQueue selectOneMessageQueue(final TopicPublishInfo tpInfo, final String lastBrokerName) {
        return tpInfo.selectOneMessageQueue(lastBrokerName);
}

/**
*  TopicPublishInfo Topic发布信息对象中
*/
public class TopicPublishInfo {
    //基于线程上下文的计数递增，用于轮询目的
    private volatile ThreadLocalIndex sendWhichQueue = new ThreadLocalIndex();
   

    public MessageQueue selectOneMessageQueue(final String lastBrokerName) {
        if (lastBrokerName == null) {
            return selectOneMessageQueue();
        } else {
            int index = this.sendWhichQueue.getAndIncrement();
            for (int i = 0; i < this.messageQueueList.size(); i++) {
                //轮询计算
                int pos = Math.abs(index++) % this.messageQueueList.size();
                if (pos < 0)
                    pos = 0;
                MessageQueue mq = this.messageQueueList.get(pos);
                if (!mq.getBrokerName().equals(lastBrokerName)) {
                    return mq;
                }
            }
            return selectOneMessageQueue();
        }
    }

    public MessageQueue selectOneMessageQueue() {
        int index = this.sendWhichQueue.getAndIncrement();
        int pos = Math.abs(index) % this.messageQueueList.size();
        if (pos < 0)
            pos = 0;
        return this.messageQueueList.get(pos);
    }
}

2.2 The default mode of delivery enhancements: Based on Queue队列polling algorithm and 消息投递延迟最小strategy delivery

The default delivery is relatively simple, but also exposed a problem that some Queue队列may be due to their backlog and other reasons, it may be long in the process of delivery, for this Queue队列will affect the subsequent delivery of results.
Based on this phenomenon, RocketMQ after every send an MQ message, the message will be delivered at the statistics 时间延迟, according to this 时间延迟, we can know what to Queue队列deliver faster.
In this scenario, it will give priority to the use of 消息投递延迟最小tactics, if not take effect, re-use Queue队列轮询way.

public class MQFaultStrategy {
    /**
     * 根据 TopicPublishInfo 内部维护的index,在每次操作时，都会递增，
     * 然后根据 index % queueList.size(),使用了轮询的基础算法
     *
     */
    public MessageQueue selectOneMessageQueue(final TopicPublishInfo tpInfo, final String lastBrokerName) {
        if (this.sendLatencyFaultEnable) {
            try {
                // 从queueid 为 0 开始，依次验证broker 是否有效，如果有效
                int index = tpInfo.getSendWhichQueue().getAndIncrement();
                for (int i = 0; i < tpInfo.getMessageQueueList().size(); i++) {
                    //基于index和队列数量取余，确定位置
                    int pos = Math.abs(index++) % tpInfo.getMessageQueueList().size();
                    if (pos < 0)
                        pos = 0;
                    MessageQueue mq = tpInfo.getMessageQueueList().get(pos);
                    if (latencyFaultTolerance.isAvailable(mq.getBrokerName())) {
                        if (null == lastBrokerName || mq.getBrokerName().equals(lastBrokerName))
                            return mq;
                    }
                }
                
                // 从延迟容错broker列表中挑选一个容错性最好的一个 broker
                final String notBestBroker = latencyFaultTolerance.pickOneAtLeast();
                int writeQueueNums = tpInfo.getQueueIdByBroker(notBestBroker);
                if (writeQueueNums > 0) {
                     // 取余挑选其中一个队列
                    final MessageQueue mq = tpInfo.selectOneMessageQueue();
                    if (notBestBroker != null) {
                        mq.setBrokerName(notBestBroker);
                        mq.setQueueId(tpInfo.getSendWhichQueue().getAndIncrement() % writeQueueNums);
                    }
                    return mq;
                } else {
                    latencyFaultTolerance.remove(notBestBroker);
                }
            } catch (Exception e) {
                log.error("Error occurred when selecting message queue", e);
            }
          // 取余挑选其中一个队列
            return tpInfo.selectOneMessageQueue();
        }

        return tpInfo.selectOneMessageQueue(lastBrokerName);
    }
}

Order message delivery embodiment 2.3

The two drop boxes belonging to the scene of a sequence of message delivery is not required, such a delivery speed and more efficient. In some scenarios, it is necessary to ensure that the order of the same type of message delivery and consumption.
For example, suppose we have TOPIC TOPIC_SALE_ORDER, have this Topic. 4 th Queue队列, the Topic for transmitting orders changes state, assuming a state in order: 未支付, 已支付, 发货中(处理中), 发货成功, 发货失败.
In time sequence, the producer may be generated from the sequence following several messages:
订单T0000001:未支付 -> 订单T0000001:已支付 -> 订单T0000001:发货中(处理中) -> 订单T0000001:发货失败
After the message to the MQ, possibly due to poll delivered, stored in the MQ message may be as follows:

In this case, we want to 消费者order and consume the message we send is the same, however, delivery and consumption mechanisms MQ above, we can not guarantee the order is correct for the unusual order of messages, 消费者 even if there is some fault-tolerant state, We can not fully handle so many combinations appear randomly.

Based on the above, the RockeMQuse of this implementation: the same message order number, through a certain policy, it is placed in a queue队列中, and 消费者then use a certain strategy (a separate processing a thread queue, to ensure that the order of message processing) sequential, to ensure consumption

As to how the order is to ensure that the consumer line of consumption, then start the detailed follow-up, we look at 生产者is how to be able to send the same message to the same order number queue队列: the
producer in the process of message delivery, the use of MessageQueueSelector a queue selection strategy interfaces, which are defined as follows:

package org.apache.rocketmq.client.producer;

import java.util.List;
import org.apache.rocketmq.common.message.Message;
import org.apache.rocketmq.common.message.MessageQueue;

public interface MessageQueueSelector {
        /**
         * 根据消息体和参数，从一批消息队列中挑选出一个合适的消息队列
         * @param mqs  待选择的MQ队列选择列表
         * @param msg  待发送的消息体
         * @param arg  附加参数
         * @return  选择后的队列
         */
        MessageQueue select(final List<MessageQueue> mqs, final Message msg, final Object arg);
}

Accordingly, currently offers several RocketMQ achieve the following:

The default implementation:

Delivery strategy	Policy implementation class	Explanation
Random allocation strategy	SelectMessageQueueByRandom	Using a simple random number selection algorithm
Hash-based allocation strategy	SelectMessageQueueByHash	The Hash value of the additional parameter according to the size of the remainder of the message queue list to take to obtain the message queue index
Allocation strategy based on the machine room location	SelectMessageQueueByMachineRoom	The open source version is no specific implementation, the basic aim should be to allocate proximity principle of the machine

The code now probably look at strategies to achieve:

public class SelectMessageQueueByHash implements MessageQueueSelector {

    @Override
    public MessageQueue select(List<MessageQueue> mqs, Message msg, Object arg) {
        int value = arg.hashCode();
        if (value < 0) {
            value = Math.abs(value);
        }

        value = value % mqs.size();
        return mqs.get(value);
    }
}

The actual operation of the following sample code, by the order number as the hash operands, can guarantee the same message order number can fall on the same queue队列上.

rocketMQTemplate.asyncSendOrderly(saleOrderTopic + ":" + tag, msg,saleOrderId /*传入订单号作为hash运算对象*/, new SendCallback() {
            @Override
            public void onSuccess(SendResult sendResult) {
                log.info("SALE ORDER NOTIFICATION SUCCESS：{}",sendResult.getMsgId());
            }
            @Override
            public void onException(Throwable throwable) {
                 //exception happens
            }
        });

3. How to allocate for the consumer `queue队列`?

RocketMQ news for consumer spending in two forms:

BROADCASTING: Broadcast consumer, in this mode, a message is notified to each消费者
CLUSTERING: Cluster consumption, this mode, a message will be delivered to a maximum of 消费者the conduct consumption
mode as follows:

广播式The message mode is relatively simple, let's introduce the next 集群式. For consumption mode MessageModel.CLUSTERINGwhen the consumer, a need to ensure that 消息only needs to be consumed once in the entire cluster. In fact, in RoketMQ underlying message is assigned to the specified achieve consumers through queue队列distribution to 消费者way to complete: that is, 消息the unit where the message is assigned queue队列. which is:

The queue队列particular assigned to the 消费者post, queue队列all messages within will be assigned to 消费者consume.

RocketMQ policy defines the interface AllocateMessageQueueStrategyfor a given 消费者分组, and 消息队列列表, 消费者列表, 当前消费者should be assigned to what queue队列is defined as follows:

/**
 * 为消费者分配queue的策略算法接口
 */
public interface AllocateMessageQueueStrategy {

    /**
     * Allocating by consumer id
     *
     * @param consumerGroup 当前 consumer群组
     * @param currentCID 当前consumer id
     * @param mqAll 当前topic的所有queue实例引用
     * @param cidAll 当前 consumer群组下所有的consumer id set集合
     * @return 根据策略给当前consumer分配的queue列表
     */
    List<MessageQueue> allocate(
        final String consumerGroup,
        final String currentCID,
        final List<MessageQueue> mqAll,
        final List<String> cidAll
    );

    /**
     * 算法名称
     *
     * @return The strategy name
     */
    String getName();
}

Accordingly, RocketMQ offers several achieve the following:

Algorithm name	meaning
`AllocateMessageQueueAveragely`	The average allocation algorithm
`AllocateMessageQueueAveragelyByCircle`	The average allocation algorithm based on ring
`AllocateMachineRoomNearby`	Room near the algorithm based on the principle of
`AllocateMessageQueueByMachineRoom`	Room allocation algorithm
`AllocateMessageQueueConsistentHash`	Consistency hash algorithm
`AllocateMessageQueueByConfig`	Allocation algorithm based on configuration

In order to understand the basic principles about the above algorithm, we assume an example, all of the following algorithm will be based on this example to explain.

Suppose the current queue with a queue topic 10more, consumers total 4number, as shown below:

The following principle introduced in turn:

3.1 AllocateMessageQueueAveragely- Average allocation algorithm

Here the so-called average allocation algorithm, does not mean completely average in the strict sense, as in the example above, the 10 queue, and consumers only 4, is not divisible relations, in addition to the divisible extra queue, They will in turn be shared equally in accordance with the order of the consumer.
According to the above example, view 10/4=2, i.e., each represents 消费者an average of two shared equally Queue; and 10%4=2, in addition to shared equally i.e., extra 2 queuehas not been allocated, then, according to the consumer's order consumer-1, , consumer-2, consumer-3, consumer-4the two extra queuerespectively to consumer-1and consumer-2. Finally, share the following relationship:
consumer-1: 3个; consumer-2: 3个; consumer-3: 2个; consumer-4: 2个, as shown below:

Its code implementation is very simple:

public class AllocateMessageQueueAveragely implements AllocateMessageQueueStrategy {
    private final InternalLogger log = ClientLogger.getLog();

    @Override
    public List<MessageQueue> allocate(String consumerGroup, String currentCID, List<MessageQueue> mqAll,
        List<String> cidAll) {
        if (currentCID == null || currentCID.length() < 1) {
            throw new IllegalArgumentException("currentCID is empty");
        }
        if (mqAll == null || mqAll.isEmpty()) {
            throw new IllegalArgumentException("mqAll is null or mqAll empty");
        }
        if (cidAll == null || cidAll.isEmpty()) {
            throw new IllegalArgumentException("cidAll is null or cidAll empty");
        }

        List<MessageQueue> result = new ArrayList<MessageQueue>();
        if (!cidAll.contains(currentCID)) {
            log.info("[BUG] ConsumerGroup: {} The consumerId: {} not in cidAll: {}",
                consumerGroup,
                currentCID,
                cidAll);
            return result;
        }

        int index = cidAll.indexOf(currentCID);
        int mod = mqAll.size() % cidAll.size();
        int averageSize =
            mqAll.size() <= cidAll.size() ? 1 : (mod > 0 && index < mod ? mqAll.size() / cidAll.size()
                + 1 : mqAll.size() / cidAll.size());
        int startIndex = (mod > 0 && index < mod) ? index * averageSize : index * averageSize + mod;
        int range = Math.min(averageSize, mqAll.size() - startIndex);
        for (int i = 0; i < range; i++) {
            result.add(mqAll.get((startIndex + i) % mqAll.size()));
        }
        return result;
    }

    @Override
    public String getName() {
        return "AVG";
    }
}

3.2 AllocateMessageQueueAveragelyByCircle - Average algorithm based on ring

Annular averaging algorithm, according to a sequence refers to consumers, by sequentially queue队列assigned one by one consisting of a ring of FIG. Specific process is as follows:

The final result of this allocation algorithm is:
consumer-1: # 0, # 4, # 8
consumer-2: # 1, # 5, # 9
consumer-3: # 2, # 6
consumer-4: # 3, # 7
which code for the following:

/**
 * Cycle average Hashing queue algorithm
 */
public class AllocateMessageQueueAveragelyByCircle implements AllocateMessageQueueStrategy {
    private final InternalLogger log = ClientLogger.getLog();

    @Override
    public List<MessageQueue> allocate(String consumerGroup, String currentCID, List<MessageQueue> mqAll,
        List<String> cidAll) {
        if (currentCID == null || currentCID.length() < 1) {
            throw new IllegalArgumentException("currentCID is empty");
        }
        if (mqAll == null || mqAll.isEmpty()) {
            throw new IllegalArgumentException("mqAll is null or mqAll empty");
        }
        if (cidAll == null || cidAll.isEmpty()) {
            throw new IllegalArgumentException("cidAll is null or cidAll empty");
        }

        List<MessageQueue> result = new ArrayList<MessageQueue>();
        if (!cidAll.contains(currentCID)) {
            log.info("[BUG] ConsumerGroup: {} The consumerId: {} not in cidAll: {}",
                consumerGroup,
                currentCID,
                cidAll);
            return result;
        }

        int index = cidAll.indexOf(currentCID);
        for (int i = index; i < mqAll.size(); i++) {
            if (i % cidAll.size() == index) {
                result.add(mqAll.get(i));
            }
        }
        return result;
    }

    @Override
    public String getName() {
        return "AVG_BY_CIRCLE";
    }
}

3.3 AllocateMachineRoomNearby- Based on the principle of the room near the algorithm

The algorithm used 装饰者设计模式for allocation strategy has been enhanced. In a production environment generally, if the micro-service architecture, deployment RocketMQ clusters may be deployed in different rooms, the basic structure may be as shown below:

For cross-room scenario, the network will exist reasons, the stability and the isolated heart, the algorithm based on queuethe deployment of the room location and 消费者consumerposition, the current filter 消费者consumerthe same room queue队列, then the combination of these algorithms such as allocation algorithm based on the average queue队列then choose a subset of the base. Related code to achieve the following:

@Override
    public List<MessageQueue> allocate(String consumerGroup, String currentCID, List<MessageQueue> mqAll,
        List<String> cidAll) {
        //省略部分代码
        List<MessageQueue> result = new ArrayList<MessageQueue>();

        //将MQ按照 机房进行分组
        Map<String/*machine room */, List<MessageQueue>> mr2Mq = new TreeMap<String, List<MessageQueue>>();
        for (MessageQueue mq : mqAll) {
            String brokerMachineRoom = machineRoomResolver.brokerDeployIn(mq);
            if (StringUtils.isNoneEmpty(brokerMachineRoom)) {
                if (mr2Mq.get(brokerMachineRoom) == null) {
                    mr2Mq.put(brokerMachineRoom, new ArrayList<MessageQueue>());
                }
                mr2Mq.get(brokerMachineRoom).add(mq);
            } else {
                throw new IllegalArgumentException("Machine room is null for mq " + mq);
            }
        }

        //将消费者 按照机房进行分组
        Map<String/*machine room */, List<String/*clientId*/>> mr2c = new TreeMap<String, List<String>>();
        for (String cid : cidAll) {
            String consumerMachineRoom = machineRoomResolver.consumerDeployIn(cid);
            if (StringUtils.isNoneEmpty(consumerMachineRoom)) {
                if (mr2c.get(consumerMachineRoom) == null) {
                    mr2c.put(consumerMachineRoom, new ArrayList<String>());
                }
                mr2c.get(consumerMachineRoom).add(cid);
            } else {
                throw new IllegalArgumentException("Machine room is null for consumer id " + cid);
            }
        }

        List<MessageQueue> allocateResults = new ArrayList<MessageQueue>();

        //1.过滤出当前机房内的MQ队列子集，在此基础上使用分配算法挑选
        String currentMachineRoom = machineRoomResolver.consumerDeployIn(currentCID);
        List<MessageQueue> mqInThisMachineRoom = mr2Mq.remove(currentMachineRoom);
        List<String> consumerInThisMachineRoom = mr2c.get(currentMachineRoom);
        if (mqInThisMachineRoom != null && !mqInThisMachineRoom.isEmpty()) {
            allocateResults.addAll(allocateMessageQueueStrategy.allocate(consumerGroup, currentCID, mqInThisMachineRoom, consumerInThisMachineRoom));
        }

        //2.不在同一机房，按照一般策略进行操作
        for (String machineRoom : mr2Mq.keySet()) {
            if (!mr2c.containsKey(machineRoom)) { // no alive consumer in the corresponding machine room, so all consumers share these queues
                allocateResults.addAll(allocateMessageQueueStrategy.allocate(consumerGroup, currentCID, mr2Mq.get(machineRoom), cidAll));
            }
        }

        return allocateResults;
    }

3.4 AllocateMessageQueueByMachineRoom- Based on room allocation algorithm

This algorithm is suitable for internal messages belonging to the same room, to allocation queue. This approach is very clear, based on the above 机房临近分配算法scenario, this more thoroughly, directly specify a policy based on consumption of the engine room. In this way a strong property convention, such as brokerthe name stitching room by name, by convention partitioned parsing algorithm.
Which code is implemented as follows:

/**
 * Computer room Hashing queue algorithm, such as Alipay logic room
 */
public class AllocateMessageQueueByMachineRoom implements AllocateMessageQueueStrategy {
    private Set<String> consumeridcs;

    @Override
    public List<MessageQueue> allocate(String consumerGroup, String currentCID, List<MessageQueue> mqAll,
        List<String> cidAll) {
        List<MessageQueue> result = new ArrayList<MessageQueue>();
        int currentIndex = cidAll.indexOf(currentCID);
        if (currentIndex < 0) {
            return result;
        }
        List<MessageQueue> premqAll = new ArrayList<MessageQueue>();
        for (MessageQueue mq : mqAll) {
            String[] temp = mq.getBrokerName().split("@");
            if (temp.length == 2 && consumeridcs.contains(temp[0])) {
                premqAll.add(mq);
            }
        }

        int mod = premqAll.size() / cidAll.size();
        int rem = premqAll.size() % cidAll.size();
        int startIndex = mod * currentIndex;
        int endIndex = startIndex + mod;
        for (int i = startIndex; i < endIndex; i++) {
            result.add(mqAll.get(i));
        }
        if (rem > currentIndex) {
            result.add(premqAll.get(currentIndex + mod * cidAll.size()));
        }
        return result;
    }

    @Override
    public String getName() {
        return "MACHINE_ROOM";
    }

    public Set<String> getConsumeridcs() {
        return consumeridcs;
    }

    public void setConsumeridcs(Set<String> consumeridcs) {
        this.consumeridcs = consumeridcs;
    }

3.5 AllocateMessageQueueConsistentHashBased on the consistency of hash algorithm

Using this algorithm, will be consumer消费者configured as a ring hash Node node, then queue队列decide which is assigned to the hash through the ring consumer消费者.
The basic modes are:

What is the consistency of hash algorithm?
Consistency hash algorithm is used in a distributed system, ensure data consistency and proposed an algorithm hash ring-based implementation, limited to the length of the article, I am not here to expand described, interested students can Bowen refer to the others: the principle of consistent hashing algorithm

The algorithm is not complicated, as shown below:

public List<MessageQueue> allocate(String consumerGroup, String currentCID, List<MessageQueue> mqAll,
        List<String> cidAll) {
        //省略部分代码
        List<MessageQueue> result = new ArrayList<MessageQueue>();
        if (!cidAll.contains(currentCID)) {
            log.info("[BUG] ConsumerGroup: {} The consumerId: {} not in cidAll: {}",
                consumerGroup,
                currentCID,
                cidAll);
            return result;
        }

        Collection<ClientNode> cidNodes = new ArrayList<ClientNode>();
        for (String cid : cidAll) {
            cidNodes.add(new ClientNode(cid));
        }
//使用consumer id 构造hash环
        final ConsistentHashRouter<ClientNode> router; //for building hash ring
        if (customHashFunction != null) {
            router = new ConsistentHashRouter<ClientNode>(cidNodes, virtualNodeCnt, customHashFunction);
        } else {
            router = new ConsistentHashRouter<ClientNode>(cidNodes, virtualNodeCnt);
        }
        //依次为 队列分配 consumer
        List<MessageQueue> results = new ArrayList<MessageQueue>();
        for (MessageQueue mq : mqAll) {
            ClientNode clientNode = router.routeNode(mq.toString());
            if (clientNode != null && currentCID.equals(clientNode.getKey())) {
                results.add(mq);
            }
        }

        return results;

    }

3.6 AllocateMessageQueueByConfig- Configure allocation algorithm based on

This algorithm is based on a simple configuration, very simple, actual use may not use. code show as below:

public class AllocateMessageQueueByConfig implements AllocateMessageQueueStrategy {
    private List<MessageQueue> messageQueueList;

    @Override
    public List<MessageQueue> allocate(String consumerGroup, String currentCID, List<MessageQueue> mqAll,
        List<String> cidAll) {
        return this.messageQueueList;
    }

    @Override
    public String getName() {
        return "CONFIG";
    }

    public List<MessageQueue> getMessageQueueList() {
        return messageQueueList;
    }

    public void setMessageQueueList(List<MessageQueue> messageQueueList) {
        this.messageQueueList = messageQueueList;
    }
}

3.7 How Consumers specify the allocation algorithm?

By default, consumers are using the AllocateMessageQueueAveragelyalgorithm, you can specify your own:

public class DefaultMQPushConsumer{    
    /**
     * Default constructor.
     */
    public DefaultMQPushConsumer() {
        this(MixAll.DEFAULT_CONSUMER_GROUP, null, new AllocateMessageQueueAveragely());
    }

    /**
     * Constructor specifying consumer group, RPC hook and message queue allocating algorithm.
     *
     * @param consumerGroup Consume queue.
     * @param rpcHook RPC hook to execute before each remoting command.
     * @param allocateMessageQueueStrategy message queue allocating algorithm.
     */
    public DefaultMQPushConsumer(final String consumerGroup, RPCHook rpcHook,
        AllocateMessageQueueStrategy allocateMessageQueueStrategy) {
        this.consumerGroup = consumerGroup;
        this.allocateMessageQueueStrategy = allocateMessageQueueStrategy;
        defaultMQPushConsumerImpl = new DefaultMQPushConsumerImpl(this, rpcHook);
    }
}

4 Conclusion

The above is from the design brief of RocketMQ delivery mechanism, if you want more detailed design principles, the public may be concerned about my account below, will be updated simultaneously, thanks to support ~
author is limited, welcome message corrections Tucao ~

Space Dandy _ _

Published 13 original articles · won praise 78 · Views 450,000 +

His message board concerns