What you need to know about RocketMQ

1 Overview

I wrote an article about Kafka a long time ago. What you need to know about Kafka . At that time, Kafka was used more in business. Now, after changing companies, Rocketmq is used more. This article I will try my best to comprehensively introduce the comparison of various key points between RocketMQ and Kafka. I hope you can gain something after reading it.

RocketMQ was formerly called MetaQ, and was renamed RocketMQ when MeataQ released version 3.0. Its essential design idea is similar to Kafka, but different from Kafka, it uses Java for development. Since the domestic Java audience is far more than Scala, Therefore, RocketMQ is the first choice for many Java language-based companies. The same RocketMQ and Kafka are both top-level projects in the Apache Foundation. Their communities are very active, and the project update iteration is very fast.

2. Introductory example

2.1 Producers

public class Producer {
    public static void main(String[] args) throws MQClientException, InterruptedException {

        DefaultMQProducer producer = new DefaultMQProducer("ProducerGroupName");
        producer.start();

        for (int i = 0; i < 128; i++)
            try {
                {
                    Message msg = new Message("TopicTest",
                        "TagA",
                        "OrderID188",
                        "Hello world".getBytes(RemotingHelper.DEFAULT_CHARSET));
                    SendResult sendResult = producer.send(msg);
                    System.out.printf("%s%n", sendResult);
                }

            } catch (Exception e) {
                e.printStackTrace();
            }

        producer.shutdown();
    }
}

Define a producer directly, create a Message, and call the send method.

2.2 Consumers

public class PushConsumer {

    public static void main(String[] args) throws InterruptedException, MQClientException {
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("CID_JODIE_1");
        consumer.subscribe("TopicTest", "*");
        consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET);
        //wrong time format 2017_0422_221800
        consumer.setConsumeTimestamp("20181109221800");
        consumer.registerMessageListener(new MessageListenerConcurrently() {

            @Override
            public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
                System.out.printf("%s Receive New Messages: %s %n", Thread.currentThread().getName(), msgs);
                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
            }
        });
        consumer.start();
        System.out.printf("Consumer Started.%n");
    }
}

3.RocketMQ architecture principle

For RocketMQ, throw a few questions first:

• What are RocketMQ's topics and queues, and how are they different from Kafka's partitions?
• What is the RocketMQ network model and how does it compare to Kafka?
• What is the RocketMQ message storage model, how does it ensure highly reliable storage, and how does it compare with Kafka?

3.1 RocketMQ Architecture Diagram

The architecture diagram of RocketMQ is not much different from Kafka in general, but there are many differences in many details, which will be described one by one next.

3.2 Explanation of RocketMQ terms

In the architecture of 3.1, we have multiple Producers, multiple Master Brokers, and multiple Slave Brokers. Each Producer can correspond to multiple Topics, and each Consumer can also consume multiple Topics.

Broker information will be reported to NameServer, and Consumer will pull Broker and Topic information from NameServer.

• Producer: The message producer, the client that sends messages to the Broker
• Consumer: message consumer, client that reads messages from Broker
• Broker: The processing node in the middle of the message. Unlike kafka, the broker of kafka does not have the concept of master and slave, and can write requests and backup data of other nodes. RocketMQ can only be written by the master broker node, and generally read through the master node. When the master node There is a fault or some other special deception to use the slave node to read, which is somewhat similar to the master-slave architecture of mysql.
• Topic: message topic, first-level message type, producers send messages to it, and consumers read its messages.
• Group: Divided into ProducerGroup and ConsumerGroup, which represent a certain type of producers and consumers. Generally speaking, the same service can be used as a Group, and the same Group generally sends and consumes the same messages.
• Tag: There is no such concept in Kafka. Tag is a secondary message type. Generally speaking, the same Tag can be used for related business, such as order message queue. Using Topic_Order, Tag can be divided into Tag_food order, Tag_clothing order etc.
• Queue: It is called Partition in kafka. Each Queue is ordered internally. RocketMQ is divided into two queues: read and write. Generally speaking, the number of read and write queues is the same. If they are inconsistent, there will be many problems.
• NameServer: ZooKeeper is used in Kafka to save the address information of the Broker and the election of the Broker's Leader. RocketMQ does not adopt the strategy of electing the Broker, so a stateless NameServer is used for storage. Since the NameServer is stateless, the cluster There is no communication between nodes, so when uploading data, it needs to be sent to all nodes.

Many friends are asking what is statelessness? The presence or absence of state is actually whether the data will be stored. If there is state, the data will be persisted. A stateless service can be understood as a memory service. NameServer itself is also a memory service. All data is stored in memory. After restarting will be lost.

3.3 Topic and Queue

Each message in RocketMQ has a Topic to distinguish different messages. A topic generally has multiple message subscribers. When a producer publishes a message to a topic, all consumers who subscribe to the topic can receive new messages written by the producer.

There are multiple Queues in Topic. This is actually the smallest unit for us to send/read message channels. We need to specify a certain Queue to write to when sending a message, and also need to specify a certain Queue when pulling a message. Queue, so our sequential messages can keep the queue ordered based on our Queue dimension. If you want to be globally ordered, you need to set the Queue size to 1, so that all data will be ordered in the Queue.

In the figure above, our Producer will select Queue through some strategies:

• Non-sequential messages: Non-sequential messages are generally sent directly by polling.
• Sequential message: Hash is performed according to a key such as our common order ID, user ID, and the same type of data is placed in the same queue to ensure our order.

Our same group of consumers will also choose Queue according to some strategies, such as average allocation or consistent Hash allocation.

It should be noted that when the Consumer goes offline or online, it needs to be rebalanced, that is, Rebalance. The rebalance mechanism of RocketMQ is as follows:

• Regularly pull the latest information on broker and topic
• Rebalance every 20s
• Randomly select a main Broker of the current topic. It should be noted here that all main brokers will be selected every time when rebalancing, because there will be one Broker and multiple Brokers.
• Get the current Broker and all machine IDs of the current ConsumerGroup.
• Then do policy assignment.

Since the rebalancing is done regularly, there may be a Queue consumed by two Consumers at the same time, so there will be repeated delivery of messages.

The rebalancing mechanism of Kafka is different from that of RocketMQ. The rebalancing of Kafka is done through the contact between the Consumer and the Coordinator. When the Coordinator perceives the change of the consumer group, it will send a rebalancing signal during the heartbeat process, and then a ConsumerLeader will perform the rebalancing. Select, and then the Coordinator will notify all consumers of the result.

3.3.1 The number of Queue reads and writes is inconsistent

In RocketMQ, Queue is divided into two types: read and write. When I first contacted RocketMQ, I always thought that there would be no problem with the inconsistent configuration of read and write queues. For example, when there are many consumer machines, we configure many read queues. However, in the actual process, it is found that the message cannot be consumed and there is no message consumption at all.

• When the number of write queues is greater than the number of read queues, the data of write queues with IDs larger than this part of the read queue cannot be consumed because it will not be allocated to consumers.
• When the number of read queues is greater than the number of write queues, no messages will be delivered for that many queues.

This function is obviously useless to me in RocketMQ, because basically it will be set to the same size of read and write queues.

This question has not received a good answer in RocketMQ's Issue.

3.4 Consumption Model

Generally speaking, the consumption models of message queues are divided into two types, the push-based message (push) model and the pull-based (poll) message model.

In a message system based on the push model, the consumption status is recorded by the message broker. After the message broker pushes the message to the consumer, it marks the message as having been consumed, but this method cannot well guarantee the processing semantics of consumption. For example, after we have sent the message to the consumer, because the consuming process hangs up or the message is not received due to network reasons, if we mark it as consumed in the consuming agent, the message will be permanently lost. If we use the method of replying after the producer receives the message, the message broker needs to record the consumption status, which is not desirable.

Students who have used RocketMQ can't help but think, aren't there two types of consumers in RocketMQ? MQPullConsumerAnd MQPushConsumer, MQPushConsumerisn't that our push model? In fact, these two models are that the client actively pulls messages, and the implementation differences are as follows:

• MQPullConsumer: Each time you pull a message, you need to pass in the offset of the pull message and how many messages are pulled each time. The specific message is pulled and how much is pulled is controlled by the client.
• MQPushConsumer: The client also actively pulls messages, but the message progress is saved by the server. The Consumer will regularly report where it consumes, so the Consumer can find the point of the last consumption when it consumes next time. Generally speaking, PushConsumer is used. We don't need to care about offset and how much data to pull, just use it directly.

3.4.1 Cluster consumption and broadcast consumption

There are two consumption modes: cluster consumption and broadcast consumption:

• Cluster consumption: The same GroupId belongs to a cluster. Generally speaking, a message will only be processed by any consumer.
• Broadcast consumption: The message of broadcast consumption will be messaged by all consumers in the cluster, but it should be noted that because the offset of broadcast consumption is too expensive to save on the server side, every time the client restarts, it will consume from the latest message, not the last time. Saved offset.

3.5 Network Model

The native socket used in Kafka implements network communication, while RocketMQ uses the Netty network framework. Now more and more middleware will not directly choose the native socket, but use the Netty framework, mainly due to the following Several reasons:

• The API is simple to use, does not need to care about too many network details, and focuses more on middleware logic.
• High performance.
• Mature and stable, jdk nio bugs have been fixed.

The choice of the framework is one aspect, and to ensure the efficiency of network communication, the network thread model is also one aspect. We commonly have 1+N (1 Acceptor thread, N IO threads), 1+N+M (1 Acceptor thread) Thread, N IO threads, M worker threads) and other models, RocketMQ uses the 1+N1+N2+M model, as shown in the following figure:

1 acceptor thread, N1 IO threads, N2 threads are used for Shake-hand, SSL verification, encoding and decoding; M threads are used for business processing. This advantage puts some potentially time-consuming operations such as encoding, decoding, and SSL verification in a separate thread pool, which will not occupy our business threads and IO threads.

3.6 Highly reliable distributed storage model

As a good message system, high-performance storage and high availability are essential.

3.6.1 High-performance log storage

The storage core designs of RocketMQ and Kafka are very different, so they are also very different in terms of write performance. This is the performance test done by the Ali middleware team on RocketMQ and Kafka under different topics in 2016:

It can be seen from the figure that:

• When the number of topics in Kafka increased from 64 to 256, the throughput dropped by 98.37%.
• When the number of topics in RocketMQ increased from 64 to 256, the throughput only dropped by 16%. Why is this? All messages under a topic in Kafka are distributed and stored on multiple nodes in the form of partitions. At the same time, on the kafka machine, each Partition will actually correspond to a log directory, and there will be multiple log segments under the directory. Therefore, if there are many topics, although Kafka writes files sequentially, in fact, there are too many files, which will cause very intense disk IO competition.

So why is RocketMQ still able to maintain more throughput in the case of multiple topics? Let's first look at the more critical files in RocketMQ:

There are four directories here (the explanation here is directly from the official RocketMQ):

• commitLog: The storage body of the message body and metadata, which stores the content of the message body written by the Producer, and the message content is not of fixed length. The default size of a single file is 1G, the length of the file name is 20 digits, the left is filled with zeros, and the rest is the starting offset. For example, 00000000000000000000 represents the first file, the starting offset is 0, and the file size is 1G=1073741824; when The first file is full, the second file is 00000000001073741824, the starting offset is 1073741824, and so on. The message is mainly written to the log file sequentially. When the file is full, it is written to the next file;
• config: Save some configuration information, including some Group, Topic, and Consumer consumption offset information.
• consumeQueue: message consumption queue. The main purpose of introduction is to improve the performance of message consumption. Since RocketMQ is a topic-based subscription model, message consumption is carried out for topics. It is very inefficient to traverse the commitlog file to retrieve messages based on topics. . The Consumer can find the message to be consumed according to the ConsumeQueue. Among them, ConsumeQueue (logical consumption queue) is used as the index of the consumption message, which saves the starting physical offset offset of the queue message under the specified topic in the CommitLog, the message size and the HashCode value of the message Tag. The consumequeue file can be regarded as a topic-based commitlog index file, so the consumequeue folder is organized as follows: topic/queue/file three-layer organizational structure, the specific storage path is: $HOME/store/consumequeue/{topic}/{queueId }/{fileName}. Similarly, the consumequeue file adopts a fixed-length design. Each entry has a total of 20 bytes, which are the 8-byte commitlog physical offset, the 4-byte message length, and the 8-byte tag hashcode. A single file consists of 30W entries. Each entry can be accessed randomly like an array, and the file size of each ConsumeQueue is about 5.72M;
• indexFile: IndexFile (index file) provides a way to query messages by key or time interval. The storage location of the Index file is: $HOME \store\index${fileName}. The file name fileName is named after the timestamp when it was created. The fixed size of a single IndexFile file is about 400M. One IndexFile can save 2000W indexes. IndexFile The underlying storage of rocketmq is designed to implement the HashMap structure in the file system, so the underlying implementation of the rocketmq index file is a hash index.

We found that our message body data is not written to multiple files like Kafka, but to one file, so that our write IO competition is very small, and we can still maintain high throughput when there are many topics. Some students said that the ConsumeQueue writing here is constantly writing, and the ConsumeQueue creates files in the Queue dimension, so the number of files is still large. Here, the amount of data written to the ConsumeQueue is very small, and each message is only 20. Bytes, 30W pieces of data are only about 6M, so in fact, the impact on us is much smaller than that between Kafka topics. Our entire logic can be as follows:

The Producer continuously adds new messages to the CommitLog. There is a scheduled task ReputService that will continuously scan the newly added CommitLog, and then continue to build the ConsumerQueue and Index.

Note: This refers to ordinary hard disks, and the concurrent writing of multiple files and single file writing on the SSD has little effect.

read message

Each Partition in Kafka will be a separate file, so when a message is consumed, sequential reading will occur very well. We know that when the OS reads the file from the physical disk, it will sequentially read other adjacent blocks. The data file is pre-read and the data is put into the PageCache, so Kafka's read message performance is better.

The RocketMQ reading process is as follows:

• First read the offset in ConsumerQueue corresponding to the physical offset of CommitLog
• Read CommitLog according to offset

ConsumerQueue is also a separate file for each Queue, and its file size is small, so it is easy to use PageCache to improve performance. And CommitLog, because the continuous messages of the same Queue are actually discontinuous in CommitLog, it will cause random reads. RocketMQ has made several optimizations for this:

• Mmap mapping read, Mmap method reduces the performance overhead of copying disk file data back and forth between the buffer of the operating system kernel address space and the buffer of the user application address space by traditional IO
• Use DeadLine scheduling algorithm + SSD storage disk
• Since Mmap mapping is limited by memory, when Mmmap is not mapping this part of the data (that is, too many messages are accumulated), the default is 40% of the memory, and the request will be sent to SLAVE to reduce the pressure on the Master

3.6.2 Availability

3.6.2.1 Cluster Mode

We first need to choose a cluster mode to adapt to the availability we can tolerate. Generally speaking, there are three types:

• Single Master: In this mode, the availability is the lowest, but the cost is also the lowest. Once it goes down, everything is unavailable. This generally applies only to local testing.
• Single Master Multiple SLAVE: In this mode, the availability is general. If the master is down, all writes are unavailable and reads are still available. If the master disk is damaged, you can rely on slave data.
• Multi-Master: In this mode, the availability is general. If some masters are down, the messages on this part of the master cannot be consumed, nor can data be written. If a topic has queues on multiple masters, it can be guaranteed that there is no The part that is down can be consumed and written normally. If the master's disk is damaged, messages will be lost.
• Multiple Masters and Multiple Slaves: This mode has the highest availability, but also the highest maintenance cost. When the master goes down, only the queues that appear on this part of the master cannot be written, but reading is still possible, and if the master The disk is damaged and can rely on slave data.

Generally speaking, if you put it into a production environment, you will choose the fourth one to ensure the highest availability.

3.6.2.2 Availability of messages

When we choose the cluster mode, then what we need to care about is how to store and replicate this data. RocketMQ provides synchronous and asynchronous strategies for message flushing to satisfy our needs. After we choose synchronous flushing, if FLUSH_DISK_TIMEOUT will be returned if the flushing times out. If it is an asynchronous flushing, it will not return the flushing-related information. Selecting the synchronous flushing can maximize our message without losing it.

In addition to the choice of storage, our master-slave synchronization provides two modes of synchronization and asynchronous for replication. Of course, choosing synchronization can improve availability, but the RT time for sending messages will drop by about 10%.

3.6.3 Dleger

We have done a lot of analysis on the master-slave deployment mode above. We found that when there is a problem with the master, our writing will not be available unless the master is restored, or our slave is manually switched to the master, which leads us to The Slave only has the function of reading in most cases. RocketMQ has launched Dleger-RocketMQ in recent versions, which uses the Raft protocol to replicate CommitLog, and automatically selects the master, so that when the master goes down, writes remain available.

More information about Dleger-RocketMQ can be found in this article: Dledger-RocketMQ commitlog repository based on Raft protocol .

3.7 Timing/Delayed Messages

Timing messages and delayed messages are often used in actual business scenarios, such as the following scenarios:

• If the order is overtime and not paid, it will be automatically closed, because in many scenarios, the inventory will be locked after the order is placed, and it needs to be closed overtime.
• Some delayed operations are required, such as some bottom-line logic. After a certain logic is completed, a delay message can be sent, such as a delay of half an hour, to perform bottom-up check and compensation.
• To send a message to the user at a certain time, delayed messages can also be used.

In the open source version of RocketMQ, delaying messages does not support any time delay. Several fixed delay levels need to be set. Currently, the default settings are: 1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h, 1s to 2h correspond to levels 1 to 18, while Alibaba Cloud The version (payable) is available to support any moment (millisecond level) within 40 days. Let's first look at the schematic diagram of timing tasks in RocketMQ:

• Step1: The Producer sets the required delay level on the messages it sends.
• Step2: The broker finds that the message is a delayed message, and replaces the topic with a delayed topic. Each delay level will be used as a separate queue, and its own topic will be stored as additional information.
• Step3: Build ConsumerQueue
• Step4: The scheduled task regularly scans the ConsumerQueue of each delay level.
• Step5: Get the Offset of CommitLog in ConsumerQueue, get the message, and judge whether the execution time has been reached
• Step6: If it is reached, restore the topic of the message and re-delivery it. If it is not reached, delay the execution of the task for the period of time that has not been reached.

It can be seen that delayed messages are implemented by creating separate topics and queues. If we want to achieve any time within 40 days, based on this scheme, we need 40 24 60 60 1000 queues, which is very expensive. How high is Alibaba Cloud’s support at any time? The guess here is to persist the second-level TimeWheel time wheel. The second-level time wheel is used to replace our ConsumeQueue, save the Commitlog-Offset, and then continuously take out the current time through the time wheel, and then deliver the message again. The specific implementation logic needs to be written in a separate article in the future.

3.8 Transaction Messages

Transaction messages are also a major feature of RocketMQ, which can help us complete the final consistency of distributed transactions. For more information on distributed transactions, you can refer to many of my previous articles for detailed introductions. Here, pay attention directly to the public account : Café latte.

The specific steps for using transaction messages are as follows:

• Step1: Call sendMessageInTransaction to send transaction messages
• Step2: If the sending is successful, execute the local transaction.
• Step3: Send a commit if the execution of the local transaction is successful, and send a rollback if it fails.
• Step4: If one of the stages such as commit fails to send, rocketMQ will periodically check back from the Broker to check the status of the local transaction.

The entire process of using transaction messages is more complicated than the previous types of messages. The following is a schematic diagram of the implementation of transaction messages:

• Step1: Send a transaction message, also called halfMessage here, which will replace the Topic with the Topic of HalfMessage.
• Step2: Send commit or rollback. If it is commit, the previous message will be queried, then the message will be restored to the original topic, and an OpMessage will be sent to record that the current message can be deleted. If it is rollback, an OpMessage will be sent directly to delete it.
• Step3: There is a timing task for processing transaction messages in the Broker. Compare halfMessage and OpMessage regularly. If there is OpMessage and the status is deleted, then the message must be committed or rollback, so the message can be deleted.
• Step4: If the transaction times out (the default is 6s) and there is no opMessage, then it is very likely that the commit information is lost, and we will check the local transaction status of our Producer here.
• Step5: Do Step2 according to the queried information.

We found that RocketMQ implements transaction messages by modifying the original topic information, just like delayed messages, and then simulates it as a consumer for consumption and does some special business logic. Of course, we can also use this method to do more expansion of RocketMQ.

4. Summary

Here let us return to a few questions mentioned in the article:

• What are RocketMQ's topics and queues, and how are they different from Kafka's partitions?
• What is the RocketMQ network model and how does it compare to Kafka?
• What is the RocketMQ message storage model, how does it ensure highly reliable storage, and how does it compare with Kafka?

After reading this article, you must already have the answer in your mind. This article mainly talks about the comprehensive design architecture of RocketMQ. If you haven't seen enough, then please pay attention to my public account.

If you think this article is helpful to you, your attention and forwarding are the greatest support for me, O(∩_∩)O:

{{o.name}}

{{m.name}}