RocketMQ you should know

1 Overview

I wrote a Kafka-related article a long time ago. What you need to know about Kafka. At that time, Kafka was more used in business, but now after changing the company, Rocketmq is more used. This article I will try my best to comprehensively introduce the comparison of RocketMQ and Kafka's key points. I hope everyone can gain something after reading.

The predecessor of RocketMQ was called MetaQ, and it was renamed RocketMQ when the 3.0 version of MetaQ was released. Its essential design idea is similar to Kafka, but unlike Kafka, it uses Java for development, because the domestic Java audience is far more than Scala. , So RocketMQ is the first choice of many companies based on Java language. The same RocketMQ and Kafka are both top projects in the Apache Foundation. Their communities are very active and the project update iterations are also very fast.

2. Getting started

2.1 Producer

public class Producer {
    public static void main(String[] args) throws MQClientException, InterruptedException {

        DefaultMQProducer producer = new DefaultMQProducer("ProducerGroupName");
        producer.start();

        for (int i = 0; i < 128; i++)
            try {
                {
                    Message msg = new Message("TopicTest",
                        "TagA",
                        "OrderID188",
                        "Hello world".getBytes(RemotingHelper.DEFAULT_CHARSET));
                    SendResult sendResult = producer.send(msg);
                    System.out.printf("%s%n", sendResult);
                }

            } catch (Exception e) {
                e.printStackTrace();
            }

        producer.shutdown();
    }
}

Define a producer directly, create a Message, and call the send method.

2.2 Consumer

public class PushConsumer {

    public static void main(String[] args) throws InterruptedException, MQClientException {
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("CID_JODIE_1");
        consumer.subscribe("TopicTest", "*");
        consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET);
        //wrong time format 2017_0422_221800
        consumer.setConsumeTimestamp("20181109221800");
        consumer.registerMessageListener(new MessageListenerConcurrently() {

            @Override
            public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
                System.out.printf("%s Receive New Messages: %s %n", Thread.currentThread().getName(), msgs);
                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
            }
        });
        consumer.start();
        System.out.printf("Consumer Started.%n");
    }
}

3. RocketMQ architecture principle

For RocketMQ first throw a few questions:

• What are the topics and queues of RocketMQ, and how are they different from Kafka partitions?
• What is the RocketMQ network model and how does it compare with Kafka?
• What is the RocketMQ message storage model, how to ensure highly reliable storage, and how does it compare with Kafka?

3.1 RocketMQ architecture diagram

For the architecture diagram of RocketMQ, there is not much difference from Kafka in general, but there are many differences in many details, which will be described one by one next.

3.2 RocketMQ noun explanation

In the 3.1 architecture, we have multiple Producers, multiple master Brokers, and multiple slave Brokers. Each Producer can correspond to multiple Topics, and each Consumer can also consume multiple Topics.

Broker information will be reported to NameServer, and Consumer will pull Broker and Topic information from NameServer.

• Producer: The message producer, the client that sends messages to the Broker
• Consumer: Message consumer, the client that reads messages from Broker
• Broker: The processing node in the middle of the message. This is different from Kafka. Kafka's Broker does not have the concept of master and slave. It can write requests and back up other node data. RocketMQ can only be written by the master Broker node. It is generally read through the master node. Read from the node will only be used if there is a failure or some other special conditions, which is a bit similar-the master-slave architecture of mysql.
• Topic: The message subject, the first-level message type, the producer sends a message to it, and the consumer reads its message.
• Group: Divided into ProducerGroup and ConsumerGroup, which represent a certain type of producers and consumers. Generally speaking, the same service can be used as a Group. Generally speaking, the same group sends and consumes the same messages.
• Tag: There is no such concept in Kafka. Tag is a secondary message type. Generally speaking, the same tag can be used for business related, such as order message queue, using Topic_Order, Tag can be divided into Tag_food order, Tag_clothing order and many more.
• Queue: It is called Partition in Kafka. Each Queue is internally ordered. In RocketMQ, it is divided into two types of queues, read and write. Generally speaking, the number of read and write queues is the same. If they are inconsistent, many problems will occur.
• NameServer: In Kafka, ZooKeeper is used to save the address information of the Broker, and the election of the Broker's Leader. In RocketMQ, the broker election strategy is not adopted, so a stateless NameServer is used for storage. Since the NameServer is stateless, the cluster There is no communication between nodes, so when uploading data, it needs to be sent to all nodes.

Many friends are asking what is stateless? The presence or absence of state is actually whether the data will be stored. If there is state, the data will be persisted. A stateless service can be understood as a memory service. NameServer itself is also a memory service. All data is stored in memory. After restarting Will be lost.

3.3 Topic and Queue

Every message in RocketMQ has a topic to distinguish different messages. A topic generally has multiple message subscribers. When the producer publishes a message to a topic, all consumers who subscribe to this topic can receive the new message written by the producer.

There are multiple Queues in Topic. This is actually the smallest unit of our sending/reading message channel. We need to specify a certain Queue when sending messages. When pulling messages, we also need to specify a certain pull Queue, so our sequential messages can keep the queue in order based on our Queue dimension. If you want to achieve global order, you need to set the Queue size to 1, so that all data will be ordered in the Queue.

In the above figure, our Producer will select Queue through some strategies:

• Non-sequential messages: Non-sequential messages are generally sent directly by means of round-robin sending.
• Sequence message: Hash according to a certain Key, such as our common order Id and user Id, and put the same type of data in the same queue to ensure our sequence.

Our same group of Consumers will also choose Queue according to some strategies, such as equal distribution or consistent Hash distribution.

It should be noted that when the Consumer goes offline or goes online, rebalancing is required here, that is, Rebalance. The rebalancing mechanism of RocketMQ is as follows:

• Pull the latest information of broker and topic regularly
• Rebalance every 20s
• Randomly select a main Broker of the current Topic. It is important to note whether all main Brokers will be selected every time you rebalance, because there will be one Broker and multiple Brokers.
• Get the current Broker, all machine IDs of the current ConsumerGroup.
• Then make policy assignments.

Since rebalancing is done regularly, there may be a Queue being consumed by two Consumers at the same time, so there will be repeated message delivery.

Kafka's rebalancing mechanism is different from RocketMQ. Kafka's rebalancing is accomplished through the connection between Consumer and Coordinator. When the Coordinator senses the change of the consumer group, it will send a rebalance signal during the heartbeat, and then a ConsumerLeader will perform rebalancing. Choose, and then the Coordinator will notify all consumers of the result.

3.3.1 The number of Queue reads and writes is inconsistent

Queue in RocketMQ is divided into two types: read and write. When I first came into contact with RocketMQ, I always thought that there would be no problems with inconsistent configuration of the number of read and write queues. For example, when there are many consumer machines, we configure many read queues. However, in the actual process, it was discovered that the message could not be consumed and there was no message consumption at all.

• When the number of write queues is greater than the number of read queues, the data of the write queue with IDs greater than that of the read queue cannot be consumed because it will not be allocated to consumers.
• When the number of read queues is greater than the number of write queues, no messages will be delivered to that many queues.

This feature is obviously useless in RocketMQ, because basically it will be set to the same read and write queue size. Why not unify it directly, but it is easy to make mistakes in user configuration.

This question has not received a good answer in RocketMQ's Issue.

3.4 Consumption model

Generally speaking, there are two consumption models of message queues, a push model based on push and a message model based on poll.

Based on the message system of the push model, the message agent records the consumption status. After the message broker pushes the message to the consumer, it marks the message as being consumed, but this method cannot guarantee the processing semantics of consumption. For example, when we send a message to the consumer, the consumption process hangs up or the message is not received due to network reasons. If we mark it as consumed at the consumption agent, the message will be permanently lost. If we use this method of replying after the producer receives the message, the message agent needs to record the consumption status, which is not desirable.

Students who have used RocketMQ can't help but think that there are two types of consumers in RocketMQ?
MQPullConsumer and MQPushConsumer, among which MQPushConsumer is our push model? In fact, in these two models, the client takes the initiative to pull messages, and the implementation differences are as follows:

• MQPullConsumer: Each time you pull a message, you need to pass in the offset of the pull message and how much message you pull each time, where to pull the message, and how much you pull is controlled by the client.
• MQPushConsumer: The client is also actively pulling messages, but the message progress is saved by the server, and the Consumer will regularly report where it has consumed, so the Consumer can find the last consumption point when it consumes the next time. Generally speaking, PushConsumer is used We don't need to care about the offset and how much data is pulled, just use it directly.

3.4.1 Cluster consumption and broadcast consumption

We have two consumption modes, cluster consumption and broadcast consumption:

• Cluster consumption: The same GroupId belongs to a cluster. Generally speaking, a message can only be processed by any consumer.
• Broadcast consumption: The message of broadcast consumption will be messaged by all consumers in the cluster, but it should be noted that because the offset of broadcast consumption is too expensive to save on the server side, each time the client restarts, it will consume from the latest news instead of the last time. The saved offset.

3.5 Network model

The native socket used in Kafka realizes network communication, while RocketMQ uses the Netty network framework. Now more and more middleware will not directly choose the native socket, but use the Netty framework, mainly due to the following Several reasons:

• API is simple to use, no need to care about too many network details, and more focus on middleware logic.
• High performance.
• Mature and stable, the bugs of jdk nio have been fixed.

Choosing a framework is one aspect, and if you want to ensure the efficiency of network communication, the network thread model is also one aspect. Our common ones are 1+N (1 Acceptor thread, N IO threads), 1+N+M (1 acceptor) Thread, N IO threads, M worker threads) and other models, RocketMQ uses a 1+N1+N2+M model, as shown in the following figure:

1 acceptor thread, N1 IO thread, N2 threads are used for Shake-hand, SSL verification, encoding and decoding; M threads are used for business processing. This advantage puts some potentially time-consuming operations such as encoding, decoding, and SSL verification in a separate thread pool, and will not occupy our business threads and IO threads.

3.6 Highly reliable distributed storage model

As a good message system, high-performance storage and high availability are indispensable.

3.6.1 High-performance log storage

The storage core design of RocketMQ and Kafka is very different, so there is also a big difference in write performance. This is the performance test performed by the Ali middleware team on RocketMQ and Kafka under different topics in 16 years:

As can be seen from the picture:

• When Kafka increased the number of topics from 64 to 256, the throughput dropped by 98.37%.
• RocketMQ's throughput dropped by only 16% when the number of topics increased from 64 to 256.
  Why is this? All messages under a topic in Kafka are distributed and stored on multiple nodes in a partition manner. At the same time, on the Kafka machine, each Partition actually corresponds to a log directory, and there are multiple log segments under the directory. So if there are many topics, Kafka writes files sequentially, but in fact there are too many files, which will cause very fierce disk IO competition.

So why does RocketMQ still maintain more throughput even with multiple topics? Let's first look at the key files in RocketMQ:

There are four directories here (the explanation here is the official RocketMQ directly):

• commitLog: The storage body of the message body and metadata, which stores the content of the message body written by the Producer. The message content is not fixed-length. The default size of a single file is 1G, the length of the file name is 20 digits, the left is filled with zeros, and the remaining is the starting offset. For example, 00000000000000000000 represents the first file, the starting offset is 0, and the file size is 1G=1073741824; The first file is full, the second file is 00000000001073741824, the starting offset is 1073741824, and so on. Messages are mainly written to the log file sequentially. When the file is full, it is written to the next file;
• config: Save some configuration information, including some Group, Topic, and Consumer consumption offsets.
• consumeQueue: Message consumption queue. The main purpose of introducing it is to improve the performance of message consumption. Because RocketMQ is a topic-based subscription model, message consumption is for topics. It is very inefficient to traverse the commitlog file and retrieve messages based on topic. . Consumer can find the message to be consumed according to ConsumeQueue. Among them, ConsumeQueue (logical consumption queue) as the index of consumption messages, saves the starting physical offset offset of the queue message under the specified topic in the CommitLog, the message size size and the HashCode value of the message tag. The consumequeue file can be regarded as a topic-based commitlog index file, so the consumequeue folder is organized as follows: topic/queue/file three-tier organization structure, the specific storage path is:
  
  HOME \store\index\${fileName}, file name fileName is named after the timestamp when it was created. The fixed single IndexFile file size is about 400M, and one IndexFile can store 2000W indexes. The underlying storage of IndexFile is designed to implement the HashMap structure in the file system, so the rocketmq index file has its bottom layer. Implemented as a hash index.

We found that our message body data is not written to multiple files like Kafka, but to one file, so that our write IO competition is very small, and we can still maintain a high throughput when there are many topics. Some students said that the ConsumeQueue writing here is constantly writing, and the ConsumeQueue uses the Queue dimension to create files, so the number of files is still a lot, the amount of data written in the ConsumeQueue here is very small, each message is only 20 Bytes, 30W pieces of data is only about 6M, so in fact, the impact on us is much smaller than the impact between Kafka's topics. Our entire logic can be as follows:

Producer continuously adds new messages to CommitLog. There is a timed task ReputService that will continuously scan the newly added CommitLog, and then continuously build ConsumerQueue and Index.

Note: This refers to ordinary hard disks. The concurrent writing of multiple files and the writing of a single file on the SSD have little effect.

Read message

Each Partition in Kafka will be a separate file, so when a message is consumed, it will be read in order. We know that when the OS reads the file from the physical disk, it will sequentially read other adjacent blocks. The data file is pre-read and the data is put into PageCache, so Kafka's message reading performance is better.

The RocketMQ reading process is as follows:

• First read the offset in ConsumerQueue corresponding to the physical offset of CommitLog
• Read CommitLog according to offset

ConsumerQueue is also a separate file for each Queue, and its file size is small, so it is easy to use PageCache to improve performance. As for CommitLog, since the continuous messages of the same Queue are actually discontinuous in CommitLog, it will cause random reads. RocketMQ has made several optimizations:

• Mmap mapping reads, the Mmap method reduces the performance overhead of traditional IO copying disk file data back and forth between the buffer of the operating system kernel address space and the buffer of the user application address space
• Use DeadLine scheduling algorithm + SSD storage disk
• Since Mmap mapping is limited by memory, when this part of the data is not mapped by Mmmap (that is, the message is accumulated too much), the default is 40% of the memory, and the request will be sent to the SLAVE to reduce the pressure on the Master

  3.6.2 Availability

3.6.2.1 Cluster Mode

We first need to choose a cluster mode to adapt to the degree of usability that we can endure. Generally speaking, there are three types:

• Single Master: In this mode, the availability is the lowest, but the cost is also the lowest. Once down, everything is unavailable. This generally only applies to local testing.
• Single Master with multiple SLAVE: This mode has general availability. If the master is down, all writes are unavailable and reads are still available. If the master disk is damaged, you can rely on the data of the slave.
• Multi-Master: This mode has general availability. If part of the master is down, the messages on this part of the master cannot be consumed, and data cannot be written. If a Topic queue is on multiple Masters, it can be guaranteed that there is no The down part can be consumed and written normally. If the master's disk is damaged, the message will be lost.
• Multi-Master and Multi-Slave: This mode has the highest availability, but the highest maintenance cost. When the master goes down, only the queue on this part of the master will not be writable, but reading is still possible, and if the master The disk is damaged and can rely on the data of the slave.

Generally speaking, the fourth type will be selected when it is put into production environment to ensure the highest availability.

3.6.2.2 Availability of messages

When we choose the cluster mode, then what we need to care about is how to store and copy this data. RocketMQ provides synchronous and asynchronous strategies for message flashing to meet our needs. When we choose synchronous flashing, if FLUSH_DISK_TIMEOUT will be returned after the flashing timeout. If it is an asynchronous flashing, it will not return the related information of the flashing. Choosing the synchronous flashing can satisfy our message as far as possible without losing.

In addition to storage options, our master-slave synchronization provides synchronous and asynchronous modes for replication. Of course, choosing synchronization can improve availability, but the message sending RT time will drop by about 10%.

3.6.3 Dleger

We have done a lot of analysis on the master-slave deployment mode above. We found that when there is a problem with the master, our writes will be unavailable unless the master is restored, or our slave is manually switched to the master, which leads to us In most cases, the Slave can only read. RocketMQ has launched Dleger-RocketMQ in recent versions. It uses the Raft protocol to replicate CommitLog and automatically selects the master, so that when the master is down, writing remains available.

For more information about Dleger-RocketMQ, you can check this article: Dledger-RocketMQ commitlog repository based on Raft protocol.

3.7 Timing/Delayed Message

Timed messages and delayed messages are often used in actual business scenarios, such as the following scenarios:

• Orders are automatically closed if they are not paid for overtime, because the inventory is locked after the order is placed in many scenarios, and it needs to be closed over time.
• Some delayed operations are required, such as some bottoming logic. After completing a certain logic, you can send a delayed message, such as a half-hour delay, to perform bottoming check compensation.
• To send a message to the user at a certain time, a delayed message can also be used.

In the open source version of RocketMQ, the delay message does not support any time delay. You need to set several fixed delay levels. The current default setting is: 1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h, from 1s to 2h correspond to levels 1 to 18, and the version in Alibaba Cloud (with payment) can support any time (millisecond level) within 40 days. Let's first look at the schematic diagram of the timing task in RocketMQ:

• Step1: Producer sets the level of delay required on the message sent by itself.
• Step2: Broker finds that this message is a delayed message and replaces Topic with delayed Topic. Each delay level will be used as a separate queue, and its Topic will be stored as additional information.
• Step3: Construct ConsumerQueue
• Step4: Timed tasks scan the ConsumerQueue of each delay level regularly.
• Step5: Get the Offset of CommitLog in ConsumerQueue, get the message, and judge whether the execution time has been reached
• Step6: If it reaches, restore the topic of the message and re-deliver it. If it is not reached, the execution of the task will be delayed for the period of time not reached.

It can be seen that the delayed message is realized by creating a separate Topic and Queue. If we want to achieve any time within 40 days, based on this scheme, we need 402460601000 queues. This cost is very high. So how is the support on Alibaba Cloud realized at any time? The guess here is to persist the secondary TimeWheel time wheel. The secondary time wheel is used to replace our ConsumeQueue, save the Commitlog-Offset, and then continuously take out the current time that has arrived through the time wheel, and then deliver the message again. The specific implementation logic requires a separate article to be written later.

3.8 Transaction message

Transaction messages are also a major feature in RocketMQ, which can help us complete the final consistency of distributed transactions. For distributed transactions, see many of my previous articles with many detailed introductions. Here, please pay attention to the official account directly. : Coffee latte.

The specific steps for using transaction messages are as follows:

• Step1: Call sendMessageInTransaction to send transaction messages
• Step2: If the transmission is successful, execute the local transaction.
• Step3: If the execution of the local transaction succeeds, send a commit, if it fails, send a rollback.
• Step4: If one of the stages, such as commit sending fails, rocketMQ will periodically check back from Broker, the status of the local transaction.

The whole process of using transaction message is more complicated than the previous several messages. The following is the schematic diagram of the realization of transaction message:

• Step1: Send a transaction message, also called halfMessage here, and replace Topic with Topic of HalfMessage.
• Step2: Send a commit or rollback. If it is a commit, the previous message will be queried, then the message will be restored to the original Topic, and an OpMessage will be sent to record that the current message can be deleted. If it is a rollback, an OpMessage will be sent directly to delete.
• Step3: There is a timing task for processing transaction messages in Broker. Compare halfMessage and OpMessage regularly. If there is OpMessage and the status is delete, then the message must be commit or rollback, so this message can be deleted.
• Step4: If the transaction times out (the default is 6s) and there is no opMessage yet, then it is very likely that the commit message is lost. Here we will go back and check the local transaction status of our Producer.
• Step5: Do Step2 according to the information queried.

We found that RocketMQ implements transaction messages by modifying the original Topic information, just like the delayed message, and then simulates it as a consumer for consumption and does some special business logic. Of course, we can also use this method to do more expansion of RocketMQ.

4. Summary

Here let us return to the questions mentioned in the article:

• What are the topics and queues of RocketMQ, and how are they different from Kafka partitions?
• What is the RocketMQ network model and how does it compare with Kafka?
• What is the RocketMQ message storage model, how to ensure highly reliable storage, and how does it compare with Kafka?

After reading this article, you must have the answer in your mind. This article mainly talks about RocketMQ's comprehensive design architecture. If you haven't read enough, then please pay attention to my official account.

If you think this article is helpful to you, your attention and forwarding are my greatest support, O(∩_∩)O: