Detailed explanation of Rocket MQ

Rocket MQ

1. What is it and where does it come from?

RocketMQ is an open source distributed messaging middleware originally developed by Alibaba Group. Its design goal is to achieve reliable message transmission in high-concurrency and high-throughput scenarios, and to have good scalability and expansibility.

RocketMQ supports multiple message modes, including synchronous, asynchronous, one-way and scheduled messages. At the same time, RocketMQ also has the advantages of high reliability, low latency, high throughput , distributed deployment, etc., and can be widely used in many fields, such as e-commerce, finance, logistics, etc.

RocketMQ has now become one of the top open source projects under the Apache Software Foundation and has been widely used and recognized around the world.

message model

Simple message model:

RocketMQ’s expanded message model:

ps：

• There are two main load balancing modes for consumers under the same ConsumerGroup, namely broadcast mode and cluster mode (the most commonly used cluster mode is shown in the figure).
• In cluster mode, the Consumer instances in the same ConsumerGroup are load-balanced consumers. In the figure, ConsumerGroupA subscribes to TopicA, and TopicA corresponds to 3 queues. Then Consumer1 in GroupA consumes messages from MessageQueue 0 and MessageQueue 1, and Consumer2 consumes them. It is the message of MessageQueue2.
• In broadcast mode, each Consumer instance in the same ConsumerGroup handles the entire queue. It should be noted that in broadcast mode, because each Consumer instance needs to process all messages (ConsumerGroup B in the figure), this mode is only recommended for small traffic scenarios such as notification push and configuration synchronization.

Second, what can you do?

RocketMQ is a highly reliable, high-throughput, scalable, easy-to-deploy and manage distributed message middleware that can be used to implement functions such as asynchronous transmission of messages, decoupling, and peak-shaving and valley-filling. Specifically, RocketMQ can be used in the following scenarios:

1. Event-driven architecture: RocketMQ can be used to asynchronously transmit events to subscribers, thereby realizing an event-driven architecture, reducing the coupling of the system, and improving the scalability and maintainability of the system.
2. Streaming computing: RocketMQ can be used to transmit real-time streaming data, such as logs, monitoring data, transaction data, etc., for streaming computing and real-time data analysis.
3. Asynchronous processing: RocketMQ can be used to implement asynchronous processing and transfer some time-consuming tasks to asynchronous threads or other services for execution, thereby improving the throughput and performance of the system.
4. Peak shaving and valley filling: RocketMQ can be used to implement the peak shaving and valley filling function of the message queue. When the number of requests the system bears exceeds the system's processing capacity, the requests can be converted into messages and cached through RocketMQ, waiting for the system to return to normal. Process again.
5. Distributed transactions: RocketMQ provides message-based distributed transaction support, which can ensure the atomicity and consistency of messages among multiple nodes.

In short, RocketMQ is a powerful distributed message middleware, suitable for message transmission and asynchronous processing in various scenarios.

RocketMQ emphasizes the characteristics of low latency, high reliability, and high throughput.

3. What are similar products?

1. ActiveMQ (2004): Apache ActiveMQ is an open source message middleware based on the JMS protocol, written in the Java language, and can be used to build distributed systems and enterprise applications. ActiveMQ supports a variety of protocols, including OpenWire, Stomp, AMQP, MQTT, etc., and also provides REST and WebSocket interfaces to support message persistence and transactions.
2. RabbitMQ (2007): RabbitMQ is an open source message middleware based on the AMQP protocol, written in the Erlang language, with high performance and scalability. RabbitMQ supports multiple language clients, including Java, Python, Ruby, etc. It supports features such as message persistence and transactions, and also supports message confirmation and retransmission mechanisms. The plugin supports sequential messages.
3. Kafka (2011): Kafka is a distributed high-throughput message queuing system written in Scala and developed by LinkedIn. The design concept of Kafka is a messaging system based on the publish/subscribe model, which supports multiple consumers to consume messages from the same Topic in parallel. It also supports features such as horizontal expansion and data persistence.
4. RocketMQ (2012): RocketMQ is Alibaba's open source distributed message middleware, written in Java language, and supports message sequence and high availability. The design concept of RocketMQ is to implement message routing based on Topic and Tag. It also supports features such as batch sending and message transactions, and is suitable for large-scale distributed systems and real-time data processing scenarios.

4. Comparison of similar products

MQ products	performance	Cluster deployment	sequential message	timed message	Message storage	Development language
ActiveMQ	Low, even worse when the amount of data is large	More complex to manage	Supported by exclusive consumer (exclusive consumer) and message group (sent to the same consumer in the same group)	support	Supports fast persistence using JDBC and high-performance logs such as levelDB and kahaDB.	Java
RabbitMQ	Low, even worse when the amount of data is large	not support	Support (single queue)	Support (need to introduce plug-ins to support)	The Mnesia database that comes with the Erlang language or supports Mysql and PostageSql through plug-ins.	Erlang
Kafka	High throughput, low latency	Support distributed deployment and horizontal expansion	not support	not support	Disk and memory mixed. Kafka stores messages in one or more log files on disk, while using in-memory caching to improve read and write performance and throughput.	Scala
RocketMQ	High reliability, high throughput, low latency, suitable for high concurrency and large-scale data processing scenarios.	Support distributed deployment and horizontal expansion	Supported (a set of sequential messages is sent to the same queue in the same broker)	support	CommitLog, this storage method persists messages to disk and uses memory mapping technology to accelerate read and write operations. CommitLog stores messages in the form of message queues and supports multiple indexing methods, including hash index, time index, etc. RocketMQ also supports the use of external storage services to store messages, such as Hadoop, MySQL, MongoDB, etc.	Java

Five, principle

The implementation principles of RocketMQ can be divided into the following aspects:

1. Architecture design: RocketMQ's architecture is divided into four layers, namely client, Broker, NameServer and storage layer. Messages are transmitted between the client and the Broker through network communication. The NameServer is responsible for maintaining the Broker's metadata information, and the storage layer is responsible for actually storing the messages.
2. Message storage: RocketMQ uses CommitLog to store messages, that is, append messages to files on disk. There are one or more CommitLog files on each Broker node. Each file is divided into multiple fixed-size message storage segments. Each storage segment contains multiple messages. Each message consists of three parts: message length, message content and message attributes. Partially composed.
3. Message routing: RocketMQ implements message routing through Topic and Tag. When the Producer sends a message, it needs to specify the Topic and Tag to which the message belongs. The Broker distributes the message to the corresponding Consumer based on the Topic and Tag. Each Topic can have multiple Tags, and each Consumer can subscribe to messages from one or more Topics and Tags.
4. HA mechanism: RocketMQ achieves high availability through master-slave replication, that is, each Broker node has a Master node and multiple Slave nodes. The Master node is responsible for receiving messages from the Producer and sending messages to the Consumer, and at the same time synchronizing the messages to the Slave node. The Slave node is used for backup and disaster recovery. When the Master node goes down, it will automatically switch to the Slave node to continue the service.
5. Message sequence guarantee: RocketMQ can guarantee the sequence of messages through sequential messages, that is, the sequence of messages sent by the Producer remains unchanged when stored and consumed on the Broker. RocketMQ implements sequential messages through Topic and Message Queue. Each Message Queue can only be consumed by one Consumer. All messages sent by the Producer will be written to the same Message Queue in order, and the Consumer will consume the messages in the Queue in order .

In general, the implementation principles of RocketMQ mainly include message storage, message routing, HA mechanism, message sequence guarantee, etc. These technical means make RocketMQ have the characteristics of high performance, high reliability and high scalability.

6. How to use

Deployment model

ps: NameServer role

1. To manage Brokers, each Broker will be registered in a NameServer and send heartbeats regularly. NameServer provides a heartbeat detection mechanism to check whether the Broker is still alive.
2. To manage routing information, each NameServer will save the entire routing information about the entire Broker cluster and queue information for client queries. Product and Consumer can know the routing information of the entire Broker cluster through NameServer, so as to deliver and consume messages.

Deploy components

Broker，NameServer,Provider,Consumer

Deployment process

Purpose: Deploy a Master+slaver node RocketMQ under a Linux server and use it in the springboot project.

1. Download and install RocketMQ

   You can download the required version of the installation package from RocketMQ's official website (http://rocketmq.apache.org/) and unzip it to get the RocketMQ installation directory. Copy the installation directory to a suitable location on the server, /usr/local/rocketmqe.g.

2. Modify configuration file

   In /usr/local/rocketmq/confthe directory, copy broker.confthe files and rename them to broker-a.properties(configuration file for the master node) and broker-b.properties(configuration file for the slave node). In the two configuration files, the main configuration items that need to be modified include:

   • brokerName: Set the Broker name, which can be customized.
   • brokerId: Set the unique ID of the Broker. The IDs of the Master and Slave nodes cannot be the same.
   • namesrvAddr: Set the address and port of NameServer in the format <name-server-ip>:9876.
   • listenPort: Set the Broker's listening port. The ports of the Master and Slave nodes cannot be the same.

   example:

   broker-master.properties：

   # Master节点配置
   brokerName=master
   brokerId=0
   namesrvAddr=localhost:9876
   listenPort=10911
   brokerRole=SYNC_MASTER
   

   broker-slave.properties：

   # Slaver节点配置
   brokerName=slaver
   brokerId=1
   namesrvAddr=localhost:9876
   listenPort=10912
   brokerRole=SLAVE
   

3. Start NameServer

   Execute the following command on the command line to start NameServer:

   cd /usr/local/rocketmq/bin
   nohup sh mqnamesrv &
   

4. Start Master and Slave nodes

   Execute the following commands on the command line to start the Master and Slave nodes:

   cd /usr/local/rocketmq/bin
   nohup sh mqbroker -c ../conf/broker-master.properties &
   nohup sh mqbroker -c ../conf/broker-slave.properties &
   

   This will start two nodes and bind the Master node and Slave node.

   Where to view the logs:

   当前文件夹下的nohup.out或者~/logs/rocketmqlogs文件下。
   

5. Verify node status

   Execute the following command to view the node status:

   cd /usr/local/rocketmq/bin
   sh mqadmin clusterList -n <name-server-ip>:9876
   

   If the node starts normally, the information of the Master and Slave nodes will be output.

6. Using RocketMQ in Spring Boot project

   First, you need to add RocketMQ dependencies, and pom.xmladd the following dependencies to the file:

   <dependency>
       <groupId>org.apache.rocketmq</groupId>
       <artifactId>rocketmq-spring-boot-starter</artifactId>
       <version>${rocketmq.version}</version>
   </dependency>
   

   In the configuration file of the Spring Boot project, you need to add the following configuration items:

   rocketmq.name-server=192.168.1.100:9876
   

   where 192.168.1.100:9876is RocketMQ's NameServer address and port.

   In classes that need to use RocketMQ, you can use @RocketMQMessageListenerannotations to listen for messages:

   @Service
   @RocketMQMessageListener(topic = "my-topic", consumerGroup = "my-consumer-group", selectorExpression = "my-tag")
   public class MyRocketMQListener implements RocketMQListener<String> {
       @Override
       public void onMessage(String message) {
           // 处理消息
       }
   }
   

   which topicrepresents the topic to be monitored