Use of RocketMQ message middleware

Use of RocketMQ message middleware

Why choose RocketMQ?

According to our research, the ActiveMQ IO module reaches a bottleneck as the number of queues and virtual topics in use increases. We tried our best to solve this problem through throttling, circuit breaker or downgrade, but it didn't work well. So we started looking at Kafka, a popular messaging solution at the time. Unfortunately, Kafka did not meet our requirements, especially in terms of low latency and high reliability, see here for details.

In this case, we decided to invent a new messaging engine to handle a wider set of use cases, from traditional publish/subscribe scenarios to high-volume real-time zero-loss-tolerant transaction systems. We believe this solution could be beneficial, so we would like to open source it to the community. Today, more than 100 companies use the open source version of RocketMQ in their operations.

Message queue is a "first in, first out" data structure

Its application scenarios mainly include the following three aspects:

1. Application decoupling

The more coupled a system is, the less fault tolerant it is. Take e-commerce applications as an example. After a user creates an order, if the inventory system, logistics system, and payment system are coupled and called, and any subsystem fails or is temporarily unavailable due to upgrades or other reasons, it will cause abnormal ordering operations and affect the user's use. experience.

Using message queues to decouple will improve the coupling of the system. For example, if the logistics system fails, it will take several minutes to repair it. During this time, the data to be processed by the logistics system is cached in the message queue, and the user's order operation is completed normally. When the logistics system responds, it can supplement the order messages stored in the message queue. The terminal system will not be aware that the logistics system has failed for a few minutes.

2. Traffic peak clipping

If an application system encounters a sudden surge in system request traffic, it may overwhelm the system. With the message queue, a large number of requests can be cached and processed over a long period of time, which can greatly improve the stability of the system and user experience.

Generally speaking, in order to ensure the stability of the system, if the system load exceeds the threshold, user requests will be blocked, which will affect the user experience. If the message queue is used to cache the request, wait for the system to complete the processing and notify the user that the order is completed. In this way You can’t always place an order and have a better experience.

For economic reasons:

If the QPS of the business system during normal periods is 1,000 and the peak traffic is 10,000, it is obviously not cost-effective to configure a high-performance server to cope with the traffic peak. In this case, message queues can be used to reduce the peak traffic.

3. Data distribution

Message queues allow data to flow between multiple systems. The data producer does not need to care about who uses the data. It only needs to send the data to the message queue, and the data user can obtain the data directly in the message queue.

4. Advantages and Disadvantages of MQ

Advantages: decoupling, peak clipping, data distribution

Disadvantages include the following:

• Reduced system availability

  The more external dependencies introduced into the system, the worse the stability of the system. Once MQ goes down, it will have an impact on the business.

  How to ensure the high availability of MQ?

• Increased system complexity

  The addition of MQ has greatly increased the complexity of the system. In the past, synchronous remote calls were made between systems, but now asynchronous calls are made through MQ.

  How to ensure that messages are not consumed repeatedly? How to deal with message loss? So ensure the order of message delivery?

• Consistency issue

  After system A processes the business, it sends message data to systems B, C, and D through MQ. If system B and system C process it successfully, system D fails to process it.

  How to ensure the consistency of message data processing?

5. What are the differences between RocketMQ vs. ActiveMQ vs. Kafka

Messaging products	Client SDK	protocols and specifications	Ordering Information	scheduled message	Bulk messages	Broadcast message	Message filter	Server triggered redelivery	Message storage	Message tracing	Message priority	High availability and failover	Message tracking	Configuration	Management and operational tools
ActiveMQ	Java, .NET, C++, etc.	Push model, supports OpenWire, STOMP, AMQP, MQTT, JMS	Exclusive Consumer or Exclusive Queues can guarantee ordering	supported	not support	supported	supported	not support	Supports very fast persistence using JDBC and high-performance logging, such as levelDB, kahaDB	supported	supported	Supported, depends on storage, requires ZooKeeper server if using levelDB	not support	The default configuration is low-level, users need to optimize configuration parameters	supported
kafka	Java、Scala etc.	Pull model, supports TCP	Ensure ordering of messages within a partition	not support	Supported, with async producers	not support	Yes, messages can be filtered using Kafka Streams	not support	High performance file storage	Supported offset indications	not support	Supported, requires ZooKeeper server	not support	Kafka uses a key-value pair format for configuration. These values ​​can be provided from a file or programmatically.	Support, use terminal commands to expose core indicators
RocketMQ	Java, C++, Go	Pull model, support TCP, JMS, OpenMessaging	Ensures strict ordering of messages and scales out gracefully	supported	Supported, with synchronous mode to avoid message loss	supported	Supported SQL92-based attribute filter expressions	supported	High-performance and low-latency file storage	Supported timestamp and offset representations	not support	Supported master-slave models, no additional kit required	supported	Ready to use out of the box, users only need to pay attention to a few configurations	Supported, rich web and terminal commands to expose core metrics

Official website address: https://rocketmq.apache.org/

1. Download the RocketMQ installation package

**Requirements:** Preparation

Software and hardware requirements

• Linux64-bit system
• JDK1.8(64-bit)
• Source code installation requires Maven 3.2.x.
• **Note:** Integrating RocketMQ requires open ports: 9876, 10912, 10911, 10909. If these ports are not opened, the project will not be accessible.

We download version 4.9.3 here

Material: RockerMQ material download (already configured, just follow the steps below)

In order to facilitate everyone to learn quickly, the materials are prepared here, you can also follow the document to download from the official website

2. Configure using Linux

2.1 Use the command mkdir to create the /usr/local/recketMQ directory

Use the command cd to jump to the /usr/local/recketMQ directory

2.2 Use Xftp to open Xshell for the currently logged in user to perform file operations

Jump to the /usr/local/recketMQ directory for file transfer

2.3 Use unzip rocketmq-all-4.9.3-source-release.zip to decompress the file

You need to download the unzip decompression tool first, command: yum install unzip

2.4 Use cd to jump to the /usr/local/recketMQ/rocketmq-4.9.3 directory

Use ll to view file details in the current directory

2.5. Modify the initial memory

The default virtual machine memory of RocketMQ is large. Starting the Broker or NameServer may fail due to insufficient memory. Therefore, you need to edit the following two configuration files to modify the JVM memory size.

2.5.1. Modify runserver.sh

Use the vim command to open the bin/runserver.sh file. The first 71 and 76 lines have not been modified, but are modified as follows:

2.5.2 Modify runbroker.sh

Use the vim command to open the bin/runbroker.sh file. The first 85 lines have not been modified and are modified as follows:

3. Start RocketMQ service

nohup means running the command without hanging up. When the account is logged out or the terminal is closed, the program is still running.

3.1 Start NameServer

nohup sh bin/mqnamesrv & 

tail -f ~/logs/rocketmqlogs/namesrv.log 

Started nameserver successfully

3.2 Start broker

nohup sh bin/mqbroker -n 192.168.230.128:9876  &

配置broker文件之后使用：nohup sh bin/mqbroker -n 192.168.230.128:9876 -c /usr/local/recketMQ/rocketmq-4.9.3/conf/broker.conf &

tail -f ~/logs/rocketmqlogs/broker.log 

Failed to start broker,

Right-click the nohup.out file and open it with EditPlus editor to view errors.

As shown below, the memory is not configured successfully, resulting in insufficient memory. Need to reconfigure

Just reconfigure it

Started broker successfully

3.3 Send/receive message test

Send a message

export NAMESRV_ADDR=192.168.230.128:9876 

sh bin/tools.sh org.apache.rocketmq.example.quickstart.Producer

receive messages

sh bin/tools.sh org.apache.rocketmq.example.quickstart.Consumer 

3.4 Shut down the server

Whether to shut down the name server or broker, use the bin/mqshutdown command.

[root@mqOS rocketmq]# 关闭broker服务命令： sh bin/mqshutdown broker 

The mqbroker(1740) is running..	. 

Send shutdown request to mqbroker(1740) OK 

[root@mqOS rocketmq]# 关闭nameserver服务命令: sh bin/mqshutdown namesrv 

The mqnamesrv(1692) is running... 

Send shutdown request to mqnamesrv(1692) OK 

[2]+ 退出 143 nohup sh bin/mqbroker -n localhost:9876 

4. Console installation and startup

RocketMQ has a visual dashboard through which a lot of data can be viewed intuitively.

**If you don’t want to download from the official website, you can directly use the console provided by the material (already configured)** Just follow the steps below.

4.1 Installation

4.1.1 Download

Download address: https://github.com/apache/rocketmq-dashboard

4.1.2 Modify configuration

Modify its application.properties configuration file in src/main/resources.

The original port number was 8080, which was changed to an uncommon one.

Specify the name server address of RocketMQ

4.1.3 Upload rocketmq-dashboard-master to the /usr/local/recketMQ directory

Use the cd command to jump to the /usr/local/recketMQ directory

4.1.4 Use unzip to decompress rocketmq-dashboard-master

4.1.5 Use the cd command to jump to the /usr/local/recketMQ/rocketmq-dashboard-master directory

4.2 Startup

4.2.1 Front-end packaging

Annotation packaging plug-in frontend-maven-plugin

<!--            <plugin>-->
<!--                <groupId>com.github.eirslett</groupId>-->
<!--                <artifactId>frontend-maven-plugin</artifactId>-->
<!--                <version>1.11.3</version>-->
<!--	        </plugin>-->

4.2.2 Backend packaging

Run the maven packaging command in the rocketmq-dashboard-master directory. (Note: The maven warehouse must be prepared in the virtual machine)

Maven installation URL

Excuting an order:

chmod a+x /usr/local/maven/apache-maven-3.6.0/bin/mvn
mvn clean package -Dmaven.test.skip=true

**Error report:** Most of the others are version issues, just match the version.

**Solution:** Just comment out the frontend-maven-plugin plug-in.

<!--            <plugin>-->
<!--                <groupId>com.github.eirslett</groupId>-->
<!--                <artifactId>frontend-maven-plugin</artifactId>-->
<!--                <version>1.11.3</version>-->
<!--            </plugin>-->

4.2.2 Startup

Use the cd command to jump to the /usr/local/recketMQ/rocketmq-dashboard-master/target directory

Excuting an order:

nohup java -jar rocketmq-dashboard-1.0.1-SNAPSHOT.jar &

4.2.3 Access

Use http://IP address:port

At this point, our RocketMQ integration is successful

5. RocketMQ use

5.1 Open the project in the material

Click to start the project

5.2 Open the http://localhost:8380/doc.html interface document

1. General news

1. Perform data testing (normal messages)

2. Control layer code

3. Test Results

2. Delayed message

1. Perform data testing (delayed messages)

2. Control layer code

3. Test Results

6. Customized delay time level

Add the following lines to the property configuration file broker.conf on the server side (rocketmq-broker side):

messageDelayLevel=1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h
describes the corresponding mapping relationship between each level and the delay time.

1. This configuration item configures the delay time of each level starting from level 1. The delay time of this specified level can be modified;

2. Time units support: s, m, h, d, representing seconds, minutes, hours, and days respectively;

3. The default value is the one declared above and can be adjusted manually;

4. The default value is sufficient and it is not recommended to modify this value.

At this point, our study of RockerMQ is completed.