22. Rocketmq integrates SpringBoot
22.1 Build rocketmq-producer (message producer)
22.1.1 Create project, complete pom.xml
_<?_xml version="1.0" encoding="UTF-8"_?>_
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 [https://maven.apache.org/xsd/maven-4.0.0.xsd"](https://maven.apache.org/xsd/maven-4.0.0.xsd%22)
>
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.6.3</version>
<relativePath/>
_<!-- lookup parent from repository --> _
</parent>
<groupId>com.powernode</groupId>
<artifactId>01-rocketmq-producer</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>rocketmq-producer</name>
<description>Demo project for Spring Boot</description>
<properties>
<java.version>1.8</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>** **
**_<!-- rocketmq的依赖 --> _**
**<dependency> **
**<groupId>org.apache.rocketmq</groupId> **
**<artifactId>rocketmq-spring-boot-starter</artifactId> <version>2.0.2</version> **
**</dependency> **
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<excludes>
<exclude>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
</exclude>
</excludes>
</configuration>
</plugin>
</plugins>
</build>
</project>
22.1.2 Modify the configuration file application.yml
spring:
application:
name: rocketmq-producer
rocketmq:
name-server: 127.0.0.1:9876 # rocketMq的nameServer地址
producer:
group: powernode-group # 生产者组别
send-message-timeout: 3000 # 消息发送的超时时间
retry-times-when-send-async-failed: 2 # 异步消息发送失败重试次数
max-message-size: 4194304 # 消息的最大长度
22.1.3 We test sending messages in the test class
Send a simple string message to the powernode topic
/** * 注入rocketMQTemplate，我们使用它来操作mq */
@Autowiredprivate RocketMQTemplate rocketMQTemplate;
/** * 测试发送简单的消息 * * @throws Exception */
@Testpublic void testSimpleMsg() throws Exception {
// 往powernode的主题里面发送一个简单的字符串消息
SendResult sendResult = rocketMQTemplate.syncSend("powernode", "我是一个简单的消息");
// 拿到消息的发送状态
System._out_.println(sendResult.getSendStatus());
// 拿到消息的id
System._out_.println(sendResult.getMsgId());
}
Check the console after running
22.1.4 View rocketMq console
View message details
22.2 Build rocketmq-consumer (message consumer)
22.2.1 Create project, complete pom.xml
_<?_xml version="1.0" encoding="UTF-8"_?>_
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 [https://maven.apache.org/xsd/maven-4.0.0.xsd"](https://maven.apache.org/xsd/maven-4.0.0.xsd%22)
>
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId> ** **
**<version>2.6.3</version>**
<relativePath/>
_<!-- lookup parent from repository --> _
</parent>
<groupId>com.powernode</groupId>
<artifactId>02-rocketmq-consumer</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>rocketmq-consumer</name>
<description>Demo project for Spring Boot</description>
<properties>
<java.version>1.8</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency> ** **
**_<!-- rocketmq的依赖 --> _**
**_ _<dependency> **
**<groupId>org.apache.rocketmq</groupId> **
**<artifactId>rocketmq-spring-boot-starter</artifactId>**
** <version>2.0.2</version> **
**</dependency>**
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<excludes>
<exclude>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
</exclude>
</excludes>
</configuration>
</plugin>
</plugins>
</build>
</project>
22.2.2 Modify the configuration file application.yml
| spring:
application:
name: rocketmq-consumerrocketmq:
name-server: 127.0.0.1:9876
| — |
22.2.3 Add a listening class SimpleMsgListener
To consume messages, consumers add a listener
package com.powernode.listener;
import org.apache.rocketmq.spring.annotation.MessageModel;
import org.apache.rocketmq.spring.annotation.RocketMQMessageListener;
import org.apache.rocketmq.spring.core.RocketMQListener;
import org.springframework.stereotype.Component;
_/** * 创建一个简单消息的监听 * _
_1.类上添加注解@Component和@RocketMQMessageListener *
@RocketMQMessageListener(topic = "powernode", consumerGroup = "powernode-group") *
topic指定消费的主题，consumerGroup指定消费组,一个主题可以有多个消费者组,一个消息可以被多个不同的组的消费者都消费 * _
_2.实现RocketMQListener接口，**注意泛型的使用，可以为具体的类型，如果想拿到消息 * 的其他参数可以写成MessageExt** */_
@Component@RocketMQMessageListener(topic = "powernode", consumerGroup = "powernode-group",messageModel = MessageModel._CLUSTERING_)public class SimpleMsgListener implements RocketMQListener<String> {
_/** * 消费消息的方法 * * @param message */ _
@Override public void onMessage(String message) {
System._out_.println(message);
}
}
22.2.4 Start rocketmq-consumer
Check the console and find that we have listened to the message
23. RocketMQ sends object messages and collection messages
Let's do it in the above project
23.1 Sending Object Messages
The main thing is to write the type of the object in the generic when listening
23.1.1 Modify rocketmq-producer to add an Order class
package com.powernode.domain;
import lombok.AllArgsConstructor;import lombok.Data;
import lombok.NoArgsConstructor; import java.util.Date;
/** * 订单对象 */
@Data@AllArgsConstructor@NoArgsConstructorpublic class Order {
/** * 订单号 */
private String orderId;
/** * 订单名称 */
private String orderName;
/** * 订单价格 */
private Double price;
/** * 订单号创建时间 */
private Date createTime;
/** * 订单描述 */
private String desc;
}
23.1.2 Modify rocketmq-producer to add a unit test
_/** * 测试发送对象消息 * * @throws Exception */_
@Testpublic void testObjectMsg() throws Exception {
Order order = new Order();
order.setOrderId(UUID._randomUUID_().toString());
order.setOrderName("我的订单");
order.setPrice(998D);
order.setCreateTime(new Date());
order.setDesc("加急配送");
_// 往powernode-obj主题发送一个订单对象 _
rocketMQTemplate.syncSend("powernode-obj", order);
}
23.1.3 Send this message
23.1.4 Modify rocketmq-consumer and add an Order class (copy it)
23.1.5 Modify rocketmq-consumer to add an ObjMsgListener
package com.powernode.listener; import com.powernode.domain.Order;
import org.apache.rocketmq.spring.annotation.RocketMQMessageListener;
import org.apache.rocketmq.spring.core.RocketMQListener;
import org.springframework.stereotype.Component;
/** * 创建一个对象消息的监听
* 1.类上添加注解@Component和@RocketMQMessageListener
* * 2.实现RocketMQListener接口，注意泛型的使用 */
@Component@RocketMQMessageListener(topic = "powernode-obj", consumerGroup = "powernode-obj-group")
public class ObjMsgListener implements RocketMQListener<**Order**> {
/** * 消费消息的方法 * * @param message */
@Override public void onMessage(**Order **message) {
System._out_.println(message);
}}
23.1.6 Check the console after restarting rocketmq-consumer
The object message has been listened to
23.2 Send collection message
In the same way as object messages, create a collection of Orders and send them out. The listener should pay attention to modify the type in the generic type to Object, so I won’t repeat the demonstration here.
24. RocketMQ integrates SpringBoot to send different message modes
24.1 Sending Synchronous Messages
It is understood that after the message is sent from the consumer to the broker, it will get a confirmation, which is reliable.
This reliable and synchronous sending method is widely used, such as: important message notifications, SMS notifications, etc.
The message we demonstrated in the above quick start is a synchronous message, that is,
rocketMQTemplate.syncSend()
rocketMQTemplate.send()
rocketMQTemplate.convertAndSend()
These three methods of sending messages, the bottom layer is to call syncSend, and what is sent is a synchronous message
24.2 Sending asynchronous messages
rocketMQTemplate.asyncSend()
24.2.1 Modify rocketmq-producer to add a unit test
/** * 测试异步发送消息 * * @throws Exception */
@Testpublic void testAsyncSend() throws Exception {
// 发送异步消息，发送完以后会有一个异步通知
rocketMQTemplate.**asyncSend**("powernode", "发送一个异步消息", new SendCallback() {
/** * 成功的回调 * @param sendResult */
@Override public void onSuccess(SendResult sendResult) {
System._out_.println("发送成功");
}
/** * 失败的回调 * @param throwable */
@Override public void onException(Throwable throwable) {
System._out_.println("发送失败");
} });
// 测试一下异步的效果
System._out_.println("谁先执行");
// 挂起jvm 不让方法结束
System._in_.read();
}
24.2.2 Run to view the console effect
Whoever sends first prints at the front
24.3 Sending one-way messages
This method is mainly used in scenarios that do not care about the sending results. This method has a high throughput, but there is a risk of message loss, such as the sending of log information.
24.3.1 Modify rocketmq-producer to add a unit test
/** * 测试单向消息 * * @throws Exception */
@Testpublic void testOnWay() throws Exception {
// 发送单向消息，没有返回值和结果 rocketMQTemplate.**sendOneWay**("powernode", "这是一个单向消息");
}
24.4 Sending delayed messages
24.4.1 Modify rocketmq-producer to add a unit test
/** * 测试延迟消息 * * @throws Exception */
@Testpublic void testDelay() throws Exception {
// 构建消息对象
Message<String> message = MessageBuilder._withPayload_("我是一个延迟消息").build();
// 发送一个延时消息，延迟等级为4级，也就是30s后被监听消费
SendResult sendResult = rocketMQTemplate.syncSend("powernode", message, 2000, 4);
System._out_.println(sendResult.getSendStatus());
}
24.4.2 After running, check the consumer side and it will be consumed after 30s
It should be noted here that RocketMQ does not support delays at any time
. It only supports the following fixed delay levels. Level 1 corresponds to 1s, and so on, up to 2h delay is supported.
private String messageDelayLevel = “1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h”;
24.5 Sending sequential messages
24.5.1 Modify the Order table to add an order field
| /** * Order process sequence*/private Integer seq; |
|---|
24.5.2 Modify rocketmq-producer to add a unit test
/** * 测试顺序消费 * mq会根据hash的值来存放到一个队列里面去 * * @throws Exception */
@Testpublic void testOrderly() throws Exception {
List<Order> orders = Arrays._asList_(
new Order(UUID._randomUUID_().toString().substring(0, 5), "张三的下订单", null, null, null, 1),
new Order(UUID._randomUUID_().toString().substring(0, 5), "张三的发短信", null, null, null, 1),
new Order(UUID._randomUUID_().toString().substring(0, 5), "张三的物流", null, null, null, 1),
new Order(UUID._randomUUID_().toString().substring(0, 5), "张三的签收", null, null, null, 1),
new Order(UUID._randomUUID_().toString().substring(0, 5), "李四的下订单", null, null, null, 2),
new Order(UUID._randomUUID_().toString().substring(0, 5), "李四的发短信", null, null, null, 2),
new Order(UUID._randomUUID_().toString().substring(0, 5), "李四的物流", null, null, null, 2),
new Order(UUID._randomUUID_().toString().substring(0, 5), "李四的签收", null, null, null, 2) );
// 我们控制流程为 下订单->发短信->物流->签收 hash的值为seq，也就是说 seq相同的会放在同一个队列里面，顺序消费
orders.forEach(order -> {
rocketMQTemplate.syncSendOrderly("powernode-obj", order, String._valueOf_(order.getSeq()));
});
}
24.5.3 Sending Messages
24.5.4 Modify the ObjMsgListener of rocketmq-consumer
package com.powernode.listener; import com.powernode.domain.Order;
import org.apache.rocketmq.spring.annotation.ConsumeMode;
import org.apache.rocketmq.spring.annotation.RocketMQMessageListener;
import org.apache.rocketmq.spring.core.RocketMQListener;
import org.springframework.stereotype.Component;
/** * @Author 武汉动力节点 * 创建一个对象消息的监听 *
1.类上添加注解@Component和@RocketMQMessageListener *
2.实现RocketMQListener接口，注意泛型的使用 * consumeMode 指定消费类型 * CONCURRENTLY 并发消费 * ORDERLY 顺序消费 messages orderly. one queue, one thread */
@Component@RocketMQMessageListener(topic = "powernode-obj",
consumerGroup = "powernode-obj-group",
**consumeMode = ConsumeMode._ORDERLY_**)public class ObjMsgListener implements RocketMQListener<Order> {
/** * 消费消息的方法 * * @param message */
@Override public void onMessage(Order message) {
System._out_.println(message);
}}
24.5.5 Restart rocketmq-consumer
Looking at the console, the messages are consumed in the order we put them in
24.6 Send transaction message
24.6.1 Modify rocketmq-producer to add a unit test
/** * 测试事务消息 * 默认是sync（同步的） * 事务消息会有确认和回查机制 * 事务消息都会走到同一个监听回调里面，所以我们需要使用tag或者key来区分过滤 * * @throws Exception */
@Testpublic void testTrans() throws Exception {
// 构建消息体
Message<String> message = MessageBuilder._withPayload_("这是一个事务消息").build();
// 发送事务消息（同步的） 最后一个参数才是消息主题
TransactionSendResult transaction =
rocketMQTemplate.sendMessageInTransaction("powernode", message, "消息的参数");
// 拿到本地事务状态
System._out_.println(transaction.getLocalTransactionState());
// 挂起jvm，因为事务的回查需要一些时间
System._in_.read();
}
24.6.2 Modify rocketmq-producer to add a local transaction message monitor (half message)
_/** * 事务消息的监听与回查 * 类上添加注解@RocketMQTransactionListener 表示这个类是本地事务消息的监听类 * 实现RocketMQLocalTransactionListener接口 * 两个方法为执行本地事务，与回查本地事务 */_
@Component@RocketMQTransactionListener(corePoolSize = 4,maximumPoolSize = 8)
public class TmMsgListener implements RocketMQLocalTransactionListener {
_/** _
_* 执行本地事务，这里可以执行一些业务 _
_* 比如操作数据库，操作成功就return RocketMQLocalTransactionState.COMMIT; _
_* 可以使用try catch来控制成功或者失败; _
_* @param msg * @param arg * @return */ _
@Override public RocketMQLocalTransactionState executeLocalTransaction(Message msg, Object arg) {
_// 拿到消息参数 _
System._out_.println(arg);
_// 拿到消息头 _
System._out_.println(msg.getHeaders());
_// 返回状态COMMIT,UNKNOWN _
return RocketMQLocalTransactionState._UNKNOWN_;
}
_/** _
_* 回查本地事务，只有上面的执行方法返回UNKNOWN时，才执行下面的方法 默认是1min回查 _
_* 此方法为回查方法，执行需要等待一会 _
_* xxx.isSuccess() 这里可以执行一些检查的方法 _
_* 如果返回COMMIT，那么本地事务就算是提交成功了，消息就会被消费者看到 *_
_ * @param msg * @return */ _
@Override public RocketMQLocalTransactionState
checkLocalTransaction(Message msg) {
System._out_.println(msg);
return RocketMQLocalTransactionState._COMMIT_;
}}
24.6.3 Test send transaction, suggest breakpoint start
- The message will first arrive at the execution method of the transaction monitoring class,
- If the return status is COMMIT, consumers can directly monitor
- If the return status is ROLLBACK, the message failed to be sent and rolled back directly
- If the return status is UNKNOW, it will go back to the query method after a while
- If the return status of the checkback method is UNKNOW or ROLLBACK, the message sending fails and rolls back directly
- If the return status of the checkback method is COMMIT, consumers can directly monitor
25. RocketMQ integrates SpringBoot's message filtering
25.1 Tag filtering (often filtering on the consumer side)
We know from the source code comments that the tag is used after the subject: to carry, thanks for the comments
Let's go down to look at the source code,
and see the specific details in the getAndWrapMessage method of org.apache.rocketmq.spring.support.RocketMQUtil, we also know keys are carried in the message header
25.1.1 Modify rocketmq-producer to add a unit test
/** * Send a tagged message * * @throws Exception */
@Testpublic void testTagMsg() throws Exception { // Send a tag with java data rocketMQTemplate.syncSend("powernode-tag:java", "I am A tagged message"); }
25.1.2 Sending Messages
25.1.3 Modify rocketmq-consumer to add a TagMsgListener
package com.powernode.listener;
import org.apache.rocketmq.spring.annotation.RocketMQMessageListener;
import org.apache.rocketmq.spring.annotation.SelectorType;
import org.apache.rocketmq.spring.core.RocketMQListener;
import org.springframework.stereotype.Component;
/** * @Author 武汉动力节点 * 创建一个简单的标签消息的监听
* 1.类上添加注解@Component和@RocketMQMessageListener * selectorType = SelectorType.TAG, 指定使用tag过滤。(也可以使用sql92 需要在配置文件broker.conf中开启enbalePropertyFilter=true) * selectorExpression = "java" 表达式，默认是*,支持"tag1 || tag2 || tag3"
* 2.实现RocketMQListener接口，注意泛型的使用 */
@Component@RocketMQMessageListener(topic = "powernode-tag",
consumerGroup = "powernode-tag-group",
selectorType = SelectorType._TAG_,
selectorExpression = "java")
public class TagMsgListener implements RocketMQListener<String> {
/** * 消费消息的方法 *
* @param message */
@Override public void onMessage(String message) {
System._out_.println(message);
}}
25.1.4 Restart rocketmq-consumer to view the console
25.2 Key filtering (can be distinguished in the class of transaction monitoring)
25.2.1 Modify rocketmq-producer to add a unit test
/** * 发送一个带key的消息,我们使用事务消息 打断点查看消息头 * * @throws Exception */
@Testpublic void testKeyMsg() throws Exception {
// 发送一个key为spring的事务消息
Message<String> message = MessageBuilder._withPayload_("我是一个带key的消息") **.setHeader(RocketMQHeaders._KEYS_, "spring")**
.build();
rocketMQTemplate.sendMessageInTransaction("powernode", message, "我是一个带key的消息");
}
25.2.2 Breakpoint to send this message, check the message header in the transaction
We can also see it in the mq console
26. RocketMQ integrates two modes of SpringBoot message consumption
There are two modes of Rocketmq message consumption: load balancing mode and broadcast mode.
Load balancing mode means that multiple consumers alternately consume messages in the same topic.
Broadcast mode means that each consumer consumes messages of subscribed topics once.
26.1 Build another consumer rocketmq-consumer-b, the dependencies and configuration files are consistent with rocketmq-consumer, remember to modify the port to avoid occupation
26.2 rocketmq-consumer-b add a listener
package com.powernode.listener;
import org.apache.rocketmq.spring.annotation.MessageModel;
import org.apache.rocketmq.spring.annotation.RocketMQMessageListener;
import org.apache.rocketmq.spring.core.RocketMQListener;
import org.springframework.stereotype.Component;
/** * messageModel 指定消息消费的模式 * CLUSTERING 为负载均衡模式 * BROADCASTING 为广播模式 */
@Component@RocketMQMessageListener(topic = "powernode",
consumerGroup = "powernode-group",** **
**messageModel = MessageModel._CLUSTERING_**)
public class ConsumerBListener
implements RocketMQListener<String> {
@Override public void onMessage(String message) {
System._out_.println(message);
}}
26.3 Modify the SimpleMsgListener of rocketmq-consumer
_/** * 创建一个简单消息的监听 * 1.类上添加注解@Component和@RocketMQMessageListener * * @RocketMQMessageListener(topic = "powernode", consumerGroup = "powernode-group") * topic指定消费的主题，consumerGroup指定消费组,一个主题可以有多个消费者组,一个消息可以被多个不同的组的消费者都消费 * 2.实现RocketMQListener接口，注意泛型的使用 */_
@Component@RocketMQMessageListener(topic = "powernode",
consumerGroup = "powernode-group", ** **
**messageModel = MessageModel._CLUSTERING_**)
public class SimpleMsgListener
implements RocketMQListener<String> {
@Override public void onMessage(String message) {
System._out_.println(new Date());
System._out_.println(message);
}}
26.4 Starting two consumers
26.5 Add a unit test to the producer and run it
| /** * Test message consumption mode * * @throws Exception */
@Testpublic void testMsgModel() throws Exception { for (int i = 0; i < 10; i++) { rocketMQTemplate.syncSend("powernode", "I is message" + i);
| }} |
|---|
26.6 View the consoles of two consumers and find that it is a load balancing mode
26.7 Modify the mode of two consumers to BROADCASTING
Restart the test, the result is the broadcast mode, and each consumer consumes these messages
Generally, polymorphic machine consumers are deployed in the project 2-3 According to the business, you can choose a specific mode to configure
Reset the consumption point, set the consumption node of a group to a certain point in time before consumption from this point in time
Skip accumulation Select a group to skip accumulation After that, everything in this group will not be consumed
27. How to solve the message accumulation problem?
It is generally believed that when the difference of a single queue message is >= 10w, it is considered an accumulation problem
27.1 Under what circumstances will accumulation occur
- The production is too fast.
The producer can limit the flow of business
and increase the number of consumers, but the number of consumers <= the number of queues, and appropriately set the maximum number of consumer threads (according to IO(2n)/CPU(n+1)) to dynamically expand the
queue quantity, thereby increasing the number of consumers - There is a problem with consumer consumption
Troubleshoot consumer program problems
28. How to ensure that messages are not lost?
The producer uses the synchronous sending mode, and writes msgId status(0) time to its own database by the way after receiving the mq return confirmation- The status of the message after the consumer modifies the data after consumption = 1
- Write a timed task to query data every two days if there is status = 0 and time < day-2
- Set mq's brushing mechanism to synchronous brushing
- Use cluster mode, engage in active and standby mode, and persist messages on different hardware
- MQ trace mechanism and message tracking mechanism can be enabled
1. Enable message tracking traceTopicEnable=true in broker.conf
2. Restart the broker. 3. Enable
the message trace in the producer configuration file
enable-msg-trace: true
4. Enable the message trace function for the consumer, which can be given to a single Consumers enable
enableMsgTrace = true
You can view the message trace in the rocketmq panel.
By default, the data of the message trace will be stored in the topic RMQ_SYS_TRACE_TOPIC
29. Security
- Turn on the control of acl and turn on aclEnable=true in broker.conf
- Configure account password and modify plain_acl.yml
- Modify the configuration file of the control panel, release line 52/53, change line 49 to true, and upload it to the jar package level directory of the server.