1 Introduction to MQ
2 Why use MQ
Message queue is a "first in, first out" data structure,
and its application scenarios mainly include the following three aspects
2.1 Application decoupling
The more coupled a system is, the less fault-tolerant it is. Taking an e-commerce application as an example, after a user creates an order, if the inventory system, logistics system, and payment system are coupled and called, if any subsystem fails or is temporarily unavailable due to upgrades, etc., it will cause an abnormal order operation and affect the user's use. experience.
Using message queue decoupling, the coupling of the system will be improved. For example, if the logistics system breaks down, it takes a few minutes to repair. During this time, the data to be processed by the logistics system is cached in the message queue, and the user's order operation is completed normally. After the logistics system responds, it only needs to supplement and process the order messages stored in the message queue, and the terminal system will not be aware of the failure of the logistics system for a few minutes.
2.2 Flow clipping
If the application system encounters a sudden surge in system request traffic, it may overwhelm the system. With the message queue, a large number of requests can be cached and processed over a long period of time, which can greatly improve the stability of the system and the user experience.
In general, in order to ensure the stability of the system, if the system load exceeds the threshold, user requests will be blocked, which will affect the user experience. If you use a message queue to cache the request and wait for the system to complete the processing, the user will be notified that the order is completed. The experience of placing an order can never be better.
For the purpose of economic considerations:
if the QPS of the business system is 1000 during normal hours, and the peak traffic is 10000, it is obviously not cost-effective to configure a high-performance server to cope with the peak traffic. At this time, you can use the message queue to cut the peak traffic
2.3 Data distribution
Through the message queue, data can be circulated among multiple systems. The data generator does not need to care about who will use the data, it only needs to send the data to the message queue, and the data user can directly obtain the data in the message queue
3 Advantages and disadvantages of MQ
Advantages: decoupling, peak clipping, and data distribution.
Disadvantages include the following:
reduced system availability
. The more external dependencies the system introduces, the worse the system stability will be. Once MQ goes down, it will affect the business.
How to ensure the high availability of MQ?
Increased system complexity The
addition of MQ has greatly increased the complexity of the system. In the past, there were synchronous remote calls between systems, but now asynchronous calls are made through MQ.
How to ensure that messages are not consumed repeatedly? How to deal with message loss? So to ensure the order of message delivery?
Consistency problem
System A finishes processing the business, and sends message data to the three systems B, C, and D through MQ. If the processing of systems B and C succeeds, the processing of system D fails.
How to ensure the consistency of message data processing?
4 Comparison of various MQ products
Common MQ products include Kafka, ActiveMQ, RabbitMQ, RocketMQ.
5 RocketMQ quick start
RocketMQ is Alibaba's 2016 MQ middleware, which is developed in Java language. Inside Alibaba, RocketMQ undertakes message flow in high-concurrency scenarios such as "Double 11" and can handle trillion-level messages.
5.1 Preparations
Download RocketMQ
The version of RocketMQ selected here: 4.6.0
Download link: Download link
Official document: http://rocketmq.apache.org/docs/quick-start/
Environment requirements
Linux64-bit system
JDK1.8 (64-bit)
5.2 Install RocketMQ
Installation steps
I installed it as a binary package here:
unzip the installation package
and enter the installation directory Directory
introduction
bin: startup script, including shell script and CMD script
conf: instance configuration file, including broker configuration file, logback configuration file, etc.
lib: dependency jar package, including Netty, commons-lang, FastJSON, etc.
Start RocketMQ
The default virtual machine memory of RocketMQ is large. Starting Broker or NameServer may fail due to insufficient memory, so you need to edit the following two configuration files to modify the JVM memory size. Edit runbroker.sh and runserver.sh to modify the default JVM
size
$ vi bin/runbroker.sh
# reference setting
JAVA_OPT="${JAVA_OPT} -server -Xms256m -Xmx256m -Xmn128m"
$ vi bin/runserver.sh
# reference setting
JAVA_OPT="${JAVA_OPT} -server -Xms256m -Xmx256m -Xmn128m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m”
Start NameServer
Start NameServer nohup sh bin/mqnamesrv &
view startup log tail -f ~/logs/rocketmqlogs/namesrv.log
Start Broker
Start Broker nohup sh bin/mqbroker -n localhost:9876 &
view startup log tail -f ~/logs/rocketmqlogs/broker.log
Some optional parameters of bin/mqbroker:
-c: specifies the configuration file path
-n: the address of the NameServer
5.3 Testing RocketMQ
Send a message
Set the environment variable export NAMESRV_ADDR=localhost:9876 #Use
the Demo of the installation package to send a message sh bin/tools.sh org.apache.rocketmq.example.quickstart.Producer
Receive message
Set environment variable export NAMESRV_ADDR=localhost:9876
Receive message sh bin/tools.sh org.apache.rocketmq.example.quickstart.Consumer
Close RocketMQ
Close NameServer sh bin/mqshutdown namesrv
Close Broker sh bin/mqshutdown broker
Introduction of each role
Producer: sender of the message; example: sender
Consumer: message receiver; example: recipient
Consumer Group: consumer group; each consumer instance belongs to a consumer group, and each message will only be received by the same A consumer instance in the consumer group consumes. (Different consumer groups can consume the same message at the same time)
Broker: Temporarily store and transmit messages; Example: Express Company
NameServer: Manage Broker; Example: Express Company Management Organization
Topic: Differentiate the types of messages; one sender can send a message to one Or multiple Topics; the receiver of a message can subscribe to one or more Topic messages
Message Queue: Equivalent to a Topic partition; used to send and receive messages in parallel
Detailed explanation of broker configuration file
The default configuration file location of broker is: conf/broker.conf
#belonging cluster name
brokerClusterName=rocketmq-cluster
#broker name, note that different configuration files are filled in differently here
brokerName=broker-a
#0 means Master ,>0 means Slave
brokerId=0
#nameServer address, separated by semicolon
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876
#When sending a message, automatically create a topic that does not exist in the server, and the number of queues created by default
defaultTopicQueueNums=4
#Whether to allow Broker to automatically create Topic, it is recommended to open it offline, and close it online
autoCreateTopicEnable=true
#Whether to allow Broker to automatically create a subscription group, it is recommended to open it offline, and close it online
autoCreateSubscriptionGroup=true
#Broker external service listening port
listenPort=10911
# Delete file time point, the default is 4 o'clock in the morning
deleteWhen=04
#File retention time, the default is 48 hours
fileReservedTime=120
#commitLog The size of each file defaults to 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue stores 30W items by default for each file, adjust
mappedFileSizeConsumeQueue=300000 according to business conditions
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#Detect physical file disk space
diskMaxUsedSpaceR atio=88
#store path
storePathRootDir =/usr/local/rocketmq/store
#commitLog storage path
storePathCommitLog=/usr/local/rocketmq/store/commitlog
#consumption queue storage path storage path
storePathConsumeQueue=/usr/local/rocketmq/store/consumequeue
#message index storage path
storePathIndex =/usr/local/rocketmq/store/index
#checkpoint file storage path
storeCheckpoint=/usr/local/rocketmq/store/checkpoint
#abort file storage path
abortFile=/usr/local/rocketmq/store/abort
#Limited message size
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000
#Broker role
#- ASYNC_MASTER Asynchronous replication Master
#- SYNC_MASTER synchronization Double-write Master
#- SLAVE
brokerRole=SYNC_MASTER
#Flush disk mode
#- ASYNC_FLUSH asynchronous flush disk
#- SYNC_FLUSH synchronous flush disk
flushDiskType=SYNC_FLUSH
#checkTransactionMessageEnable=false
#Send message thread pool number
#sendMessageThreadPoolNums=128
#Pull message thread pool number
#pullMessageThreadPoolN ums =128
6 Visual monitoring platform construction
6.1 Overview
RocketMQ has an extended open source project incubator-rocketmq-externals. There is a submodule in this project called rocketmq-console. This is the management console project. First, pull incubator-rocketmq-externals locally, because we need Compile, package and run rocketmq-console by yourself.
6.2 Download and compile and package
Clone project
git clone https://github.com/apache/rocketmq-externals
Configure namesrv cluster address in rocketmq-console:
$ cd rocketmq-console $ vim src/main/resources/application.properties rocketmq.config.namesrvAddr=10.211 .55.4:9876
After the configuration is complete, compile and package
mvn clean package -Dmaven.test.skip=true
to start rocketmq-console:
nohup java -jar rocketmq-console-ng-2.0.0.jar > tmp.log &
after successful startup, We can access http://IP address: 8080 through the browser to enter the console interface, as shown below:
7 Message sending and consumption example (Maven)
Import MQ client dependencies
Note: The version of rocketmq-client should be consistent with the version of RocketMQ
<dependency>
<groupId>org.apache.rocketmq</groupId>
<artifactId>rocketmq-client</artifactId>
<version>4.6.0</version>
</dependency>
Message sender step analysis:
Create a message producer, and specify the producer group name
Specify the Nameserver address
Start the producer
to create a message object, specify the topic Topic, Tag and message body
Send a message
Close the producer producer
message consumer step analysis:
Create a consumer, specify the consumer group name,
specify the Nameserver address
, subscribe to the topic Topic and Tag
, set the callback function, and process the message.
Start the consumer
8 basic samples
8.1 Message sending, sending synchronous messages
This reliable and synchronous sending method is widely used, such as: important message notification, SMS notification
public class SyncProducer {
public static void main(String[] args) throws Exception {
// 实例化消息生产者Producer
DefaultMQProducer producer = new DefaultMQProducer("please_rename_unique_group_name");
// 设置NameServer的地址
producer.setNamesrvAddr("localhost:9876");
// 设置消息同步发送失败时的重试次数，默认为 2
producer.setRetryTimesWhenSendFailed(2);
// 设置消息发送超时时间，默认3000ms
producer.setSendMsgTimeout(3000);
// 启动Producer实例
producer.start();
for (int i = 0; i < 100; i++) {
// 创建消息，并指定Topic，Tag和消息体
Message msg = new Message("TopicTest" /* Topic */,
"TagA" /* Tag */,
("Hello RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET) /* Message
body */
);
// 发送消息到一个Broker
SendResult sendResult = producer.send(msg);
// 通过sendResult返回消息是否成功送达
System.out.printf("%s%n", sendResult);
}
// 如果不再发送消息，关闭Producer实例。
producer.shutdown();
}
}
8.3 Sending asynchronous messages
Asynchronous messages are usually used in business scenarios that are sensitive to response time, that is, the sender cannot tolerate waiting for a long time for the Broker's response.
```java
public class AsyncProducer {
public static void main(String[] args) throws Exception {
// 实例化消息生产者Producer
DefaultMQProducer producer = new DefaultMQProducer("please_rename_unique_group_name");
// 设置NameServer的地址
producer.setNamesrvAddr("localhost:9876");
// 设置消息异步发送失败时的重试次数，默认为 2
producer.setRetryTimesWhenSendAsyncFailed(2);
// 设置消息发送超时时间，默认3000ms
producer.setSendMsgTimeout(3000);
// 启动Producer实例
producer.start();
producer.setRetryTimesWhenSendAsyncFailed(0);
for (int i = 0; i < 100; i++) {
final int index = i;
// 创建消息，并指定Topic，Tag和消息体
Message msg = new Message("TopicTest",
"TagA",
"OrderID188",
"Hello world".getBytes(RemotingHelper.DEFAULT_CHARSET));
// SendCallback接收异步返回结果的回调
producer.send(msg, new SendCallback() {
@Override
public void onSuccess(SendResult sendResult) {
System.out.printf("%-10d OK %s %n", index,
sendResult.getMsgId());
}
@Override
public void onException(Throwable e) {
System.out.printf("%-10d Exception %s %n", index, e);
e.printStackTrace();
}
});
}
// 如果不再发送消息，关闭Producer实例。
producer.shutdown();
}
}
8.4 Sending messages in one direction
This method is mainly used in scenarios where the sending result is not particularly concerned, such as log sending.
public class OnewayProducer {
public static void main(String[] args) throws Exception{
// 实例化消息生产者Producer
DefaultMQProducer producer = new DefaultMQProducer("please_rename_unique_group_name");
// 设置NameServer的地址
producer.setNamesrvAddr("localhost:9876");
// 启动Producer实例
producer.start();
for (int i = 0; i < 100; i++) {
// 创建消息，并指定Topic，Tag和消息体
Message msg = new Message("TopicTest" /* Topic */,
"TagA" /* Tag */,
("Hello RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET) /* Message body */
);
// 发送单向消息，没有任何返回结果
producer.sendOneway(msg);
}
// 如果不再发送消息，关闭Producer实例。
producer.shutdown();
}
}
9 Consumption News
9.1 Cluster mode (load balancing)
Consumers consume messages in a cluster mode, and only one consumer in the same consumer group will consume a message
public static void main(String[] args) throws Exception {
// 实例化消息生产者,指定组名
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("group1");
// 指定Namesrv地址信息.
consumer.setNamesrvAddr("localhost:9876");
// 订阅Topic
consumer.subscribe("Test", "*");
//负载均衡模式消费
consumer.setMessageModel(MessageModel.CLUSTERING);
// 注册回调函数，处理消息
consumer.registerMessageListener(new MessageListenerConcurrently() {
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs,
ConsumeConcurrentlyContext context) {
System.out.printf("%s Receive New Messages: %s %n",
Thread.currentThread().getName(), msgs);
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
});
//启动消息者
consumer.start();
System.out.printf("Consumer Started.%n");
}
9.2 Broadcast Mode
Consumers consume messages by broadcasting, and each consumer in the same consumer group must consume a message
public static void main(String[] args) throws Exception {
// 实例化消息生产者,指定组名
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("group1");
// 指定Namesrv地址信息.
consumer.setNamesrvAddr("localhost:9876");
// 订阅Topic
consumer.subscribe("Test", "*");
//广播模式消费
consumer.setMessageModel(MessageModel.BROADCASTING);
// 注册回调函数，处理消息
consumer.registerMessageListener(new MessageListenerConcurrently() {
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt>
msgs,
ConsumeConcurrentlyContext context) {
System.out.printf("%s Receive New Messages: %s %n",
Thread.currentThread().getName(), msgs);
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
});
//启动消息者
consumer.start();
System.out.printf("Consumer Started.%n");
}
9.3 Sequential messages
Message ordering means that messages can be consumed in the order in which they are sent (FIFO). RocketMQ can strictly guarantee the order of messages, which can be divided into partition order or global order.
Analysis of the principle of sequential consumption. By default, the message sending will adopt the Round Robin polling method to send the message to different queues (partition queues); when consuming the message, the message is pulled from multiple queues. In this case, the message is sent And consumption is not guaranteed order. However, if the sequential messages sent by the control are only sent to the same queue in sequence, and only pulled from this queue in sequence when consuming, the sequence is guaranteed. When there is only one queue participating in sending and consuming, it is globally ordered; if multiple queues participate, it is partitioned ordered, that is, messages are ordered relative to each queue.
The following is an example of partition ordering with order. The sequence process of an order is: create, pay, push, complete. Messages with the same order number will be sent to the same queue one after another. When consuming, the same OrderId must get the same queue.
sequential message production
/**
* Producer，发送顺序消息
*/
public class Producer {
public static void main(String[] args) throws Exception {
DefaultMQProducer producer = new DefaultMQProducer("please_rename_unique_group_name");
producer.setNamesrvAddr("127.0.0.1:9876")
producer.start();
String[] tags = new String[]{
"TagA", "TagC", "TagD"};
// 订单列表
List<OrderStep> orderList = new Producer().buildOrders();
Date date = new Date();
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
String dateStr = sdf.format(date);
for (int i = 0; i < 10; i++) {
// 加个时间前缀
String body = dateStr + " Hello RocketMQ " + orderList.get(i);
Message msg = new Message("TopicTest", tags[i % tags.length], "KEY" + i, body.getBytes());
SendResult sendResult = producer.send(msg, new MessageQueueSelector() {
@Override
public MessageQueue select(List<MessageQueue> mqs, Message msg, Object arg) {
Long id = (Long) arg; //根据订单id选择发送queue
long index = id % mqs.size();
return mqs.get((int) index);
}
}, orderList.get(i).getOrderId());//订单id
System.out.println(String.format("SendResult status:%s, queueId:%d, body:%s",
sendResult.getSendStatus(),
sendResult.getMessageQueue().getQueueId(),
body));
}
producer.shutdown();
}
/**
* 订单的步骤
*/
private static class OrderStep {
private long orderId;
private String desc;
public long getOrderId() {
return orderId;
}
public void setOrderId(long orderId) {
this.orderId = orderId;
}
public String getDesc() {
return desc;
}
public void setDesc(String desc) {
this.desc = desc;
}
@Override
public String toString() {
return "OrderStep{" +
"orderId=" + orderId +
", desc='" + desc + '\'' +
'}';
}
}
/**
* 生成模拟订单数据
*/
private List<OrderStep> buildOrders() {
List<OrderStep> orderList = new ArrayList<OrderStep>();
OrderStep orderDemo = new OrderStep();
orderDemo.setOrderId(15103111039L);
orderDemo.setDesc("创建");
orderList.add(orderDemo);
orderDemo = new OrderStep();
orderDemo.setOrderId(15103111065L);
orderDemo.setDesc("创建");
orderList.add(orderDemo);
orderDemo = new OrderStep();
orderDemo.setOrderId(15103111039L);
orderDemo.setDesc("付款");
orderList.add(orderDemo);
orderDemo = new OrderStep();
orderDemo.setOrderId(15103117235L);
orderDemo.setDesc("创建");
orderList.add(orderDemo);
orderDemo = new OrderStep();
orderDemo.setOrderId(15103111065L);
orderDemo.setDesc("付款");
orderList.add(orderDemo);
orderDemo = new OrderStep();
orderDemo.setOrderId(15103117235L);
orderDemo.setDesc("付款");
orderList.add(orderDemo);
orderDemo = new OrderStep();
orderDemo.setOrderId(15103111065L);
orderDemo.setDesc("完成");
orderList.add(orderDemo);
orderDemo = new OrderStep();
orderDemo.setOrderId(15103111039L);
orderDemo.setDesc("推送");
orderList.add(orderDemo);
orderDemo = new OrderStep();
orderDemo.setOrderId(15103117235L);
orderDemo.setDesc("完成");
orderList.add(orderDemo);
orderDemo = new OrderStep();
orderDemo.setOrderId(15103111039L);
orderDemo.setDesc("完成");
orderList.add(orderDemo);
return orderList;
}
}
Consume messages sequentially
/**
* 顺序消息消费，带事务方式（应用可控制Offset什么时候提交）
*/
public class ConsumerInOrder {
public static void main(String[] args) throws Exception {
DefaultMQPushConsumer consumer = new
DefaultMQPushConsumer("please_rename_unique_group_name_3");
consumer.setNamesrvAddr("127.0.0.1:9876");
/**
* 设置Consumer第一次启动是从队列头部开始消费还是队列尾部开始消费<br>
* 如果非第一次启动，那么按照上次消费的位置继续消费
*/
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET);
consumer.subscribe("TopicTest", "TagA || TagC || TagD");
consumer.registerMessageListener(new MessageListenerOrderly() {
Random random = new Random();
@Override
public ConsumeOrderlyStatus consumeMessage(List<MessageExt> msgs, ConsumeOrderlyContext context) {
context.setAutoCommit(true);
for (MessageExt msg : msgs) {
// 可以看到每个queue有唯一的consume线程来消费, 订单对每个queue(分区)有序
System.out.println("consumeThread=" + Thread.currentThread().getName() + "queueId=" +
msg.getQueueId() + ", content:" + new String(msg.getBody()));
}
try {
//模拟业务逻辑处理中...
TimeUnit.SECONDS.sleep(random.nextInt(10));
} catch (Exception e) {
e.printStackTrace();
}
return ConsumeOrderlyStatus.SUCCESS;
}
});
consumer.start();
System.out.println("Consumer Started.");
}
}
9.4 Delayed messages
For example, in e-commerce, once an order is submitted, a delayed message can be sent, and the status of the order will be checked after 1 hour. If the payment is still not made, the order will be canceled to release the inventory.
Start message consumer
public class ScheduledMessageConsumer {
public static void main(String[] args) throws Exception {
// 实例化消费者
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("ExampleConsumer");
// 订阅Topics
consumer.subscribe("TestTopic", "*");
// 注册消息监听者
consumer.registerMessageListener(new MessageListenerConcurrently() {
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> messages, ConsumeConcurrentlyContext
context) {
for (MessageExt message : messages) {
// Print approximate delay time period
System.out.println("Receive message[msgId=" + message.getMsgId() + "] " +
(System.currentTimeMillis() - message.getStoreTimestamp()) + "ms later");
}
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
});
// 启动消费者
consumer.start();
}
}
9.5 Send delayed message
public class ScheduledMessageProducer {
public static void main(String[] args) throws Exception {
// 实例化一个生产者来产生延时消息
DefaultMQProducer producer = new DefaultMQProducer("ExampleProducerGroup");
// 启动生产者
producer.start();
int totalMessagesToSend = 100;
for (int i = 0; i < totalMessagesToSend; i++) {
Message message = new Message("TestTopic", ("Hello scheduled message " + i).getBytes());
// 设置延时等级3,这个消息将在10s之后发送(现在只支持固定的几个时间,详看delayTimeLevel)
message.setDelayTimeLevel(3);
// 发送消息
producer.send(message);
}
// 关闭生产者
producer.shutdown();
}
}
Verify that
you will see that messages are consumed 10 seconds later than stored
using limits
// org/apache/rocketmq/store/config/MessageStoreConfig.java
private String messageDelayLevel="1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h”;
now RocketMq does not support any time delay, you need to set several fixed delay levels, from 1s to 2h corresponding to levels 1 to 18
9.6 Bulk messages
Sending messages in batches can significantly improve the performance of delivering small messages. The restriction is that these bulk messages should have the same topic, the same waitStoreMsgOK, and cannot be delayed messages. Also, the total size of the batch of messages should not exceed 4MB.
9.7 Send bulk messages
Batching is easy to use if you only send messages no larger than 4MB at a time, as in the following example:
String topic = "BatchTest";
List<Message> messages = new ArrayList<>();
messages.add(new Message(topic, "TagA", "OrderID001", "Hello world 0".getBytes()));
messages.add(new Message(topic, "TagA", "OrderID002", "Hello world 1".getBytes()));
messages.add(new Message(topic, "TagA", "OrderID003", "Hello world 2".getBytes()));
try {
producer.send(messages);
} catch (Exception e) {
e.printStackTrace();
//处理error
}
If the total length of the message may be greater than 4MB, it is best to divide the message at this time
public class ListSplitter implements Iterator<List<Message>> {
private final int SIZE_LIMIT = 1024 * 1024 * 4;
private final List<Message> messages;
private int currIndex;
public ListSplitter(List<Message> messages) {
this.messages = messages;
}
@Override
public boolean hasNext() {
return currIndex < messages.size();
}
@Override
public List<Message> next() {
int nextIndex = currIndex;
int totalSize = 0;
for (; nextIndex < messages.size(); nextIndex++) {
Message message = messages.get(nextIndex);
int tmpSize = message.getTopic().length() + message.getBody().length;
Map<String, String> properties = message.getProperties();
for (Map.Entry<String, String> entry : properties.entrySet()) {
tmpSize += entry.getKey().length() + entry.getValue().length();
}
tmpSize = tmpSize + 20; // 增加日志的开销20字节
if (tmpSize > SIZE_LIMIT) {
//单个消息超过了最大的限制
//忽略,否则会阻塞分裂的进程
if (nextIndex - currIndex == 0) {
//假如下一个子列表没有元素,则添加这个子列表然后退出循环,否则只是退出循环
nextIndex++;
}
break;
}
if (tmpSize + totalSize > SIZE_LIMIT) {
break;
} else {
totalSize += tmpSize;
}
}
List<Message> subList = messages.subList(currIndex, nextIndex);
currIndex = nextIndex;
return subList;
}
}
//把大的消息分裂成若干个小的消息
ListSplitter splitter = new ListSplitter(messages);
while (splitter.hasNext()) {
try {
List<Message> listItem = splitter.next();
producer.send(listItem);
} catch (Exception e) {
e.printStackTrace();
//处理error
}
}
9.8 Filter messages
In most cases, TAG is a simple and useful design to select the message you want. For example:
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(“CID_EXAMPLE”); consumer.subscribe(“TOPIC”, “TAGA || TAGB || TAGC”);
Consumers will receive messages containing TAGA or TAGB or TAGC. But the limitation is that a message can only have one tag, which may not work for complex scenarios. In this case, SQL expressions can be used to filter messages. SQL properties can be computed by attributes when sending messages. Under the syntax defined by RocketMQ, some simple logic can be implemented. Below is an example:
| message |
|----------| a > 5 AND b = ‘abc’
| a = 10 | --------------------> Gotten
| b = ‘abc’|
| c = true |
|---|
| message |
|----------| a > 5 AND b = ‘abc’
| a = 1 | --------------------> Missed
| b = ‘abc’|
| c = true |
|---|
9.9 Basic syntax of SQL
RocketMQ only defines some basic syntax to support this feature. You can also easily extend it.
Numerical comparison, such as: >, >=, <, <=, BETWEEN, =
Character comparison, such as: =, <>, IN
IS NULL or IS NOT NULL
Logical symbols AND, OR, NOT
constant support types are:
Values, such as: 123, 3.1415
characters, such as: 'abc', must be wrapped in single quotes
NULL, special constant
Boolean value, TRUE or FALSE
Only consumers using the push mode can use SQL92 standard sql statements, the interface is as follows :
public void subscribe(finalString topic, final MessageSelector messageSelector)
9.10 Message producers
When sending a message, you can set the property of the message through putUserProperty
DefaultMQProducer producer = new DefaultMQProducer("please_rename_unique_group_name");
producer.start();
Message msg = new Message("TopicTest",
tag,
("Hello RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET)
);
// 设置一些属性
msg.putUserProperty("a", String.valueOf(i));
SendResult sendResult = producer.send(msg);
producer.shutdown();
9.11 Message Consumers
Use MessageSelector.bySql to filter messages using sql
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("please_rename_unique_group_name_4");
// 只有订阅的消息有这个属性a, a >=0 and a <= 3
consumer.subscribe("TopicTest", MessageSelector.bySql("a between 0 and 3");
consumer.registerMessageListener(new MessageListenerConcurrently() {
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
});
consumer.start();
10 business messages
10.1 Process Analysis
The general scheme of the transaction message, which is divided into two processes: the sending and submission of the normal transaction message, and the compensation process of the transaction message.
1) Transaction message sending and submission
(1) Send a message (half message).
(2) The server responds to the message writing result.
(3) Execute the local transaction according to the sending result (if the writing fails, the half message is not visible to the business at this time, and the local logic is not executed).
(4) Execute Commit or Rollback according to the local transaction state (the Commit operation generates a message index, and the message is visible to consumers)
2) Transaction compensation
(1) For transaction messages without Commit/Rollback (messages in pending status), initiate a "checkback" from the server (2)
Producer receives the checkback message and checks the local transaction corresponding to the checkback message State
(3) According to the state of the local transaction, re-commit or rollback
. The compensation stage is used to solve the timeout or failure of the message commit or rollback.
3) Transaction message status
There are three statuses for transaction messages, commit status, rollback status, and intermediate status:
TransactionStatus.CommitTransaction: Commit the transaction, which allows consumers to consume this message.
TransactionStatus.RollbackTransaction: Rollback transaction, which means that the message will be deleted and not allowed to be consumed.
TransactionStatus.Unknown: Intermediate status, which represents the need to check the message queue to determine the status.
10.2 Send transaction message
- Create a transactional producer
Using the TransactionMQProducer class to create a producer and specify a unique ProducerGroup, you can set up a custom thread pool to handle these inspection requests. After executing a local transaction, it is necessary to reply to the message queue according to the execution result. Please refer to the previous section for the returned transaction status.
public class Producer {
public static void main(String[] args) throws MQClientException, InterruptedException {
//创建事务监听器
TransactionListener transactionListener = new TransactionListenerImpl();
//创建消息生产者
TransactionMQProducer producer = new TransactionMQProducer("group6");
producer.setNamesrvAddr("192.168.25.135:9876;192.168.25.138:9876");
//生产者这是监听器
producer.setTransactionListener(transactionListener);
//启动消息生产者
producer.start();
String[] tags = new String[]{
"TagA", "TagB", "TagC"};
for (int i = 0; i < 3; i++) {
try {
Message msg = new Message("TransactionTopic", tags[i % tags.length], "KEY" + i,
("Hello RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET));
SendResult sendResult = producer.sendMessageInTransaction(msg, null);
System.out.printf("%s%n", sendResult);
TimeUnit.SECONDS.sleep(1);
} catch (MQClientException | UnsupportedEncodingException e) {
e.printStackTrace();
}
}
//producer.shutdown();
}
}
2) Realize the monitoring interface of the transaction.
When the half-message is successfully sent, we use the executeLocalTransaction method to execute the local transaction. It returns one of the three transaction states mentioned in the previous section. The checkLocalTranscation method is used to check the local transaction status and respond to the check request of the message queue. It also returns one of the three transaction states mentioned in the previous section.
public class TransactionListenerImpl implements TransactionListener {
@Override
public LocalTransactionState executeLocalTransaction(Message msg, Object arg) {
System.out.println("执行本地事务");
if (StringUtils.equals("TagA", msg.getTags())) {
return LocalTransactionState.COMMIT_MESSAGE;
} else if (StringUtils.equals("TagB", msg.getTags())) {
return LocalTransactionState.ROLLBACK_MESSAGE;
} else {
return LocalTransactionState.UNKNOW;
}
}
@Override
public LocalTransactionState checkLocalTransaction(MessageExt msg) {
System.out.println("MQ检查消息Tag【"+msg.getTags()+"】的本地事务执行结果");
return LocalTransactionState.COMMIT_MESSAGE;
}
}
Usage restrictions
Transaction messages do not support delayed messages and batch messages.
In order to avoid the accumulation of semi-queue messages caused by a single message being checked too many times, we limit the number of checks for a single message to 15 times by default, but users can modify this limit through the transactionCheckMax parameter of the Broker configuration file. If a message has been checked more than N times (N = transactionCheckMax), Broker will discard the message and print the error log at the same time by default. Users can modify this behavior by overriding the AbstractTransactionCheckListener class.
Transaction messages will be checked after a certain amount of time such as the parameter transactionMsgTimeout in the Broker configuration file. When sending a transaction message, the user can also change this limit by setting the user attribute CHECK_IMMUNITY_TIME_IN_SECONDS, which takes precedence over the transactionMsgTimeout parameter.
Transactional messages may be inspected or consumed more than once.
Submitting messages to the user's target topic may fail, currently subject to logging. Its high availability is guaranteed by RocketMQ's own high availability mechanism
Producer IDs for transactional messages cannot be shared with producer IDs for other types of messages. Unlike other types of messages, transactional messages allow reverse queries, and MQ servers can query consumers through their producer IDs.
10.3 Configure AK and Secret when connecting to Alibaba Cloud RocketMQ
If you are calling Alibaba Cloud's RocketMQ, you also need to specify the AK and Secret.
Producer
The producer sets the AK and the Secert operation are the same, you only need to specify it when creating the Producer, here is an example of sending a normal message:
public class SyncAKProducer {
private static RPCHook getAclRPCHook() {
return new AclClientRPCHook(new SessionCredentials("设置自己的ACCESS_KEY", "设置自己的SECRET_KEY"));
}
public static void main(String[] args) throws Exception {
/**
* 创建Producer，并开启消息轨迹
* 如果不想开启消息轨迹，可以按照如下方式创建：
* DefaultMQProducer producer = new DefaultMQProducer(M"设置自己的GroupName（唯一）", getAclRPCHook());
*/
DefaultMQProducer producer = new DefaultMQProducer("设置自己的GroupName（唯一）", getAclRPCHook(), true, null);
/**
* 设置使用接入方式为阿里云，在使用云上消息轨迹的时候，需要设置此项，如果不开启消息轨迹功能，则运行不设置此项.
*/
producer.setAccessChannel(AccessChannel.CLOUD);
// 设置NameServer的地址
producer.setNamesrvAddr("localhost:9876");
// 设置消息同步发送失败时的重试次数，默认为 2
producer.setRetryTimesWhenSendFailed(2);
// 设置消息发送超时时间，默认3000ms
producer.setSendMsgTimeout(3000);
// 启动Producer实例
producer.start();
for (int i = 0; i < 100; i++) {
// 创建消息，并指定Topic，Tag和消息体
Message msg = new Message("TopicTest" /* Topic */,
"TagA" /* Tag */,
("Hello RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET) /* Message body */
);
// 发送消息到一个Broker
SendResult sendResult = producer.send(msg);
// 通过sendResult返回消息是否成功送达
System.out.printf("%s%n", sendResult);
}
// 如果不再发送消息，关闭Producer实例。
producer.shutdown();
}
}
10.4 Consumers
The operations of consumers setting AK and Secert are the same, only need to be specified when creating Consumer, here is an example of receiving ordinary messages:
private static RPCHook getAclRPCHook() {
return new AclClientRPCHook(new SessionCredentials("设置自己的ACCESS_KEY", "设置自己的SECRET_KEY"));
}
public static void main(String[] args) throws Exception {
/**
* 创建Consumer，并开启消息轨迹
* 如果不想开启消息轨迹，可以按照如下方式创建：
* DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(MqConfig.GROUP_ID, getAclRPCHook(), null);
*/
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("group1", getAclRPCHook(), new AllocateMessageQueueAveragely(), true, null);
/**
* 设置使用接入方式为阿里云，在使用云上消息轨迹的时候，需要设置此项，如果不开启消息轨迹功能，则运行不设置此项.
*/
consumer.setAccessChannel(AccessChannel.CLOUD);
// 指定Namesrv地址信息.
consumer.setNamesrvAddr("localhost:9876");
// 订阅Topic
consumer.subscribe("Test", "*");
//负载均衡模式消费
consumer.setMessageModel(MessageModel.CLUSTERING);
// 注册回调函数，处理消息
consumer.registerMessageListener(new MessageListenerConcurrently() {
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs,
ConsumeConcurrentlyContext context) {
System.out.printf("%s Receive New Messages: %s %n",
Thread.currentThread().getName(), msgs);
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
});
//启动消息者
consumer.start();
System.out.printf("Consumer Started.%n");
}
11 Message sending and consumption example (Spring Boot)
11.1 Import dependencies
<dependency>
<groupId>org.apache.rocketmq</groupId>
<artifactId>rocketmq-spring-boot-starter</artifactId>
<version>2.1.1</version>
</dependency>
11.2 Producers
application.yaml 配置文件
application.yaml
rocketmq:
name-server: 10.124.128.200:9876
producer:
group: test-group
# When sending a synchronous message fails, the number of retries, the default is 2
retry-times-when-send-failed: 2
# Send an asynchronous message When it fails, the number of retries, the default is 2
retry-times-when-send-async-failed: 2
# Send message timeout, the default is 3s
send-message-timeout: 3000
# 连接阿里云RocketMQ时需要配置AK与SK
access-key:
secret-key:
11.3 Code
@RestController
@RequestMapping("/test")
public class ProducerTest {
//自动注入
@Autowired
private RocketMQTemplate rocketMQTemplate;
@PostMapping("/sendSyncMessage")
public void sendSyncMessage(@RequestBody Map<String, Object> msgMap){
//构建消息
Message message = new Message("TopicName", "Tag", hash, JSON.toJSONBytes(msgData));
//发送同步消息
//方法1：使用与第三章相同的方法，调用 getProducer() 方法时会返回DefaultMQProducer对象，然后调用其方法第三章的一样了。
SendResult sendResult = rocketMQTemplate.getProducer().send(message);
//方法2：使用rocketMQTemplate封装的消息发送方法
// 第一个参数指定Topic与Tag，格式: `topicName:tags`
// 第二个参数，Message对象
sendResult = rocketMQTemplate.syncSend("TopicName:Tag", message);
}
}
11.4 Consumers
application.yaml 配置文件
rocketmq:
name-server: 10.124.128.200:9876
# 下面的配置只有在用阿里云的RocketMQ时，才配置，自己搭建的不需要配置
consumer:
access-key:
secret-key:
access-channel: CLOUD
11.5 Consumer message listener
@Slf4j
@Component
@RocketMQMessageListener(topic = "springboot-mq",
consumerGroup = "springboot-mq-consumer-1",
selectorExpression = "*")
public class Consumer implements RocketMQListener<String> {
@Override
public void onMessage(String message) {
log.info("Receive message：" + message);
//如果消费失败，则抛出RuntimeException，RocketMQ会自动重试
//可以手动抛出，也可以使用 Lombok 的 @SneakyThrows 注解来抛出 RuntimeException
throw new RuntimeException("消费失败");
}
}
@RocketMQMessageListener 注解的常用配置参数：
12 message storage
Because distributed queues have high reliability requirements, data must be stored persistently.
The message generator sends a message
to MQ, receives the message, persists the message, adds a new record in the storage,
and returns ACK to the producer
MQ pushes the message to the corresponding consumer, and then waits for the consumer to return ACK
. If the message consumer successfully returns ack within the specified time, then MQ considers the message consumption successful and deletes the message in the storage; if MQ does not receive the ACK within the specified time , it will be considered that the message consumption has failed, and it will try to push the message again
12.1 Storage media
At present, several products commonly used in the industry (RocketMQ/Kafka/RabbitMQ) all use message flashing to the file system of the deployed virtual machine/physical machine for persistence (the flashing can generally be divided into asynchronous flashing and synchronous flashing). disc two modes).
Message flushing provides a high-efficiency, high-reliability, and high-performance data persistence method for message storage. Unless the MQ machine itself is deployed or the local disk is hung up, generally there will be no failures that cannot be persisted.
12.2 Message storage structure
The storage of RocketMQ messages is completed by the cooperation of ConsumeQueue and CommitLog. The real physical storage file of the message is CommitLog. ConsumeQueue is the logical queue of the message, similar to the index file of the database, and stores the address pointing to the physical storage. Each Message Queue under each Topic has a corresponding ConsumeQueue file.
CommitLog: Store the metadata of the message
ConsumerQueue: Store the index of the message in the CommitLog
IndexFile: Provide a method for querying the message by key or time interval for message query, this method of searching for the message through the IndexFile does not affect sending and consuming messages main process of
12.3 Sequential write
RocketMQ messages are written sequentially, which ensures the speed of message storage.
If the disk is used properly, the speed of the disk can completely match the data transmission speed of the network. The current high-performance disk, the sequential write speed can reach 600MB/s, exceeding the transmission speed of the general network card. But the disk random write speed is only about 100KB/s, which is 6000 times worse than the sequential write performance! Because there is such a huge speed difference, a good message queuing system will be orders of magnitude faster than a normal message queuing system.
12.4 Brush Disk Mechanism
RocketMQ messages are stored on disk, which not only ensures recovery after power failure, but also allows the amount of stored messages to exceed the memory limit. In order to improve performance, RocketMQ will try to ensure the sequential writing of the disk as much as possible. When the message is written to RocketMQ through the Producer, there are two ways to write to the disk, distributed synchronous flushing and asynchronous flushing
1) Synchronously flushing the disk
When the write success status is returned, the message has been written to the disk. The specific process is that after the message is written into the PAGECACHE of the memory, immediately notify the flashing thread to flush the disk, and then wait for the completion of the flushing. After the execution of the flushing thread is completed, the waiting thread is awakened, and the status of the message writing is returned successfully.
2) When the asynchronous flash disk returns the write success status, the message may only be written into the PAGECACHE of the memory, the write operation returns quickly, and the throughput is large; when the amount of messages in the memory accumulates to a certain level, the action of writing to the disk is triggered uniformly , fast write.
3) Configuration
Both synchronous disk flushing and asynchronous disk flushing are set through the flushDiskType parameter in the Broker configuration file, and this parameter is configured as one of SYNC_FLUSH (synchronous) and ASYNC_FLUSH (asynchronous).
12.5 Zero copy
The Linux operating system is divided into [user mode] and [kernel mode]. File operations and network operations need to involve switching between these two modes, and data copying is inevitable.
A server sends the content of the local disk file to the client, generally divided into two steps:
read: read the content of the local file;
write: send the read content through the network.
These two seemingly simple operations actually carried out 4 data copies, namely:
copying data from disk to kernel mode memory;
copying from kernel mode memory to user mode memory;
and then copying from user mode memory to network driver's kernel state memory;
finally, it is copied from the kernel state memory of the network driver to the network card for transmission.
By using mmap, memory copying to the user mode can be omitted and the speed can be improved. This mechanism is implemented in Java through MappedByteBuffer.
RocketMQ makes full use of the above features, which is the so-called "zero copy" technology, to improve the speed of message storage and network transmission.
It should be noted here that the memory mapping method of MappedByteBuffer has several limitations, one of which is that it can only map no more than 1.5 files to the virtual memory in user mode at a time, which is why RocketMQ sets a single CommitLog by default (storage message The reason why the data file is 1G
13 High Availability Mechanisms
The RocketMQ distributed cluster achieves high availability through the cooperation of Master and Slave.
The difference between Master and Slave: In the Broker configuration file, the value of the parameter brokerId is 0 indicating that the Broker is a Master, and greater than 0 indicates that the Broker is a Slave, and the brokerRole parameter will also indicate whether the Broker is a Master or a Slave.
The Broker of the Master role supports reading and writing, and the Broker of the Slave role only supports reading, that is, the Producer can only connect to the Broker of the Master role to write messages; the Consumer can connect to the Broker of the Master role, and can also connect to the Broker of the Slave role to read information.
13.1 High Availability of Message Consumption
In the Consumer configuration file, there is no need to set whether to read from the Master or the Slave. When the Master is unavailable or busy, the Consumer will automatically switch to read from the Slave. With the mechanism of automatic switching of consumers, when a machine with a master role fails, the consumer can still read messages from the slave without affecting the consumer program. This achieves high availability on the consumer side.
13.2 Message Sending High Availability
When creating a Topic, create multiple Message Queues of the Topic on multiple Broker groups (machines with the same Broker name and different brokerIds form a Broker group), so that when the Master of a Broker group is unavailable, the Masters of other groups Still available, Producer can still send messages. RocketMQ does not currently support the automatic conversion of Slave to Master. If the machine resources are insufficient and you need to convert Slave to Master, you must manually stop the Broker in the role of Slave, change the configuration file, and start the Broker with the new configuration file.
13.3 Master-slave replication
If a Broker group has Master and Slave, messages need to be copied from Master to Slave, and there are two ways of copying: synchronous and asynchronous.
1) Synchronous replication
The synchronous replication method is to wait for both Master and Slave to write successfully before feeding back the successful writing status to the client;
in the synchronous replication mode, if the Master fails, there will be all backup data on the Slave, which is easy to restore, but synchronous replication It will increase the data writing delay and reduce the system throughput.
2) Asynchronous replication
The asynchronous replication method is that as long as the master writes successfully, it can feedback the write success status to the client.
In the asynchronous replication mode, the system has lower latency and higher throughput, but if the Master fails, some data may be lost because it has not been written to the Slave;
3) Configuration
Synchronous replication and asynchronous replication are set through the brokerRole parameter in the Broker configuration file. The values of this parameter are:
ASYNC_MASTER: asynchronous replication master node
SYNC_MASTER: synchronous replication master node
SLAVE: slave node
Summary
In practical applications, it is necessary to combine the business scenarios and reasonably set the disk flushing method and master-slave replication method, especially the SYNC_FLUSH (synchronous disk flushing) method, which will significantly reduce performance due to frequent triggering of disk write actions.
Normally, Master and Slave should be configured as ASYNC_FLUSH (asynchronous disk flushing), and the master and slave should be configured as SYNC_MASTER (synchronous replication) replication, so that even if one machine fails, the data can still be guaranteed. Lost, is a good choice.
14 load balancing
14.1 Producer load balancing
On the Producer side, when each instance sends a message, it will poll all message queues for sending by default, so that the messages fall on different queues on average. Since the queue can be scattered in different brokers, the messages are sent to different brokers. The
labels on the arrow lines represent the order. The publisher will send the first message to Queue 0, and then send the second message to Queue 1. , and so on.
14.2 Consumer load balancing
1) Cluster mode
In cluster consumption mode, each consumer group that subscribes to this topic will receive messages, and each message will only be consumed by one instance in a consumer group. RocketMQ uses active pull to pull and consume messages. When pulling, it is necessary to specify which message queue to pull.
Whenever the number of instances changes, a load balancing of all instances will be triggered. At this time, the queues will be evenly distributed to each instance according to the number of queues and the number of instances.
The default allocation algorithm is AllocateMessageQueueAveragely, as shown below:
There is another average algorithm, AllocateMessageQueueAveragelyByCircle, which also allocates each queue equally, but divides the queues in a circular manner, as shown in the following figure:
It should be noted that in the cluster mode, only one instance is allowed to be allocated to the queue. This is because if multiple instances consume messages from a queue at the same time, which messages to pull are actively controlled by the consumer, which will result in the same message It is consumed multiple times under different instances, so the algorithm is that one queue is only allocated to one consumer instance, and one consumer instance can be allocated to different queues at the same time.
By adding consumer instances to share the consumption of the queue, it can play a role in horizontally expanding the consumption capacity. When an instance goes offline, load balancing will be triggered again. At this time, the originally allocated queue will be allocated to other instances to continue consumption.
However, if the number of consumer instances is greater than the total number of message queues, the extra consumer instances will not be assigned to queues, and messages will not be consumed, and the load sharing will not be possible. Therefore, it is necessary to control the total number of queues to be greater than or equal to the number of consumers.
14.3 Broadcast Mode
Since the broadcast mode requires that a message needs to be delivered to all consumer instances under a consumer group, there is no saying that the message is allocated for consumption.
In terms of implementation, one of the differences is that when consumers allocate queues, all consumers are assigned to all queues.
15 message retries
15.1 Retries of Sequenced Messages
For sequential messages, when consumers fail to consume messages, the message queue RocketMQ will automatically retry messages continuously (each interval is 1 second), at this time, the application will be blocked for message consumption. Therefore, when using sequential messages, it is important to ensure that the application can monitor and handle consumption failures in a timely manner to avoid blocking.
15.2 Retries for out-of-order messages
For out-of-order messages (ordinary, timed, delayed, and transactional messages), when consumers fail to consume messages, you can achieve the result of message retry by setting the return status.
The retry of out-of-order messages is only effective for the cluster consumption mode; the broadcast mode does not provide the failure retry feature, that is, after the consumption fails, the failed message will not be retried, and the new message will continue to be consumed.
1) Number of retries
The message queue RocketMQ allows each message to retry up to 16 times by default, and the interval between each retry is as follows:
If the message still fails after 16 retries, the message will not be delivered. If calculated strictly according to the above retry interval, a message will be retried 16 times in the next 4 hours and 46 minutes under the premise that consumption fails all the time. Messages beyond this time range will not retry delivery .
Note: No matter how many times a message is retried, the Message ID of these retried messages will not change.
15.3 Configuration method
15.4 After the consumption fails, retry the configuration method
In the cluster consumption mode, if the message consumption fails and the message is expected to be retried, it needs to be explicitly configured in the implementation of the message listener interface (choose one of the three methods): return Action.ReconsumeLater (recommended) return Null and throw an exception public
class MessageListenerImpl implements MessageListener { @Override public Action consume(Message message, ConsumeContext context) { //Process message doConsumeMessage(message); //Method 1: Return Action.ReconsumeLater, the message will be retried return Action.ReconsumeLater; //Method 2: Return null, the message will be retried return null; //Method 3: Throw an exception directly, the message will be retried throw new RuntimeException(“Consumer Message exceotion”); } }
15.5 After consumption fails, do not retry the configuration method
In the cluster consumption mode, it is expected that the message will not be retried after the message fails, and it is necessary to catch the exception that may be thrown in the consumption logic, and finally return Action.CommitMessage, after which this message will not be retried.
public class MessageListenerImpl implements MessageListener {
@Override
public Action consume(Message message, ConsumeContext context) {
try {
doConsumeMessage(message);
} catch (Throwable e) {
//捕获消费逻辑中的所有异常，并返回 Action.CommitMessage;
return Action.CommitMessage;
}
//消息处理正常，直接返回 Action.CommitMessage;
return Action.CommitMessage;
}
}
15.6 Maximum number of retries for custom messages
The message queue RocketMQ allows the consumer to set the maximum number of retries when starting, and the retry interval will follow the following strategy: if the
maximum number of retries is less than or equal to 16 times, the retry interval is the same as that described in the above table.
The maximum number of retries is greater than 16, and the interval between retries exceeding 16 times is 2 hours each time.
Properties properties = new Properties();
//配置对应 Group ID 的最大消息重试次数为 20 次
properties.put(PropertyKeyConst.MaxReconsumeTimes,"20");
Consumer consumer =ONSFactory.createConsumer(properties);
Note:
The setting of the maximum number of message retries is valid for all Consumer instances under the same Group ID.
If MaxReconsumeTimes is set for only one of the two Consumer instances under the same Group ID, the configuration will take effect for both Consumer instances.
The configuration takes effect by overwriting, that is, the last consumer instance started will overwrite the configuration of the previously started instance
15.7 Obtaining the number of message retries
After the consumer receives the message, it can obtain the number of retries of the message as follows:
public class MessageListenerImpl implements MessageListener {
@Override
public Action consume(Message message, ConsumeContext context) {
//获取消息的重试次数
System.out.println(message.getReconsumeTimes());
return Action.CommitMessage;
}
}
15.8 Multi-consumer group retry
Suppose there are A consumer group and B consumer group. When A and B listen to the same topic at the same time, both A and B get the same message, but A fails to consume (return Action.ReconsumeLater), but B consumes successfully. Then when retrying, rocketMQ will only send the message to the B consumer group, and will not send it to the A consumer group.
16 dead letter queue
When a message fails to be consumed for the first time, the message queue RocketMQ will automatically retry the message; after reaching the maximum number of retries, if the consumption still fails, it indicates that the consumer cannot consume the message correctly under normal circumstances. At this time, the message queue RocketMQ The message will not be discarded immediately, but will be sent to the special queue corresponding to the consumer.
In the message queue RocketMQ, messages that cannot be consumed under normal circumstances are called Dead-Letter Messages, and special queues that store dead-letter messages are called Dead-Letter Queues.
16.1 Dead Letter Characteristics
Dead letter messages have the following characteristics:
they will no longer be normally consumed by consumers.
The validity period is the same as normal messages, both are 3 days, and will be automatically deleted after 3 days. Therefore, please deal with the dead letter message in time within 3 days after it is generated.
The dead letter queue has the following characteristics:
a dead letter queue corresponds to a Group ID, rather than a single consumer instance.
If a Group ID does not generate a dead letter message, the message queue RocketMQ will not create a corresponding dead letter queue for it.
A dead letter queue contains all dead letter messages generated by the corresponding Group ID, no matter which Topic the message belongs to.
16.2 View dead letter information
Query the topic information of the dead letter queue in the console
Query dead letter messages according to the subject in the message interface
Choosing to resend a message
A message entering the dead letter queue means that some factors prevent the consumer from consuming the message normally, so you usually need to take special care of it. After troubleshooting suspicious factors and solving the problem, you can resend the message on the RocketMQ console of the message queue, so that consumers can consume it again.
17 consumption idempotence
After the message queue RocketMQ consumer receives the message, it is necessary to perform idempotent processing on the message according to the unique key in the business.
17.1 The necessity of consumption idempotency
In Internet applications, especially when the network is unstable, the messages of the message queue RocketMQ may be repeated. This repetition can be summarized as follows:
Duplicate message when sending
When a message has been successfully sent to the server and completed persistence, there is a sudden network disconnection or client downtime, causing the server to fail to respond to the client. If the producer realizes that the message has failed to send and tries to send the message again, the consumer will receive two messages with the same content and the same Message ID.
Message duplication during delivery
In the message consumption scenario, the message has been delivered to the consumer and the business processing has been completed. When the client responds to the server, the network is disconnected. In order to ensure that the message is consumed at least once, the message queue RocketMQ server will try to deliver the previously processed message again after the network is restored, and the consumer will subsequently receive two messages with the same content and the same Message ID.
Message duplication during load balancing (including but not limited to network jitter, Broker restart, and subscriber application restart).
When the broker or client of the message queue RocketMQ restarts, expands or shrinks, Rebalance will be triggered. At this time, consumers may receive Repeat message.
17.2 Processing method
Because Message ID may conflict (duplicate), it is not recommended to use Message ID as the basis for truly safe idempotent processing. The best way is to use the unique identifier of the business as the key basis for idempotent processing, and the unique identifier of the business can be set through the message Key:
Message message = new Message();
message.setKey("ORDERID_100");
SendResult sendResult = producer.send(message);
订阅方收到消息时可以根据消息的 Key 进行幂等处理：
consumer.subscribe("ons_test", "*", new MessageListener() {
public Action consume(Message message, ConsumeContext context) {
String key = message.getKey()
// 根据业务唯一标识的 key 做幂等处理
}
});
18. Precautions for using RocketMQ
Under the same consumer group, the consumer logic should be the same (the monitored topic and tag must be the same).
By default, different consumer groups share messages (all consumer groups can get the same message), and the consumer group The consumers within are load balanced (only one consumer will get the message)