[RocketMQ Series 5] Message Example-Implementation of Sequential Messages & Delayed Messages & Broadcast Messages

1 Introduction

In the previous article, we introduced the implementation of simple messages. This article will mainly introduce the implementation of sequential messages. Sequential messages are divided into local sequential messages and global sequential messages.

Sequential messages mean that when consumers consume messages, they consume them in the order in which the producers send them. That is, the first sent first consumes [FIFO].

Sequential messages are divided into global sequential messages and local sequential messages.

Global sequential messages use a queue globally.

Local sequential messages are messages with sequential dependencies placed in the same queue, and multiple queues consume them in parallel.

2. Partial sequential messages

By default, RocketMQ will send messages to a queue in a broker based on polling, so there is no guarantee that messages are in order.

For example, in the ordering scenario on a shopping website: there are four messages: 1. Create order ----> 2. Order payment ----> 3. Order shipped ----> 4. Order completed. These four messages must be logically ordered. However, if RocketMQ's default message delivery method is used, the same order may be created and delivered to MessageQueue1, and the order payment may be delivered to MessageQueue2. Since the messages are in different MessageQueue, the order payment message may appear before the order creation message when the consumer consumes.

The partial sequence message is to ensure that the four messages of the same order are placed in the same queue, so that the order payment message will not be consumed before the order creation message. Just like the picture below:
Insert image description here

Local sequential message consumers consume messages in a queue of a topic sequentially. Consumers use the MessageListenerOrderly class to listen for messages.

2.1. Define producers

A topic named part_order_topic_test is defined here. After running the program, the topic can be routed to two brokers, broker-a and broker-b.

public class OrderProducer {
    
    
	// 局部顺序消费,核心就是自己选择Queue，保证需要顺序保障的消息落到同一个队列中
	public static void main(String[] args) throws MQClientException, MQBrokerException, RemotingException, InterruptedException {
    
    
		DefaultMQProducer defaultMQProducer = new DefaultMQProducer("order_producer_group");
		defaultMQProducer.setNamesrvAddr("172.31.184.89:9876");
		defaultMQProducer.start();

		for (int i = 0; i < 10; i++) {
    
    
			int orderId = i;
			for (int j = 0; j < 5; j++) {
    
    
				// 构建消息体,tags和key 只是做一个简单区分
				Message partOrderMsg = new Message("part_order_topic_test", "order_" + orderId, "KEY_" + orderId, ("局部顺序消息处理_" + orderId + ";step_" + j).getBytes());
				SendResult send = defaultMQProducer.send(partOrderMsg, new MessageQueueSelector() {
    
    
					@Override
					//这里的arg参数就是外面的orderId
					public MessageQueue select(List<MessageQueue> mqs, Message msg, Object arg) {
    
    
						Integer orderId = (Integer) arg;
						int index = orderId % mqs.size();
						return mqs.get(index);
					}
				}, orderId);

				System.out.printf("%s%n", send);
			}
		}
		defaultMQProducer.shutdown();
	}
}

Implement the MessageQueueSelector interface when sending messages to specify the queue when sending messages. Among them, public MessageQueue select(List<MessageQueue> mqs, Message msg, Object arg)the method has three parameters: among them, mqs represents the number of all queues routed by the current topic. Here are 8 queues, broker-a has 4 queues, and broker-b has 4 queues. msg is the incoming message body, and arg is the incoming orderId.
Here, the modulo modulus of the orderId and the number of queues is used to obtain which queue the message should be sent to. This ensures that messages with the same orderId will fall into the same queue.
```
Integer orderId = (Integer) arg;
int index = orderId % mqs.size();
return mqs.get(index);
```

Producer running results (partial screenshots)

It can be seen from the running results that messages with the same orderId are delivered to the same MessageQueue, and the same MessageQueue queues are naturally ordered.

2.2. Define consumers

After talking about producers, let’s talk about consumers. The consumer's logic mainly needs to implement the MessageListenerOrderly class to monitor messages when consuming. The core code is:

	// 2.订阅消费消息
		defaultMQPushConsumer.registerMessageListener(new MessageListenerOrderly() {
    
    
			@Override
			public ConsumeOrderlyStatus consumeMessage(List<MessageExt> msgs, ConsumeOrderlyContext context) {
    
    
				for (MessageExt msg : msgs) {
    
    
					System.out.println("消费得到的消息是={}" + msg);
					System.out.println("消息体内容是={}" + new String(msg.getBody()));
				}
				return ConsumeOrderlyStatus.SUCCESS;
			}
		});

Three consumers are started here. Regardless of the order in which the consumers consume, the five messages under the same orderId are consumed sequentially.

3. Problems encountered

During the first debugging, a broker is full error occurred. This is caused by insufficient disk space. You can use df -hthe command to check the current disk space usage. When the disk space usage exceeds 90%, this error will be reported.

4. Global sequential messages

Global sequential messaging means that all messages consumed by consumers are sequential. This can only be achieved by sending all messages to the same MessageQueue, which will greatly affect efficiency in high concurrency scenarios.

5. Broadcast messages

Broadcast messages are sent to all subscribers of a topic. Multiple consumers subscribing to the same topic can receive all messages sent by the producer.

The producer implementation of broadcast messages is consistent with the producer implementation of ordinary synchronization messages. The difference is that the message mode of the consumer is different. Here are the differences in consumer implementations.

	DefaultMQPushConsumer defaultMQPushConsumer = new DefaultMQPushConsumer("broadCastGroup");
		defaultMQPushConsumer.setNamesrvAddr("172.31.184.89:9876");
		// 设置消费者的模式是广播模式
		defaultMQPushConsumer.setMessageModel(MessageModel.BROADCASTING);

		//从第一位开始消费
		defaultMQPushConsumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET);

6. Delayed messages

Delayed messages differ from ordinary messages in that they are not delivered until a specified amount of time has elapsed. The producer does not delay sending the message, but sends it to the topic, and the consumer delays consumption for a specified time.

6.1. Delayed message producers

DefaultMQProducer defaultMQProducer = new DefaultMQProducer("scheduled_group");
		defaultMQProducer.setNamesrvAddr("172.31.186.180:9876");
		defaultMQProducer.start();
		for (int i = 0; i < 100; i++) {
    
    
			Message message = new Message("Schedule_topic", ("延迟消息测试" + i).getBytes());
			//设置延迟级别，默认有18个延迟级别,这个消息将延迟10秒消费
			message.setDelayTimeLevel(3);
			defaultMQProducer.send(message);
		}
		System.out.println("所有延迟消息发送完成");
		defaultMQProducer.shutdown();

The main difference between delayed message producers and ordinary message producers is that delayed messages need to call setDelayTimeLevelthe method to set the delay level. The setting level here is 3, which means a delay of 10 seconds. RocketMQ provides 18 latency levels. This can be found in the broker configuration in the cluster in RocketMQ's dashboard.

Consumers of delayed messages are the same as consumers of ordinary messages. RocketMQ internally stores delayed messages through a topic named SCHEDULE_TOPIC_XXXX.

7. Batch messages

Sending messages in batches improves the performance of message delivery. It is officially recommended that the total size of batch messages should not exceed 1M, but in fact it should not exceed 4M. If the batch message exceeds 4M, it needs to be processed in batches. At the same time, set the broker's configuration parameters to 4M (modify in the broker's configuration file: maxMessageSize=4194304). The core code is as follows:

	//4.创建消息
		List<Message> messageList = new ArrayList<>();
		for (int i = 0; i < 100*100; i++) {
    
    
			// 创建消息，指定topic,以及消息体
			messageList.add(new Message("batch_topic", ("飞哥测试批量消息" + i).getBytes()));
		}
		//批量消息消息小于4M的处理
		SendResult send = defaultMQProducer.send(messageList);
		System.out.println(send);

8. Filter messages

Filter using tags

In most cases, labels are a simple and useful design for selecting the message you want.

The first is to filter messages based on tags. The producer specifies the tag of the message when sending the message, and the consumer can filter the messages based on the tag.

8.1. Filter message producers

Three tags are defined here, namely tagA, tagB and tagC. The producer assigns different tags to each message when producing messages.

DefaultMQProducer defaultMQProducer = new DefaultMQProducer("TagProducer_group");
		defaultMQProducer.setNamesrvAddr("172.31.184.89:9876");
		defaultMQProducer.start();
		String[] tags = new String[]{
    
    "tagA", "tagB", "tagC"};
		for (int i = 0; i < 15; i++) {
    
    
			Message message = new Message("TagFilterTest", tags[i % tags.length], ("飞哥tag消息过滤" + tags[i % tags.length]).getBytes());
			SendResult send = defaultMQProducer.send(message);
			System.out.printf("%s%n", send);
		}
		defaultMQProducer.shutdown();

8.2. Consumers who filter messages

The consumer filters out the messages with tagA and tagC for consumption. Here, the broker actually pushes the messages that the consumer needs to the consumer.

	DefaultMQPushConsumer defaultMQPushConsumer = new DefaultMQPushConsumer("tagConsumer");
		defaultMQPushConsumer.setNamesrvAddr("172.31.184.89:9876");
		defaultMQPushConsumer.subscribe("TagFilterTest", "tagA||tagC");
		defaultMQPushConsumer.registerMessageListener((MessageListenerConcurrently) (msgs, context) -> {
    
    
			for (MessageExt msg : msgs) {
    
    
				System.out.println("接收到的消息=" + msg);
				System.out.println("接收到的消息体=" + new String(msg.getBody()));
			}
			return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
		});
		defaultMQPushConsumer.start();
		System.out.println("消费者已经启动");

Filter using SQL

The SQL function can perform some calculations through the attributes entered when sending the message. Under the syntax defined by RocketMQ, some interesting logic can be implemented.

grammar

RocketMQ only defines some basic syntax classes to support this feature.

1. 数值比较：如 `>`,`>=`,`<=`,`BETWEEN`,`=`;
2. 字符比较：如 `=`,'<>',`IN`;
3. `IS NULL` 或 `IS NOT NULL` ;
4. 逻辑`AND`,`OR`,`NOT`;

The constant types are:

1. 数字，如 123，
2. 字符，如 'abc',必须用单引号；
3. `NULL`,特殊常数；
4. 布尔值，`TRUE` 或 `FALSE`;

SQL filter producer

The producer mainly sets attribute filtering message.putUserProperty("a", String.valueOf(i));, which means that the first message key-value pair is a=0, and the second message key-value pair is a=1.

	DefaultMQProducer defaultMQProducer = new DefaultMQProducer("TagProducer_group");
		defaultMQProducer.setNamesrvAddr("172.31.184.89:9876");
		defaultMQProducer.start();
		String[] tags = new String[]{
    
    "tagA", "tagB", "tagC"};
		for (int i = 0; i < 15; i++) {
    
    
			Message message = new Message("SQLFilterTest", tags[i % tags.length], ("飞哥sql消息过滤" + tags[i % tags.length]).getBytes());

			message.putUserProperty("a", String.valueOf(i));
			SendResult send = defaultMQProducer.send(message);
			System.out.printf("%s%n", send);
		}
		defaultMQProducer.shutdown();

SQL filter consumer:

	DefaultMQPushConsumer defaultMQPushConsumer = new DefaultMQPushConsumer("tagConsumer");
		defaultMQPushConsumer.setNamesrvAddr("172.31.184.89:9876");
		defaultMQPushConsumer.subscribe("SQLFilterTest", MessageSelector.bySql("(TAGS is not null and TAGS in ('tagA','tagC'))"+" and (a is null and a between 0 and 3)"));
		defaultMQPushConsumer.registerMessageListener((MessageListenerConcurrently) (msgs, context) -> {
    
    
			for (MessageExt msg : msgs) {
    
    
				System.out.println("接收到的消息=" + msg);
				System.out.println("接收到的消息体=" + new String(msg.getBody()));
			}
			return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
		});
		defaultMQPushConsumer.start();
		System.out.println("消费者已经启动");

If you run The broker does not support consumer to filter message by SQL92

You need to modify the broker.conf file and add the following configuration:

# 开启对 propertyfilter的支持
enablePropertyFilter = true 
filterSupportRetry = true

Then restart the broker.

Summarize

This article introduces local sequential messages, global sequential messages, broadcast messages, delayed messages, and how to send messages in batches and filter messages.