1. Ordinary messages
1 Message sending classification
Producer also has multiple choices for how to send messages, and different methods will produce different system effects.
Sending messages synchronously
means that after Producer sends a message, it will send the next message after receiving the ACK returned by MQ. This method has the highest message reliability, but the message sending efficiency is too low.
Asynchronously sending messages
Asynchronously sending messages means that after Producer sends a message, it does not need to wait for MQ to return ACK and directly sends the next message. The reliability of messages in this method can be guaranteed, and the efficiency of message sending can also be improved.
One-way message sending
means that the Producer is only responsible for sending messages and does not wait for or process MQ's ACK. MQ does not return ACK in this sending method. This method has the highest message sending efficiency, but the message reliability is poor.
2 Code examples
Create a project
Create a Maven Java project rocketmq-test.
Import dependencies
Import rocketmq's client dependencies.
<properties>
<project.build.sourceEncoding>UTF-
8</project.build.sourceEncoding>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.rocketmq</groupId>
<artifactId>rocketmq-client</artifactId>
<version>4.8.0</version>
</dependency>
</dependencies>
Define synchronization message sending producer
public class SyncProducer {
public static void main(String[] args) throws Exception {
// 创建一个producer，参数为Producer Group名称
DefaultMQProducer producer = new DefaultMQProducer("pg");
// 指定nameServer地址
producer.setNamesrvAddr("rocketmqOS:9876");
// 设置当发送失败时重试发送的次数，默认为2次
producer.setRetryTimesWhenSendFailed(3);
// 设置发送超时时限为5s，默认3s
producer.setSendMsgTimeout(5000);
// 开启生产者
producer.start();
// 生产并发送100条消息
for (int i = 0; i < 100; i++) {
byte[] body = ("Hi," + i).getBytes();
Message msg = new Message("someTopic", "someTag", body);
// 为消息指定key
msg.setKeys("key-" + i);
// 发送消息
SendResult sendResult = producer.send(msg);
System.out.println(sendResult);
}
// 关闭producer
producer.shutdown();
}
}
// 消息发送的状态
public enum SendStatus {
SEND_OK, // 发送成功
FLUSH_DISK_TIMEOUT, // 刷盘超时。当Broker设置的刷盘策略为同步刷盘时才可能出
现这种异常状态。异步刷盘不会出现
FLUSH_SLAVE_TIMEOUT, // Slave同步超时。当Broker集群设置的Master-Slave的复
制方式为同步复制时才可能出现这种异常状态。异步复制不会出现
SLAVE_NOT_AVAILABLE, // 没有可用的Slave。当Broker集群设置为Master-Slave的
复制方式为同步复制时才可能出现这种异常状态。异步复制不会出现
}
Define asynchronous message sending producer
public class AsyncProducer {
public static void main(String[] args) throws Exception {
DefaultMQProducer producer = new DefaultMQProducer("pg");
producer.setNamesrvAddr("rocketmqOS:9876");
// 指定异步发送失败后不进行重试发送
producer.setRetryTimesWhenSendAsyncFailed(0);
// 指定新创建的Topic的Queue数量为2，默认为4
producer.setDefaultTopicQueueNums(2);
producer.start();
for (int i = 0; i < 100; i++) {
byte[] body = ("Hi," + i).getBytes();
try {
Message msg = new Message("myTopicA", "myTag", body);
// 异步发送。指定回调
producer.send(msg, new SendCallback() {
// 当producer接收到MQ发送来的ACK后就会触发该回调方法的执行
@Override
public void onSuccess(SendResult sendResult) {
System.out.println(sendResult);
}
@Override
public void onException(Throwable e) {
e.printStackTrace();
}
});
} catch (Exception e) {
e.printStackTrace();
}
} // end-for
// sleep一会儿
// 由于采用的是异步发送，所以若这里不sleep，
// 则消息还未发送就会将producer给关闭，报错
TimeUnit.SECONDS.sleep(3);
producer.shutdown();
}
}
Define one-way message sending producer
public class OnewayProducer {
public static void main(String[] args) throws Exception{
DefaultMQProducer producer = new DefaultMQProducer("pg");
producer.setNamesrvAddr("rocketmqOS:9876");
producer.start();
for (int i = 0; i < 10; i++) {
byte[] body = ("Hi," + i).getBytes();
Message msg = new Message("single", "someTag", body);
// 单向发送
producer.sendOneway(msg);
}
producer.shutdown();
System.out.println("producer shutdown");
}
}
Define message consumers
public class SomeConsumer {
public static void main(String[] args) throws MQClientException {
// 定义一个pull消费者
// DefaultLitePullConsumer consumer = new
DefaultLitePullConsumer("cg");
// 定义一个push消费者
DefaultMQPushConsumer consumer = new
DefaultMQPushConsumer("cg");
// 指定nameServer
consumer.setNamesrvAddr("rocketmqOS:9876");
// 指定从第一条消息开始消费
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET
);
// 指定消费topic与tag
consumer.subscribe("someTopic", "*");
// 指定采用“广播模式”进行消费，默认为“集群模式”
// consumer.setMessageModel(MessageModel.BROADCASTING);
// 注册消息监听器
consumer.registerMessageListener(new
MessageListenerConcurrently() {
// 一旦broker中有了其订阅的消息就会触发该方法的执行，
// 其返回值为当前consumer消费的状态
@Override
public ConsumeConcurrentlyStatus
consumeMessage(List<MessageExt> msgs,
ConsumeConcurrentlyContext context) {
// 逐条消费消息
for (MessageExt msg : msgs) {
System.out.println(msg);
}
// 返回消费状态：消费成功
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
});
// 开启消费者消费
consumer.start();
System.out.println("Consumer Started");
}
}
2. Sequential messages
1. What is sequential messages
? Sequential messages refer to messages (FIFO) that are consumed strictly in the order in which the messages are sent.
By default, producers will send messages to different Queue partition queues in Round Robin polling mode; when consuming messages, they will
pull messages from multiple Queues. In this case, the order of sending and consumption cannot be guaranteed. If the message is only sent to the same
Queue, and the message is only pulled from this Queue when consuming, the order of the message is strictly guaranteed.
2 Why sequential messages are needed
? For example, there is TOPIC ORDER_STATUS (order status), and there are 4 Queue queues under it. Different messages in this Topic are used to
describe different statuses of the current order. Assume that the order has status: Unpaid, Paid, Shipping, Shipping Successful, Shipping Failed.
Based on the above order status, the producer can generate the following messages in time sequence:
Order T0000001: Unpaid --> Order T0000001: Paid --> Order T0000001: Delivery --> Order
T0000001: Delivery failure
message sent After entering MQ, if Queue is selected using a polling strategy, the message storage in MQ may be as follows:
In this case, we hope that the order in which the Consumer consumes messages is consistent with what we send. However, with the above MQ delivery and consumption methods
, we cannot guarantee that the order is correct. For messages with abnormal sequence, even if the Consumer is set with a certain state fault tolerance, it cannot
fully handle so many random combinations.
Based on the above situation, the following solution can be designed: for messages with the same order number, they are placed in a Queue through a certain strategy,
and then the consumer adopts a certain strategy (for example, a thread independently processes a queue to ensure processing The order of messages
) can ensure the order of consumption.
3 Orderliness classification
Depending on the ordering scope, RocketMQ can strictly guarantee the orderliness of two kinds of messages: partition ordering and global ordering.
Globally ordered
When there is only one Queue involved in sending and consuming, the order guaranteed is the order of messages in the entire Topic, which is called global order.
Specify the number of Queues when creating a Topic. There are three ways to specify:
1) When creating a Producer in the code, you can specify the number of Queues for the automatically created Topic.
2) Specify the number of Queues when manually creating a Topic in the RocketMQ visual console.
3) Specify the number of Queues when manually creating a Topic using the mqadmin command.
Partition ordering
If there are multiple Queues participating, it can only guarantee the order of messages on the Queue partition queue, which is called partition ordering.
How to implement Queue selection? When defining Producer, we can specify the message queue selector, and this selector is
defined by our own implementation of the MessageQueueSelector interface.
When defining the selection algorithm of a selector, you generally need to use the selection key. This selection key can be a message key or other
data. But no matter who selects the key, it cannot be repeated and is unique.
The general selection algorithm is to take the selection key (or its hash value) modulo the number of Queues contained in the Topic, and the result
is the QueueId of the selected Queue.
There is a problem with the modulo algorithm: the modulo results of different selected keys and Queue numbers may be the same, that is,
messages with different selected keys may appear in the same Queue, that is, the same Consuemr may consume messages with different selected keys. How to solve this problem
? The general approach is to obtain the selection key from the message and judge it. If it is a message that the current Consumer needs
to consume, consume it directly. Otherwise, do nothing.
This approach requires that the selected key can be obtained by the Consumer along with the message . At this time, it is a better practice to use the message key as the selection key.
Will the above approach cause the following new problems? If the message that does not belong to that Consumer is pulled, can
the Consumer who should consume the message still consume the message? Messages in the same Queue cannot be processed by messages in the same Group.
Different Consumers consume at the same time. Therefore, Consumers consuming messages with different selected keys in a Queue must belong to different
Groups. The consumption between consumers in different groups is isolated from each other and does not affect each other.
public class OrderedProducer {
public static void main(String[] args) throws Exception {
DefaultMQProducer producer = new DefaultMQProducer("pg");
producer.setNamesrvAddr("rocketmqOS:9876");
producer.start();
for (int i = 0; i < 100; i++) {
Integer orderId = i;
byte[] body = ("Hi," + i).getBytes();
Message msg = new Message("TopicA", "TagA", body);
SendResult sendResult = producer.send(msg, new
MessageQueueSelector() {
@Override
public MessageQueue select(List<MessageQueue> mqs,
Message msg, Object arg) {
Integer id = (Integer) arg;
int index = id % mqs.size();
return mqs.get(index);
}
}, orderId);
System.out.println(sendResult);
}
producer.shutdown();
}
}
3. Delayed message
1. What is a delayed message?
When a message is written to the Broker, the message can be consumed and processed after a specified period of time, which is called a delayed message.
Using RocketMQ's delayed messages can realize the function of scheduled tasks without using a timer. Typical application scenarios are the
scenario in e-commerce transactions where the order is closed due to timeout and non-payment, and the scenario where the 12306 platform ticket booking is canceled after timeout and non-payment.
In e-commerce platforms, a delayed message is sent when an order is created. This message will be delivered to the backend business system
(Consumer) after 30 minutes. After receiving the message, the backend business system will determine whether the corresponding order has been paid. If it is not
completed, the order is canceled and the goods are put back into inventory again; if the payment is completed, it is ignored.
In the 12306 platform, a delayed message will be sent after the ticket reservation is successful. This message will be delivered to the backend
business system (Consumer) after 45 minutes. After receiving the message, the backend business system will determine whether the corresponding order has been paid. If
it is not completed, cancel the reservation and put the ticket back into the ticket pool; if the payment is completed, it is ignored.
2 Delay Level
The delay length of delayed messages does not support arbitrary length delays and is specified through a specific delay level. The delay level is defined in
the following variables in the MessageStoreConfig class of the RocketMQ server:
that is, if the specified delay level is 3, it means that the delay length is 10s, that is, the delay level starts counting from 1.
Of course, if you need a customized delay level, you can add the following configuration to the configuration loaded by the broker (for example, the
level 1d of 1 day is added below). The configuration file is in the conf directory under the RocketMQ installation directory.
messageDelayLevel = 1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h 1d
3 Delayed message implementation principle
The specific implementation plan is: after
modifying the message
Producer to send the message to the Broker, the Broker will first write the message to the commitlog file, and then distribute it to the corresponding
consumequeue. However, before distribution, the system will first determine whether the message contains a delay level. If not, it will be
distributed normally; if it is, you need to go through a complicated process:
modify the Topic of the message to SCHEDULE_TOPIC_XXXX . According to the delay level, create the corresponding queueId directory and consumequeue file
under the SCHEDULE_TOPIC_XXXX topic in the consumequeue directory (if there are no these
directories and files).
The corresponding relationship between delay level delayLevel and queueId is queueId = delayLevel -1.
It should be noted that when creating the queueId directory, the directories corresponding to all delay levels are not created at once,
but which directory is created according to which delay level is used
. Message index unit content. The Message Tag HashCode part in the index unit originally stores
the Hash value of the message Tag. Now modified to the delivery time of the message. Delivery time refers to the time it takes for the message to be re-modified to the original Topic and then written to
the commitlog again. Delivery time = message storage time + delay level time. The message storage time refers to
the timestamp when the message is sent to the Broker.
Write the message index to the corresponding consumequeue under the SCHEDULE_TOPIC_XXXX topic
How are the messages in the Queue of each delay level in the SCHEDULE_TOPIC_XXXX directory sorted?
They are sorted according to message delivery time. All delayed messages of the same level in a Broker will be written to the
same Queue in the SCHEDULE_TOPIC_XXXX directory in the consumerqueue directory. That is, the delay level time of message delivery time in a Queue
is the same. Then the delivery time depends on the message storage time.
That is, they are sorted according to the time when the messages are sent to the Broker .
Delivery of delayed messages
There is a delayed message service class ScheuleMessageService inside the Broker, which consumes the messages in SCHEDULE_TOPIC_XXXX
, that is, delivers the delayed messages to the target Topic according to the delivery time of each message. However, before delivery,
the originally written message will be read out from the commitlog again, and its original delay level will be set to 0, that is, the original message will become an ordinary message without delay
. Then deliver the message to the target Topic again.
When the Broker starts, ScheuleMessageService will create and start a timer TImer to perform corresponding scheduled
tasks. The system will define a corresponding number of TimerTasks based on the number of delay levels. Each TimerTask is responsible for the consumption and delivery of messages of a delay
level. Each TimerTask will detect whether the first message of the corresponding Queue queue has expired. If the
first message has not expired, all subsequent messages will not expire (the messages are sorted according to delivery time); if the first
message has expired, the message will be delivered to the target Topic, that is, the message will be consumed. information.
Re-write the message to the commitlog
The delayed message service class ScheuleMessageService sends the delayed message to the commitlog again, forms a new message index
entry again, and distributes it to the corresponding Queue.
This is actually an ordinary message sending. It's just that this time the message Producer is the delayed message service class
ScheuleMessageService.
4 Code Example
Define DelayProducer class
public class DelayProducer {
public static void main(String[] args) throws Exception {
DefaultMQProducer producer = new DefaultMQProducer("pg");
producer.setNamesrvAddr("rocketmqOS:9876");
producer.start();
for (int i = 0; i < 10; i++) {
byte[] body = ("Hi," + i).getBytes();
Message msg = new Message("TopicB", "someTag", body);
// 指定消息延迟等级为3级，即延迟10s
// msg.setDelayTimeLevel(3);
SendResult sendResult = producer.send(msg);
// 输出消息被发送的时间
System.out.print(new SimpleDateFormat("mm:ss").format(new
Date()));
System.out.println(" ," + sendResult);
}
producer.shutdown();
}
}
Define OtherConsumer class
public class OtherConsumer {
public static void main(String[] args) throws MQClientException {
DefaultMQPushConsumer consumer = new
DefaultMQPushConsumer("cg");
consumer.setNamesrvAddr("rocketmqOS:9876");
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET
);
consumer.subscribe("TopicB", "*");
consumer.registerMessageListener(new
MessageListenerConcurrently() {
@Override
public ConsumeConcurrentlyStatus
consumeMessage(List<MessageExt> msgs,
ConsumeConcurrentlyContext context) {
for (MessageExt msg : msgs) {
// 输出消息被消费的时间
SimpleDateFormat("mm:ss").format(new Date()));
System.out.println(" ," + msg);
}
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
});
consumer.start();
System.out.println("Consumer Started");
}
}
4. Transaction Message
1 Problem Introduction
One of the demand scenarios here is: ICBC user A transfers 10,000 yuan to CCB user B.
We can use synchronous messages to handle this requirement scenario:
- The ICBC system sends a synchronization message M to the Broker to add 10,000 yuan to B.
- After the message is successfully received by the Broker, a successful ACK is sent to the ICBC system.
- After receiving the successful ACK, the ICBC system deducts 10,000 yuan from user A.
- The CCB system obtains message M from Broker
- The CCB system consumes message M, which means adding 10,000 yuan to user B.
There is a problem: if the deduction operation in step 3 fails, the message has been successfully sent to the Broker. For MQ
, as long as the message is written successfully, the message can be consumed. At this time, user B in the CCB system has increased by 10,000 yuan.
A data inconsistency problem has occurred .
2 Solution idea
The solution idea is to make steps 1, 2 and 3 atomic, either all succeed or all fail. That is, after the message is sent successfully, it must be
ensured that the deduction is successful. If the deduction fails, the successful message will be rolled back. This idea is to use transaction messages.
A distributed transaction solution is used here .
Use transaction messages to handle this requirement scenario:
- The transaction manager TM initiates an instruction to the transaction coordinator TC to start a global transaction
- The ICBC system sends a transaction message M to TC to add 10,000 yuan to B.
- TC will send the semi-transaction message prepareHalf to the Broker and pre-submit the message M to the Broker. At this time, the CCB system cannot see
the message M in the Broker. - The Broker will report the pre-commit execution results to TC.
- If the pre-submission fails, TC will report the pre-submission failure response to TM, and the global transaction ends; if the pre-submission is successful, TC will
call the callback operation of the ICBC system to complete the pre-deduction of 10,000 yuan for ICBC user A. - The ICBC system will send the withholding execution result to TC, that is, the execution status of the local transaction
- After receiving the withholding execution results, TC will report the results to TM.
There are three possibilities for the withholding execution results:
// 描述本地事务执行状态
public enum LocalTransactionState {
COMMIT_MESSAGE, // 本地事务执行成功
ROLLBACK_MESSAGE, // 本地事务执行失败
UNKNOW, // 不确定，表示需要进行回查以确定本地事务的执行结果
}
- TM will issue different confirmation instructions to TC based on the reporting results.
If the withholding is successful (the local transaction status is COMMIT_MESSAGE), the TM will send the Global Commit instruction to the TC.
If the withholding fails (the local transaction status is ROLLBACK_MESSAGE), the TM will send the Global Commit command to the TC. Send the Global Rollback command.
If the current status is unknown (the local transaction status is UNKNOW), the local transaction status review operation of the ICBC system will be triggered. The review
operation will report the review result, namely COMMIT_MESSAGE or ROLLBACK_MESSAGE, to TC. TC reports the results
to TM, and TM will send the final confirmation instruction Global Commit or Global Rollback to TC. - After receiving the instruction, TC will send a confirmation instruction to the Broker and ICBC systems
. If the TC receives a Global Commit instruction, it will send a Branch Commit instruction to the Broker and ICBC systems. At this time,
the message M in the Broker can be seen by the CCB system; at this time, the deduction operation in ICBC user A is actually confirmed. If the
TC receives the Global Rollback command, it sends the Branch Rollback command to the Broker and ICBC systems. At this time,
the message M in the Broker will be revoked; the deduction operation in ICBC user A will be rolled back. The
above scheme is to ensure that the message delivery and deduction operations can be in one transaction. If both succeed, if one fails,
then Roll back all.
The above solution is not a typical XA model. Because the branch transaction in XA mode is asynchronous, while
the message pre-commit and pre-deduction operations in the transaction message scheme are synchronous.
3 Basics
Distributed Transactions
For distributed transactions, in layman's terms, one operation consists of several branch operations. These branch operations belong to different applications and are distributed on
different servers. Distributed transactions need to ensure that these branch operations either all succeed or all fail. Distributed transactions are the same as ordinary transactions
, in order to ensure the consistency of operation results.
Transaction messages
RocketMQ provides a distributed transaction function similar to X/Open XA, and the final consistency of distributed transactions can be achieved through transaction messages. XA
is a distributed transaction solution, a distributed transaction processing model.
Semi-transactional message
is a message that cannot be delivered temporarily. The sender has successfully sent the message to the Broker, but the Broker has not received the final confirmation instruction. At this time,
the message is marked as "temporarily undeliverable", that is, it cannot be seen by the consumer. . Messages in this state are semi-transactional messages.
The result of the execution of the local transaction status
Producer callback operation is the local transaction status, which will be sent to TC, and TC will then send it to TM.
TM will determine the global transaction confirmation instruction based on the local transaction status sent by TC .
// 描述本地事务执行状态
public enum LocalTransactionState {
COMMIT_MESSAGE, // 本地事务执行成功
ROLLBACK_MESSAGE, // 本地事务执行失败
UNKNOW, // 不确定，表示需要进行回查以确定本地事务的执行结果
}
Message review
Message review means re-querying the execution status of local transactions. This example is to go back to the DB to check whether the withholding operation is successful.
Note that message review does not mean re-executing the callback operation. The callback operation is to perform the withholding operation, while the message checkback is to check the results of the withholding operation.
There are two most common reasons for triggering message review:
1) The callback operation returns UNKNWON
2) TC does not receive the final global transaction confirmation instruction from TM
Message review settings in RocketMQ Regarding
message review, there are three common property settings. They are all set in the configuration file loaded by the broker, for example:
transactionTimeout=20, specifies that TM should send the final confirmation status to TC within 20 seconds, otherwise a message review will be triggered. The default
is 60 seconds
transactionCheckMax=5, specifying a maximum of 5 checkbacks, after which the message will be discarded and an error log will be recorded. The default is 15 times.
transactionCheckInterval=10, specifies that the time interval for multiple message checkbacks is 10 seconds. The default is 60 seconds.
4 XA mode Three Musketeers
XA protocol
XA (Unix Transaction) is a distributed transaction solution, a distributed transaction processing mode, based on the XA protocol. The XA protocol was first proposed by
Tuxedo (Transaction for Unix has been Extended for Distributed Operation, the Unix transaction system after the expansion of distributed operations ) and handed over to the X/Open organization as an interface standard for resource managers and transaction managers. . There are three important components in XA mode: TC, TM, and RM. TC Transaction Coordinator, transaction coordinator. Maintain the status of global and branch transactions and drive global transaction submission or rollback. In RocketMQ, Broker acts as TC. TM Transaction Manager, transaction manager. Define the scope of a global transaction: start a global transaction, commit or rollback a global transaction. It is actually the initiator of global transactions. The Producer of transaction messages in RocketMQ acts as a TM. RM Resource Manager, resource manager. Manages resources for branch transaction processing, talks to the TC to register branch transactions and report the status of branch transactions, and drives branch transaction commits or rollbacks. The Producer and Broker of transaction messages in RocketMQ are both RM.
5 XA mode architecture
XA mode is a typical 2PC, and its execution principle is as follows:
- TM initiates an instruction to TC to start a global transaction.
- According to business requirements, each RM will register branch transactions with TC one by one, and then TC will issue pre-execution instructions to RM one by one.
- Each RM will perform local transaction pre-execution after receiving the instruction.
- RM reports the pre-execution results to TC. Of course, this result may be a success or a failure.
- After receiving the Report from each RM, TC will report the summary results to TM, and TM will issue a confirmation instruction to TC based on the summary results
.
If all results are successful responses, the Global Commit command is sent to the TC.
As long as the result is a failure response, the Global Rollback instruction is sent to the TC. - After receiving the instruction, TC sends the confirmation instruction to RM again.
The transaction messaging scheme is not a typical XA pattern. Because branch transactions in XA mode are asynchronous, while
message pre-commit and withholding operations in transaction message solutions are synchronous.
6 Note that
transaction messages do not support delayed messages.
For transaction messages, idempotence must be checked because transaction messages may be consumed more than once (because there is a
situation of rolling back and then submitting).
7 Code examples
Define ICBC transaction listener
public class ICBCTransactionListener implements TransactionListener {
// 回调操作方法
// 消息预提交成功就会触发该方法的执行，用于完成本地事务
@Override
public LocalTransactionState executeLocalTransaction(Message msg,
Object arg) {
System.out.println("预提交消息成功：" + msg);
// 假设接收到TAGA的消息就表示扣款操作成功，TAGB的消息表示扣款失败，
// TAGC表示扣款结果不清楚，需要执行消息回查
if (StringUtils.equals("TAGA", msg.getTags())) {
return LocalTransactionState.COMMIT_MESSAGE;
} else if (StringUtils.equals("TAGB", msg.getTags())) {
return LocalTransactionState.ROLLBACK_MESSAGE;
} else if (StringUtils.equals("TAGC", msg.getTags())) {
return LocalTransactionState.UNKNOW;
}
return LocalTransactionState.UNKNOW;
}
// 消息回查方法
// 引发消息回查的原因最常见的有两个：
// 1)回调操作返回UNKNWON
// 2)TC没有接收到TM的最终全局事务确认指令
@Override
public LocalTransactionState checkLocalTransaction(MessageExt msg) {
System.out.println("执行消息回查" + msg.getTags());
return LocalTransactionState.COMMIT_MESSAGE;
}
}
Define transaction message producer
public class TransactionProducer {
public static void main(String[] args) throws Exception {
TransactionMQProducer producer = new
TransactionMQProducer("tpg");
producer.setNamesrvAddr("rocketmqOS:9876");
/**
* 定义一个线程池
* @param corePoolSize 线程池中核心线程数量
* @param maximumPoolSize 线程池中最多线程数
* @param keepAliveTime 这是一个时间。当线程池中线程数量大于核心线程数量
是
* 多余空闲线程的存活时长
* @param unit 时间单位
* @param workQueue 临时存放任务的队列，其参数就是队列的长度
* @param threadFactory 线程工厂
*/
ExecutorService executorService = new ThreadPoolExecutor(2, 5,
100, TimeUnit.SECONDS,
new ArrayBlockingQueue<Runnable>(2000), new
ThreadFactory() {
@Override
public Thread newThread(Runnable r) {
Thread thread = new Thread(r);
thread.setName("client-transaction-msg-check-thread");
return thread;
}
});
// 为生产者指定一个线程池
producer.setExecutorService(executorService);
// 为生产者添加事务监听器
producer.setTransactionListener(new ICBCTransactionListener());
producer.start();
String[] tags = {
"TAGA","TAGB","TAGC"};
for (int i = 0; i < 3; i++) {
byte[] body = ("Hi," + i).getBytes();
Message msg = new Message("TTopic", tags[i], body);
// 发送事务消息
// 第二个参数用于指定在执行本地事务时要使用的业务参数
SendResult sendResult =
producer.sendMessageInTransaction(msg,null);
System.out.println("发送结果为：" +
sendResult.getSendStatus());
}
}
}
To define consumers
, just use SomeConsumer of ordinary messages as the consumer.
public class SomeConsumer {
public static void main(String[] args) throws MQClientException {
// 定义一个pull消费者
faultLitePullConsumer("cg");
// 定义一个push消费者
DefaultMQPushConsumer consumer = new
DefaultMQPushConsumer("cg");
// 指定nameServer
consumer.setNamesrvAddr("rocketmqOS:9876");
// 指定从第一条消息开始消费
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET
);
// 指定消费topic与tag
consumer.subscribe("TTopic", "*");
// 指定采用“广播模式”进行消费，默认为“集群模式”
// consumer.setMessageModel(MessageModel.BROADCASTING);
// 注册消息监听器
consumer.registerMessageListener(new
MessageListenerConcurrently() {
// 一旦broker中有了其订阅的消息就会触发该方法的执行，
// 其返回值为当前consumer消费的状态
@Override
public ConsumeConcurrentlyStatus
consumeMessage(List<MessageExt> msgs,
ConsumeConcurrentlyContext context) {
// 逐条消费消息
for (MessageExt msg : msgs) {
System.out.println(msg);
}
// 返回消费状态：消费成功
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
});
// 开启消费者消费
consumer.start();
System.out.println("Consumer Started");
}
}
5. Batch messages
1. Batch sending messages.
Sending restrictions.
When sending messages, producers can send multiple messages at one time, which can greatly improve the sending efficiency of Producer. However, you need to pay attention to
the following points:
Messages sent in batches must have the same Topic.
Messages sent in batches must have the same flushing strategy.
Messages sent in batches cannot be delayed messages or transaction messages.
The batch sending size
is, by default, sent in one batch. The total message size cannot exceed 4MB bytes. If you want to exceed this value, there are two solutions:
Solution 1: Split the batch message into several message sets not larger than 4M and send them in batches multiple times.
Solution 2: Modify the properties on the Producer and Broker sides
** The Producer side needs to set the Producer's maxMessageSize attribute before sending
** The Broker side needs to modify the maxMessageSize attribute in the configuration file it loads.
The size of the message sent by the producer.
The Message sent by the producer through the send() method is not a direct serialization of the Message. Then it is sent to the network, but
a string is generated and sent out through this Message. This string consists of four parts: Topic, message body, message log
(occupying 20 bytes), and a bunch of attribute key-value used to describe the message. These attributes include, for example, the producer address, production time,
QueueId to be sent, etc. The data ultimately written to the message unit in Broker comes from these attributes.
2 Consume messages in batches.
Modify the batch attribute
Consumer's MessageListenerConcurrently listening interface. The first parameter of the consumeMessage() method is the message list
, but by default only one message can be consumed at a time. To enable it to consume multiple messages at a time, you can
specify it by modifying the consumerMessageBatchMaxSize property of the Consumer. However, the value cannot exceed 32. Because by default,
the maximum number of messages that consumers can pull each time is 32. To modify the maximum value of a pull, you can
specify it by modifying the Consumer's pullBatchSize attribute.
Problems:
Should the consumer's pullBatchSize attribute and consumeMessageBatchMaxSize attribute be set to a larger value? Of course not
.
The larger the pullBatchSize value is set, the longer it will take for the Consumer to pull each time, and the higher the possibility of transmission problems on the network
. If a problem occurs during the pulling process, all messages in this batch need to be pulled again
.
The larger the consumeMessageBatchMaxSize value is set, the lower the Consumer's concurrent message consumption capability is, and this batch of consumed
messages has the same consumption result. Because a batch of messages specified by consumeMessageBatchMaxSize will only
be processed by one thread, and as long as there is a message processing exception during the processing, this batch of messages needs to be consumed and processed again
.
3 Code examples
The requirement for this batch sending is not to modify the default value of the maximum sending of 4M, but to prevent the batch messages sent from exceeding the 4M limit.
Define message list splitter
// 消息列表分割器：其只会处理每条消息的大小不超4M的情况。
// 若存在某条消息，其本身大小大于4M，这个分割器无法处理，
// 其直接将这条消息构成一个子列表返回。并没有再进行分割
public class MessageListSplitter implements Iterator<List<Message>> {
// 指定极限值为4M
private final int SIZE_LIMIT = 4 *1024 * 1024;
// 存放所有要发送的消息
private final List<Message> messages;
// 要进行批量发送消息的小集合起始索引
private int currIndex;
public MessageListSplitter(List<Message> messages) {
this.messages = messages;
}
@Override
public boolean hasNext() {
// 判断当前开始遍历的消息索引要小于消息总数
return currIndex < messages.size();
}
@Override
public List<Message> next() {
int nextIndex = currIndex;
// 记录当前要发送的这一小批次消息列表的大小
int totalSize = 0;
for (; nextIndex < messages.size(); nextIndex++) {
// 获取当前遍历的消息
Message message = messages.get(nextIndex);
// 统计当前遍历的message的大小
int tmpSize = message.getTopic().length() +
message.getBody().length;
Map<String, String> properties = message.getProperties();
for (Map.Entry<String, String> entry :
properties.entrySet()) {
tmpSize += entry.getKey().length() +
entry.getValue().length();
}
tmpSize = tmpSize + 20;
// 判断当前消息本身是否大于4M
if (tmpSize > SIZE_LIMIT) {
if (nextIndex - currIndex == 0) {
nextIndex++;
}
break;
}
if (tmpSize + totalSize > SIZE_LIMIT) {
break;
} else {
totalSize += tmpSize;
}
} // end-for
// 获取当前messages列表的子集合[currIndex, nextIndex)
List<Message> subList = messages.subList(currIndex, nextIndex);
// 下次遍历的开始索引
currIndex = nextIndex;
return subList;
}
}
Define bulk message producer
public class BatchProducer {
public static void main(String[] args) throws Exception {
DefaultMQProducer producer = new DefaultMQProducer("pg");
producer.setNamesrvAddr("rocketmqOS:9876");
// 指定要发送的消息的最大大小，默认是4M
// 不过，仅修改该属性是不行的，还需要同时修改broker加载的配置文件中的
// maxMessageSize属性
// producer.setMaxMessageSize(8 * 1024 * 1024);
producer.start();
// 定义要发送的消息集合
List<Message> messages = new ArrayList<>();
for (int i = 0; i < 100; i++) {
byte[] body = ("Hi," + i).getBytes();
Message msg = new Message("someTopic", "someTag", body);
messages.add(msg);
}
// 定义消息列表分割器，将消息列表分割为多个不超出4M大小的小列表
MessageListSplitter splitter = new
MessageListSplitter(messages);
while (splitter.hasNext()) {
try {
List<Message> listItem = splitter.next();
producer.send(listItem);
} catch (Exception e) {
e.printStackTrace();
}
}
producer.shutdown();
}
}
Define bulk message consumer
public class BatchConsumer {
public static void main(String[] args) throws MQClientException {
DefaultMQPushConsumer consumer = new
DefaultMQPushConsumer("cg");
consumer.setNamesrvAddr("rocketmqOS:9876");
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET
);
consumer.subscribe("someTopicA", "*");
// 指定每次可以消费10条消息，默认为1
consumer.setConsumeMessageBatchMaxSize(10);
// 指定每次可以从Broker拉取40条消息，默认为32
consumer.setPullBatchSize(40);
consumer.registerMessageListener(new
MessageListenerConcurrently() {
@Override
public ConsumeConcurrentlyStatus
consumeMessage(List<MessageExt> msgs,
ConsumeConcurrentlyContext context) {
for (MessageExt msg : msgs) {
System.out.println(msg);
}
// 消费成功的返回结果
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
// 消费异常时的返回结果
// return ConsumeConcurrentlyStatus.RECONSUME_LATER;
}
});
consumer.start();
System.out.println("Consumer Started");
}
}