RocketMQ
1. Overview of MQ
Message Queue is a middleware that provides message queuing services. A software system that provides an API for the entire process of message production, storage, and consumption.
The role of MQ
- Current limiting and peak clipping: When users send excessive requests, the requests are temporarily stored for later processing. If you do not use MQ temporary storage to directly request to the business system, it is easy to cause the system to crash.
- Asynchronous decoupling: If the upstream system and downstream system are called synchronously, the throughput and concurrency of the system will be greatly reduced. The MQ layer implements asynchronous calls between two systems
- Data collection: Distributed systems will generate massive data streams, such as business logs, monitoring data, and user behavior. Collect and summarize these data streams for big data analysis.
MQ products for mainstream applications
- Kafka: Scala/Java language development. It is characterized by high throughput, but it will lose data. It is commonly used in real-time computing and log collection in the field of big data. Do not follow any MQ protocol, use self-developed protocol.
- RocketMQ: Java language development. After several years of Alibaba Double Eleven tests, the performance and stability are very high, and the functions are comprehensive. Do not follow any MQ protocol, use self-developed protocol. The open source version is not as good as the cloud version (Ali commercial version)
MQ Common Protocols
-
JMS : Java Messaging Service. The technical specification of MOM (Message Orientated Middleware) on the Java platform. He facilitates the message exchange of Java applications and provides a standard interface to simplify development. Typical implementation of ActiveMQ
-
STOMP : Streaming Text Oriented Message Protocol. is a simple text protocol for MOM. STOMP provides an interoperable connection format that allows clients to interact with any STOMP message broker. Typical implementation of ActiveMQ
-
AMQP : Advanced Message Queuing Protocol. An application layer standard that provides unified messaging services and is an open standard for application layer protocols. RabbitMQ is a typical implementation
-
MQTT : Message Queueing Telemetry Transport. An instant messaging protocol (binary protocol) developed by IBM, mainly used for communication between servers and low-power IoT devices
2. Basic concepts
Topic: Indicates a collection of a type of message (which can be understood as the type of message). Each message can only belong to one topic, which is the basic unit of RocketMQ for message subscription. A producer can send multiple Topic messages at the same time, but a consumer can only receive one Topic message
Tag (Tag): used to quickly filter messages
3. Linux deploys RocketMQ service
1. Download the compiled binary compressed package on the official website, version 5.0.0, and upload it to Linux
2. Decompress
3. Configure environment variables ROCKETMQ_HOME and NAMESRV_ADDR
4. Configure runserver.sh in the bin directory, and modify the memory parameters of the JVM according to the actual situation
5. Configure runbroker.sh in the bin directory, and modify the memory parameters of the JVM according to the actual situation
6. Execute the nohup command to run the RocketMQ service in the background (the nameserver must be started first, and the broker needs to be registered on the nameserver)
# 启动nameserver
nohup bin/mqnamesrv &
# 启动broker
nohup bin/mqbroker -c [confFile] & # -c可指定加载的配置文件，默认为conf/broker.conf
# 查看日志rocketmq是否成功启动
tail nohup.out
# 查看进程
jps
# 停止broker
sh bin/mqshutdown broker
# 停止namesrv
sh bin/mqshutdown namesrv
7. Execute the command test (the test sample provided by rocketmq, the producer will send a thousand messages)
bin/tools.sh org.apache.rocketmq.example.quickstart.Producer
8. Execute the command test (the test sample provided by rocketmq, the consumer will accept a thousand messages)
bin/tools.sh org.apache.rocketmq.example.quickstart.Consumer
4. RocketMQ API
Producer sends messages synchronously
public void test_SyncProducer() throws MQClientException {
DefaultMQProducer producer = new DefaultMQProducer("producer_group_name");
//设置注册服务的ip地址的端口
producer.setNamesrvAddr(RocketMQConstant.NAME_SRV_ADDR);
//启动生产者
producer.start();
for(int i=0; i<3; i++){
try {
// 封装消息，设置topic，tag（用于消息快速过滤），消息数据
Message message = new Message(
"TopicTest",
"TagA",
"ID04287777",
("Hello, RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET));
//同步发送消息并获取发送结果，producer从broker获取发送结果
SendResult sendResult = producer.send(message);
System.out.println(sendResult);
Thread.sleep(1500);
} catch (Exception e) {
throw new RuntimeException(e);
}
}
producer.shutdown();
}
Producers send messages asynchronously
public void test_AsyncProducer() throws Exception{
DefaultMQProducer producer = new DefaultMQProducer(RocketMQConstant.PRODUCER_GROUP_NAME);
producer.setNamesrvAddr(RocketMQConstant.NAME_SRV_ADDR);
producer.start();
producer.setRetryTimesWhenSendAsyncFailed(0);
int messageCount = 10;
final CountDownLatch countDownLatch = new CountDownLatch(messageCount);
for(int i=0; i<messageCount; i++){
final int index = i;
// 封装消息，设置topic，tag（用于消息快速过滤），消息数据
Message message = new Message(
"TopicTest",
"TagA",
"ID04287777",
("Hello, RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET));
// 异步发送消息，若broker有响应会调用SendCallback中的方法
producer.send(message, new SendCallback() {
public void onSuccess(SendResult sendResult) {
countDownLatch.countDown();
System.out.println(" Send Message "+ index +" OK: "+sendResult);
}
public void onException(Throwable throwable) {
countDownLatch.countDown();
System.out.println(" Send Message "+ index +" Exception: "+throwable);
}
});
//单向发送
producer.sendOneway(message);
System.out.println("Message "+index+" send done");
}
//在100条消息发送完后关闭
countDownLatch.await(5, TimeUnit.SECONDS);
producer.shutdown();
}
Producer sends messages in one direction
public void test_OneWayProducer() throws Exception{
DefaultMQProducer producer = new DefaultMQProducer(RocketMQConstant.PRODUCER_GROUP_NAME);
producer.setNamesrvAddr(RocketMQConstant.NAME_SRV_ADDR);
producer.start();
producer.setRetryTimesWhenSendAsyncFailed(0);
int messageCount = 10;
final CountDownLatch countDownLatch = new CountDownLatch(messageCount);
for(int i=0; i<messageCount; i++){
final int index = i;
// 封装消息，设置topic，tag（用于消息快速过滤），消息数据
Message message = new Message(
"TopicTest",
"TagA",
"ID04287777",
("Hello, RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET));
//单向发送
producer.sendOneway(message);
System.out.println("Message "+index+" send done");
}
//在100条消息发送完后关闭
countDownLatch.await(5, TimeUnit.SECONDS);
producer.shutdown();
}
consumer push model
public static void test_PushConsumer() throws Exception{
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("consumer_group_name");
consumer.setNamesrvAddr(RocketMQConstant.NAME_SRV_ADDR);
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);
//消费者订阅的消息topic和tag（subExpression，*表示任意）
consumer.subscribe("TopicTest", "*");
consumer.registerMessageListener(new MessageListenerConcurrently() {
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> list, ConsumeConcurrentlyContext consumeConcurrentlyContext) {
System.out.println("Receive New Message : "+list);
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
});
consumer.start();
System.out.println("Consumer Start...");
}
consumer pull model
Different from the push mode consumer, the pull mode needs to manually manage the mapping relationship between MessageQueue and offset. However, the latest underlying source code of LitePullConsumer has realized the management of mq and offset, which is more convenient.
//拉模式消费者
public static void test_LitePullConsumer() throws Exception{
DefaultLitePullConsumer litePullConsumer = new DefaultLitePullConsumer(RocketMQConstant.CONSUMER_GROUP_NAME);
litePullConsumer.setNamesrvAddr(RocketMQConstant.NAME_SRV_ADDR);
litePullConsumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);
litePullConsumer.subscribe("TopicTest", "*");
litePullConsumer.start();
try {
while(true){
List<MessageExt> messageExts = litePullConsumer.poll();
System.out.printf("%s%n", messageExts);
}
}finally {
litePullConsumer.shutdown();
}
}
RocketMQ transfers objects, and the class to which the object belongs needs to implement the serialization interface, and convert the object into a byte array and store it in the message body.
sequential message
Ensure local ordering of messages (ordering of several messages, not necessarily ordering of all messages) to prevent being affected by network transmission.
Realization principle
The producer sends a group of ordered messages to the same MessageQueue at a time (depending on the characteristics of the queue to ensure local order). Consumers will consume the next MessageQueue message after consuming a MessageQueue message.
public class OrderProducer {
public static void main(String[] args) {
DefaultMQProducer producer = new DefaultMQProducer(WanfengConstant.PRODUCER_GROUP_NAME);
try {
producer.setNamesrvAddr(WanfengConstant.NAMESRV_ADDR);
producer.start();
for(int i=0; i<5; i++){
//用于指定顺序的id
int orderId = i;
for(int j=0; j<5; j++){
Message message = new Message(
WanfengConstant.ORDER_TOPIC,
"order_"+orderId,
"KEY"+orderId,
("order_"+orderId+" step "+j).getBytes(RemotingHelper.DEFAULT_CHARSET)
);
//实现消息队列选择器对象，使同一个orderId的消息发送到同一个消息队列
SendResult sendResult = producer.send(
message,
new MessageQueueSelector() {
@Override
public MessageQueue select(List<MessageQueue> mqs, Message msg, Object arg) {
Integer id = (Integer) arg;
int index = id % mqs.size();
return mqs.get(index);
}
},
orderId
);
System.out.printf("%s%n", sendResult);
}
}
}catch(Exception e){
e.printStackTrace();
producer.shutdown();
}
}
}
public class OrderConsumer {
public static void main(String[] args) {
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(WanfengConstant.CONSUMER_GROUP_NAME);
consumer.setNamesrvAddr(WanfengConstant.NAMESRV_ADDR);
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);
try {
consumer.subscribe(WanfengConstant.ORDER_TOPIC, "*");
//实现顺序消息监听者接口
consumer.registerMessageListener(new MessageListenerOrderly() {
@Override
public ConsumeOrderlyStatus consumeMessage(List<MessageExt> msgs, ConsumeOrderlyContext context) {
context.setAutoCommit(true);
for(MessageExt messageExt : msgs){
System.out.println("Receive Message: " + new String(messageExt.getBody()));
}
return ConsumeOrderlyStatus.SUCCESS;
}
});
consumer.start();
System.out.println("Consumer Start...");
} catch (Exception e) {
e.printStackTrace();
consumer.shutdown();
}
}
}
broadcast message
The message sent by the producer is pushed to all consumers of the group
Implementation principle: set the consumer to the MessageModel in broadcast mode.
public class BroadcastConsumer {
public static void main(String[] args) {
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(WanfengConstant.CONSUMER_GROUP_NAME);
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);
//设定消息模式为广播
consumer.setMessageModel(MessageModel.BROADCASTING);
try {
consumer.subscribe(WanfengConstant.ARCHIVE_TOPIC, "*");
consumer.registerMessageListener(new MessageListenerConcurrently() {
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
msgs.forEach(messageExt -> {
Archive archive = (Archive) WanfengObjectUtil.bytesToObject(messageExt.getBody());
System.out.println("Receive Message : "+archive.getId());
});
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
});
consumer.start();
System.out.println("Broadcast Consumer Start...");
}catch (Exception e){
e.printStackTrace();
consumer.shutdown();
}
}
}
If the specified MessageModel is CLUSTERING, the message sent by the producer will randomly specify the consumer for consumption.
delayed message
As the name implies, when the message is sent to the broker, it is delayed for a specified time before being sent to the consumer. Often used for timing sending
filter messages
Filtering messages is implemented by tag, and the filter tag can be specified on the consumer side.
//消费者订阅tag1或tag2的消息
consumer.subscribe("TopicTest", "tag1 || tag2");
In RocketMQ, after the consumer specifies the filter condition, it is pushed up to the Broker, and tag filtering is performed in the Broker to reduce network IO, but it also increases the busyness of the Broker.
transactional message
public class TransactionProducer {
public static void main(String[] args) {
TransactionMQProducer producer = new TransactionMQProducer(WanfengConstant.PRODUCER_GROUP_NAME);
TransactionListener transactionListener = new TransactionListener() {
@Override
public LocalTransactionState executeLocalTransaction(Message msg, Object arg) {
System.out.println("[WANFENG-INFO] TransactionProducer.executeLocalTransaction(): 执行成功...");
String tags = msg.getTags();
if (StringUtils.contains(tags, "TagA")) {
//消息提交（发送出去）
return LocalTransactionState.COMMIT_MESSAGE;
} else if (StringUtils.contains(tags, "TagB")) {
//消息回滚（丢掉消息）
return LocalTransactionState.ROLLBACK_MESSAGE;
} else {
return LocalTransactionState.UNKNOW;
}
}
@Override
public LocalTransactionState checkLocalTransaction(MessageExt msg) {
System.out.println("[WANFENG-INFO] TransactionProducer.checkLocalTransaction(): 执行成功...");
String tags = msg.getTags();
if (StringUtils.contains(tags, "TagC")) {
return LocalTransactionState.COMMIT_MESSAGE;
} else {
return LocalTransactionState.UNKNOW;
}
}
};
ExecutorService executorService = new ThreadPoolExecutor(
2,
5,
100, TimeUnit.SECONDS,
new ArrayBlockingQueue<>(3)
);
producer.setExecutorService(executorService);
producer.setTransactionListener(transactionListener);
try {
producer.start();
} catch (Exception e) {
e.printStackTrace();
}
String[] tags = new String[]{
"TagA", "TagB", "TagC"};
CountDownLatch countDownLatch = new CountDownLatch(9);
for (int i = 0; i < 9; i++) {
try {
Message message = new Message("TopicTest", tags[i % tags.length], "Key" + i, ("Hello RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET));
SendResult sendResult = producer.sendMessageInTransaction(message, null);
System.out.println(sendResult);
Thread.sleep(1000);
countDownLatch.countDown();
} catch (Exception e) {
e.printStackTrace();
}
}
try {
countDownLatch.await();
} catch (InterruptedException e) {
throw new RuntimeException(e);
} finally {
try {
Thread.sleep(100000);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
producer.shutdown();
}
}
}
ACL permission control
ACL controls the user's access to topic resources
Introduce the dependency package of acl in the pom dependency
<dependency>
<groupId>org.apache.rocketmq</groupId>
<artifactId>rocketmq-acl</artifactId>
<version>5.0.0</version>
</dependency>
In the conf/broker.conf file on the server side, add configuration and enable acl
aclEnable=true
In the conf/plain_acl.yml file on the server side, configure specific permission rules (hot loading, no need to restart mq)
accounts:
- accessKey: RocketMQ #用户名
secretKey: 12345678 #密码
whiteRemoteAddress: #访问地址白名单
admin: false #是否为管理员（管理员可以访问所有Topic）
defaultTopicPerm: DENY #默认Topic访问权限
defaultGroupPerm: SUB #默认组权限
topicPerms: #Topic对应的权限，若这里找不到则采用defaultTopicPerm
- topicA=DENY
- topicB=PUB|SUB
- topicC=SUB
groupPerms:
# the group should convert to retry topic
- groupA=DENY
- groupB=PUB|SUB
- groupC=SUB
RPCHook (acl user information) needs to be added when creating a producer object
public class AclProducer {
private static final String ACL_ACCESS_KEY = "RocketMQ";
private static final String ACL_SECRET_KEY = "12345678";
/**
* 通过用户名和密码获取RPCHook
* @return
*/
public static RPCHook getAclRPCHook(){
return new AclClientRPCHook(new SessionCredentials(ACL_ACCESS_KEY, ACL_SECRET_KEY));
}
public static void main(String[] args) throws MQClientException, InterruptedException {
//创建生产者时加入用户信息，即RPCHook
DefaultMQProducer producer = new DefaultMQProducer(WanfengConstant.PRODUCER_GROUP_NAME, getAclRPCHook());
producer.setNamesrvAddr(WanfengConstant.NAMESRV_ADDR);
producer.start();
for (int i = 0; i < 20; i++) {
try {
Message message = new Message(
"TopicTest",
WanfengConstant.TAGS_NAME,
("Hello RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET) /*消息体转换成二进制数组*/
);
SendResult sendResult = producer.send(message);
System.out.printf("%s%n", sendResult);
} catch (Exception e) {
e.printStackTrace();
Thread.sleep(1000);
}
}
}
}
message track
Producer, Consumer, and Broker process information about messages
The realization principle of the message track is that MQ puts the message track in the Topic of RMQ_SYS_TRACE_TOPIC
Open the message track in the Broker configuration file
traceTopicEnable=true
When creating a producer, specify the enableMsgTrace parameter as true to enable message traces. You can also specify the customizedTraceTopic parameter to customize the topic of the message trace.
public class TraceProducer {
public static void main(String[] args) throws MQClientException {
//指定enableMsgTrace参数为true，开启消息轨迹
DefaultMQProducer producer = new DefaultMQProducer(WanfengConstant.PRODUCER_GROUP_NAME, true);
producer.setNamesrvAddr(WanfengConstant.NAMESRV_ADDR);
producer.start();
for (int i = 0; i < 20; i++) {
try {
Message message = new Message(
"TopicTest",
WanfengConstant.TAGS_NAME,
("Hello RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET) /*消息体转换成二进制数组*/
);
SendResult sendResult = producer.send(message);
System.out.printf("%s%n", sendResult);
} catch (Exception e) {
e.printStackTrace();
}
}
}
}