After the configuration and demo of rocketmq are done, there are still many problems to be solved before going online.
Such as: How to integrate into the project, and make it standardized and easy to use. What problems need to be considered in use, and how to solve the problems of monitoring operation and maintenance.
This article first addresses the client-side integration, and the final code and configuration after my trial and error are posted below.
The code and configuration posted in this article have been repeatedly tested and verified, and are used in actual projects. Currently, only a few important parameters are used. For more detailed configuration, please refer to the official documentation.
pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>rockmqtest</groupId>
<artifactId>rockmqtest</artifactId>
<version>0.0.1-SNAPSHOT</version>
<properties>
<spring.version>3.1.1.RELEASE</spring.version>
<slf4j.version>1.7.13</slf4j.version>
</properties>
<dependencies>
<!-- RocketMQ Java SDK -->
<dependency>
<groupId>com.alibaba.rocketmq</groupId>
<artifactId>rocketmq-client</artifactId>
<version>3.5.8</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
<version>3.1.1.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-test</artifactId>
<version>3.1.1.RELEASE</version>
</dependency>
<dependency>
<groupId>ch.ethz.ganymed</groupId>
<artifactId>ganymed-ssh2</artifactId>
<version>build209</version>
</dependency>
<dependency>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
<version>2.6</version>
</dependency>
<dependency>
<groupId>com.jcraft</groupId>
<artifactId>jsch</artifactId>
<version>0.1.46</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>jcl-over-slf4j</artifactId>
<version>1.5.8</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.5.8</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.5.8</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.14</version>
</dependency>
</dependencies>
</project>
Configuration:
<!-- The same topic shares a producer, different topics do not share, to prevent performance problems caused by fast sending speed groupName= topic+producer--> <bean id="mqProducer" class="com.sfbest.rocketmq.online.MQProducer" init-method="start"> <constructor-arg> <value>FirstGroupName</value> </constructor-arg> <!-- the subject of the target --> <property name="topic" value="TopicTest1"></property> <property name="namesrvAddr" value="10.103.16.77:9876"></property> </bean> <!-- rocketMq consumer configuration default cluster mode consumption --> <!-- A topic represents a big business tag represents a small business such as updateA, updateB. The same topic shares a consumer consumerGroupName=topic+Consumer--> <bean id="mqPushConsumer" class="com.sfbest.rocketmq.online.MQConsumer" init-method="init"> <!-- Consumer group name, the same business group name is the same, different group names are different --> <property name="consumerGroupName" value="firstSpringConsumer"></property> <!-- nameserver address and port number--> <property name="namesrvAddr" value="10.103.16.77:9876"></property> <!-- SUBSCRIBE TOPIC--> <property name="topic" value="TopicTest1"></property> <!-- The mark * of the subscription topic indicates all TAG--> <property name="tagExpr" value="*"></property> <property name="mqBusinessMap"> <map> <entry key="TagA" value-ref="mqBusinessTestA"/> <entry key="TagB" value-ref="mqBusinessTestB"/> </map> </property> </bean> <!-- Customize the specific business implementation class--> <bean id="mqBusinessTestA" class="com.sfbest.rocketmq.online.MQBusinessTestAImpl"></bean> <bean id="mqBusinessTestB" class="com.sfbest.rocketmq.online.MQBusinessTestBImpl"></bean>
Client implementation to consider, scalability, ease of use, performance, specifications. And it needs to be written according to the characteristics of rocketMq. Let's talk about my implementation.
Key parameters can be configured, design naming conventions and interface specifications.
The same topic shares a producer, and different topics do not share, preventing performance problems caused by fast sending.
The same topic shares a tag that is not used by the cosumer for distribution, and reserves the interface for business implementation.
Producer client public class
/**
* defaultMQProducer decorator
* Encapsulate an easy-to-use sending method
* @author chenchangqun
*
*/
public class BestMQProducer extends DefaultMQProducer{
private String topic;
private static final Logger LOG = LoggerFactory.getLogger(BestMQConsumer.class);
public BestMQProducer(String producerGroupName){
super(producerGroupName);
//default parameter settings
}
public void init(){
try {
super.start();
} catch (MQClientException e) {
LOG.error("rockmq producer start fail",e);
}
}
/**
* Sending method
* @author chenchangqun
* @param key
* @param content
* @return
*/
public boolean sendMsg(String tag,String key,String content){
Message msg=null; // body
try {
msg = new Message(topic, // topic
day, // day
key, // key
content.getBytes("UTF-8"));
} catch (Exception e) {
LOG.error("create message fail ,cannot send msg ,invoke end 。topic={},tag={},key={},content={}",topic,tag,key,content,e);
return false;
}
try {
// If not initialized, initialize
if(this.defaultMQProducerImpl.getServiceState()==ServiceState.CREATE_JUST){
LOG.info("mqProducer should be start");
this.init();
}
LOG.debug("mq send param tag:{},key:{},content:{}",tag,key,content);
SendResult sendResult = super.send(msg);
LOG.debug("sendResult:{}",sendResult.getSendStatus().toString());
// sendResult.getSendStatus()==SendStatus.
// System.out.println("sendResult:{}"+sendResult);
return sendResult.getSendStatus()==SendStatus.SEND_OK;
} catch (MQClientException e) {
LOG.error("send fail",e);
} catch (RemotingException e) {
LOG.error("send fail",e);
} catch (MQBrokerException e) {
LOG.error("send fail",e);
} catch (InterruptedException e) {
LOG.error("send fail",e);
}
return false;
}
public String getTopic() {
return topic;
}
public void setTopic(String topic) {
this.topic = topic;
}
}
There are two considerations for the implementation of producer
(1) Override the start of the parent class to facilitate adding business logic at startup in the future.
(2) Provide a standardized sending method, the parameters include tag, key, content, because the tocpic of the same producer is determined. The remaining parameters need to be passed. Can topic parameters be provided? I don't think it should. If there are more sending methods in the future, passing the topic directly will lead to poor code readability and high coupling. May also cause performance issues.
Consumer client public class
/**
* mq singleton factory class, which sets some default parameters
*
* @author chenchangqun
*
*/
public class BaseConsumer {
private String nameServAddr;
private static DefaultMQPushConsumer consumer ;
private BaseConsumer() {
}
public static DefaultMQPushConsumer getDefaultMQPushConsumer(String consumerName){
if(consumer==null){
consumer = new DefaultMQPushConsumer(consumerName);
consumer.setMessageModel(MessageModel.CLUSTERING);
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);
}
return consumer;
}
public String getNameServAddr() {
return nameServAddr;
}
public void setNameServAddr(String nameServAddr) {
this.nameServAddr = nameServAddr;
}
}
Note the section below
consumer.setMessageModel(MessageModel.CLUSTERING);
An important feature of rocketMQ is to support cluster consumption. What is cluster consumption means that a business has multiple implementations. We usually hope that after one instance is successfully consumed, other instances will not be consumed. The above configuration is cluster consumption.
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);
First explain a misunderstanding: if a message is successfully consumed by my application, is the message useless and can be killed or discarded. Answer: No, in terms of broker, you are only one of the subscribers, this message may be subscribed by other consumers, and the broker can only record your consumption progress.
offset is the important name that marks the message position
How do I know how much has been consumed and how many messages are left unconsumed? Answer: The broker does not count this. It can only be calculated by offset subtraction, that is, the currently produced offset minus the last cosumet offset.
To start consumption from the last consumption position, configure as above.
Consumer client public class
* mq parameter setting and initialization
* Create listeners and distribute them according to different TAGs
* @author chenchangqun
*
*/
public class BestMQConsumer {
private static final Logger LOG = LoggerFactory.getLogger(BestMQConsumer.class);
private String topic;
private String tagExpr;
private String namesrvAddr;
private Map<String,IMQBusiness> mqBusinessMap;
private String consumerGroupName;
//int count =0;
public void init(){
// Get message producer
DefaultMQPushConsumer consumer = BaseConsumer.getDefaultMQPushConsumer(consumerGroupName);
// Subscription subject
try {
consumer.setNamesrvAddr(namesrvAddr);
consumer.subscribe(topic, tagExpr);
consumer.registerMessageListener( new MessageListenerConcurrently(){
//Consume multiple msgs at a time, if the first one is successful and the second one fails, return success and failure?
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs , ConsumeConcurrentlyContext context) {
for(MessageExt msg:msgs){
if(msg.getTags()!=null){
IMQBusiness mqBusiness= mqBusinessMap.get(msg.getTags());
if(mqBusiness!=null){
if(! mqBusiness.invokeMessage(msg)){
LOG.error("cosume fail return RECONSUME_LATER");
return ConsumeConcurrentlyStatus.RECONSUME_LATER;
}
}
}
}
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
}
);
/**
* The Consumer object must be initialized by calling start before it is used, and it can be initialized once <br>
*/
consumer.start();
System.out.println("spring Consumer first Started.");
} catch (MQClientException e) {
LOG.error("cousumner init fail",e);
}
}
public void setTopic(String topic) {
this.topic = topic;
}
public void setTagExpr(String tagExpr) {
this.tagExpr = tagExpr;
}
public void setNamesrvAddr(String namesrvAddr) {
this.namesrvAddr = namesrvAddr;
}
public void setConsumerGroupName(String consumerGroupName) {
this.consumerGroupName = consumerGroupName;
}
public void setMqBusinessMap(Map<String, IMQBusiness> mqBusinessMap) {
this.mqBusinessMap = mqBusinessMap;
}
}
interface class
/**
* Business interface for consuming MQ
* @author chenchangqun
*
*/
public interface IMQBusiness {
public boolean invokeMessage(MessageExt msg);
}
interface implementation class
public class MQBusinessTestAImpl implements IMQBusiness {
@Override
public boolean invokeMessage(MessageExt msg) {
System.out.println(" invoke A rec:"+new String(msg.getBody()));
return true;
}
}
The cousumer uses the PUSH method, which is passive notification. In fact, the implementation uses a long connection.
The above cosumer code has two main points, one defect
(1) After testing, if CONSUME_SUCCESS is not returned, it will always retry.
(2) Implemented tag-based distribution, which is convenient for business customization.
defect
The number of pieces consumed each time is not configured, and the default is used to consume one piece at a time. If multiple pieces of msg are consumed at a time, if the first one succeeds and the second fails, return success or failure? , this situation also needs to be handled according to the specific business situation
Pay attention to my attachment, which contains the complete code and test classes. The documents in the e-book are collected by me with great effort, and should be the best rocketmq documents and learning materials at present.