RocketMQ message retry mechanism and power

A. Retry mechanism

　　1. Since MQ often in complex distributed systems, regardless of network fluctuations, service downtime, program abnormal factors, most failed to send a message or consumption problems may occur. Therefore, the retry is all MQ middleware must take into account a key point of the message. If there is no message retry, message loss may arise problems that may have a significant impact on the system. So, Bing Chengning be more than a message, the message can not be lost in principle, most of all MQ message retry provide good support.

　　Encapsulating the message to the user 2.RocketMQ retry processing flow, the developer without manual handling. RocketMQ support the retry mechanism for the production side and the consumer side categories.

Analog abnormal

　　Consumer news consumption end of two states:

 

package com.alibaba.rocketmq.client.consumer.listener;

public enum ConsumeConcurrentlyStatus {
    CONSUME_SUCCESS,
    RECONSUME_LATER;

    private ConsumeConcurrentlyStatus() {
    }
}

 

　　It is a success (CONSUME_SUCCESS), & retry a failed (RECONSUME_LATER);

　　Consumer consumption in order to ensure the success of the message, only to use side made it clear that the consumer succeeds, the return CONSUME_SUCCESS, RocketMQ would think the message consumer success.

　　If the message consumer fails, simply return ConsumeConcurrentlyStatus.RECONSUME_LATER, RocketMQ consumer will think that the message has failed, need to re-delivery.

　　1. abnormal

 

package com.wn.consumer;

import com.alibaba.rocketmq.client.consumer.DefaultMQPushConsumer;
import com.alibaba.rocketmq.client.consumer.listener.ConsumeConcurrentlyContext;
import com.alibaba.rocketmq.client.consumer.listener.ConsumeConcurrentlyStatus;
import com.alibaba.rocketmq.client.consumer.listener.MessageListenerConcurrently;
import com.alibaba.rocketmq.client.exception.MQClientException;
import com.alibaba.rocketmq.common.message.MessageExt;

import java.util.List;

public class MQConsumer {
    public static void main(String[] args) throws MQClientException {
        // Create the customer
        DefaultMQPushConsumer consumer=new DefaultMQPushConsumer("rmq-group");
        // set NameServer address
        consumer.setNamesrvAddr("192.168.138.187:9876;192.168.138.188:9876");
        // Set consumers instance name
        consumer.setInstanceName("consumer");
        // Subscribe topic
        consumer.subscribe("wn02","TagA");
        // listen for messages
        consumer.registerMessageListener(new MessageListenerConcurrently() {
            @Override
            public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> list, ConsumeConcurrentlyContext consumeConcurrentlyContext) {
                // Get the message
                for (MessageExt msg:list){
                    System.out.println(msg.getMsgId()+"---"+new String(msg.getBody()));
                }

                try {
                    int i=1/0;
                }catch (Exception e){
                    e.printStackTrace ();
                    // needs to be retried
                    return ConsumeConcurrentlyStatus.RECONSUME_LATER;
                } 
// Message success
                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
            }
        });
        consumer.start();
        System.out.println("Consumer Started...");

    }
}

 

　　2. network latency

 

package com.wn.consumer;

import com.alibaba.rocketmq.client.consumer.DefaultMQPushConsumer;
import com.alibaba.rocketmq.client.consumer.listener.ConsumeConcurrentlyContext;
import com.alibaba.rocketmq.client.consumer.listener.ConsumeConcurrentlyStatus;
import com.alibaba.rocketmq.client.consumer.listener.MessageListenerConcurrently;
import com.alibaba.rocketmq.client.exception.MQClientException;
import com.alibaba.rocketmq.common.message.MessageExt;

import java.util.List;

public class MQConsumer {
    public static void main(String[] args) throws MQClientException {
        // Create the customer
        DefaultMQPushConsumer consumer=new DefaultMQPushConsumer("rmq-group");
        // set NameServer address
        consumer.setNamesrvAddr("192.168.138.187:9876;192.168.138.188:9876");
        // Set consumers instance name
        consumer.setInstanceName("consumer");
        // Subscribe topic
        consumer.subscribe("wn03","TagA");
        // listen for messages
        consumer.registerMessageListener(new MessageListenerConcurrently() {
            @Override
            public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> list, ConsumeConcurrentlyContext consumeConcurrentlyContext) {
                // Get the message
                for (MessageExt msg:list){
                    System.out.println(msg.getMsgId()+"---"+new String(msg.getBody()));
                }

                // network latency
                try {
                    Thread.sleep(600000);
                } catch (InterruptedException e) {
                    e.printStackTrace ();
                }
// Message success
                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
            }
        });
        consumer.start();
        System.out.println("Consumer Started...");

    }
}

 

Second, the message idempotent

1, under what circumstances the message repeated consumption will occur RocketMQ

　　 ①, when the link system calls relatively long time, such as system call A system B, system B then transmits a message to the RocketMQ, when the system call A system B, system B if the process is successful, but has yet to be successful results back to the calling system a, the system will attempt to re-initiate requests a to system B, B causes the system repeats the process, launched several messages to RocketMQ caused by repeated consumption;

　　②, when RocketMQ sent to the system B, the above is also possible and the same problem occurs, the message send the timeout, retry phalanx system B, resulting in the re-read RocketMQ received message;

　　③, when RocketMQ successfully received the message, and the message to the consumer process, not enough time to complete the submission if consumer spending to offset RocketMQ, myself down or restart, then RocketMQ not received offset, it will consider consumption failed, will resend the message to consumers spending again;

2, how to solve the message repeated consumption

　　By idempotency to ensure that, as long as the message does not duplicate the results of an impact, it is the perfect solution to this problem.

In the end the producers guarantee idempotency, about in two ways:

　　①, RocketMQ support query message, just go RocketMQ inquiries about whether the article has been sent a message on it, does not exist is sent, there is not sent;

　　②, Redis introduced, after transmitting the message to RocketMQ successful, the data is inserted into a Redis, if transmission retry, then go Redis query whether a message has been sent over the article, is not present, the message is repeatedly transmitted;

　　Method One: Performance RocketMQ message queries is not particularly good, if under high concurrency scenarios, each message when sending to RocketMQ went to check what may affect the performance of the interface;

　　Method 2: In some extreme scenarios, Redis can not guarantee success after the message is sent, it will be able to write Redis success, such as writing a message Redis success this time down, then the query Redis again to determine whether the message has been sent, You can not get the right result;

3, producer

 

package com.zn.idempotent;

import com.alibaba.rocketmq.client.exception.MQBrokerException;
import com.alibaba.rocketmq.client.exception.MQClientException;
import com.alibaba.rocketmq.client.producer.DefaultMQProducer;
import com.alibaba.rocketmq.client.producer.SendResult;
import com.alibaba.rocketmq.common.message.Message;
import com.alibaba.rocketmq.remoting.exception.RemotingException;

/**
 * Message and other power producers
 */
public class IdempotentProvider {
    public static void main(String[] args) throws MQClientException, InterruptedException, RemotingException, MQBrokerException {
        // Create a Producer
        DefaultMQProducer producer=new DefaultMQProducer("rmq-group");
        // set NameServer address
        producer.setNamesrvAddr("192.168.33.135:9876;192.168.33.136:9876");
        // set the instance name producers
        producer.setInstanceName("producer");
        // start producer
        producer.start();

            //send messages
            for (int i=1;i<=1;i++){
                // simulate network delay, send every second MQ
                Thread.sleep(1000);
                // create a message, topic topic name tags provisional value represents small class, body representative of the message body
                Message message=new Message("itmayiedu-topic03","TagA",("itmayiedu-"+i).getBytes());
                // uniquely identifies the message
                message.setKeys ( "Order Message:" + i);
                //send messages
                SendResult sendResult=producer.send(message);
                System.out.println ( "information is power and so the question is:" + sendResult.toString ());
            }
        producer.shutdown();
    }
}

 

4, consumer

 

package com.zn.idempotent;

import com.alibaba.rocketmq.client.consumer.DefaultMQPushConsumer;
import com.alibaba.rocketmq.client.consumer.listener.ConsumeConcurrentlyContext;
import com.alibaba.rocketmq.client.consumer.listener.ConsumeConcurrentlyStatus;
import com.alibaba.rocketmq.client.consumer.listener.MessageListenerConcurrently;
import com.alibaba.rocketmq.client.exception.MQClientException;
import com.alibaba.rocketmq.common.message.MessageExt;

import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.logging.LogManager;
import java.util.logging.Logger;

/**
 * Message and other power consumers
 */
public class IdempotentConsumer {

    static private Map<String, Object> logMap = new HashMap<>();

    public static void main(String[] args) throws MQClientException {
        // Create the customer
        DefaultMQPushConsumer consumer=new DefaultMQPushConsumer("rmq-group");
        // set NameServer address
        consumer.setNamesrvAddr("192.168.33.135:9876;192.168.33.136:9876");
        // set the instance name
        consumer.setInstanceName("consumer");
        // Subscribe topic
        consumer.subscribe("itmayiedu-topic03","TagA");

        // listen for messages
        consumer.registerMessageListener(new MessageListenerConcurrently() {
            @Override
            public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> list, ConsumeConcurrentlyContext consumeConcurrentlyContext) {
                String key=null;
                String msgId=null;

                    for (MessageExt messageExt:list){
                        key=messageExt.getKeys();
                        // interpretation redis there is no current message key
                        if (logMap.containsKey(key)) {
                            // no need to continue to retry.
                            System.out.println ( "key:" + key + ", the consumer has no need to retry ...");
                            return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
                        }
                        // RocketMQ Because it is a clustered environment, so the message ID generated may be repeated
                        msgId = messageExt.getMsgId();
                        System.out.println("key:" + key + ",msgid:" + msgId + "---" + new String(messageExt.getBody()));
                        // the current key stored in the redis
                        logMap.put(messageExt.getKeys(),messageExt);
                    }
                try {
                    int i=5/0;
                }catch (Exception e){
                    e.printStackTrace ();
                    // labor compensation
                    return ConsumeConcurrentlyStatus.RECONSUME_LATER;
                }
                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
        }
    });
        // Start Consumers
        consumer.start();
        System.out.println("Consumer Started!");
    }
}