Push is the server (MQ) actively pushing messages to the client. The advantage is that it is timely . However, if the client does not do flow control well, once the server pushes a large number of messages to the client, it will cause the client to accumulate messages or even collapse.
Pull means that the client needs to take the initiative to retrieve data from the server (MQ). The advantage is that the client can consume according to its own consumption capacity, but the frequency of pulling also needs to be controlled by the user. Frequent pulling can easily cause pressure on the server and client. , long pull intervals can easily lead to untimely consumption.

Push mode is also based on Pull mode, so whether it is Push mode or Pull mode, it is Pull mode. Under normal circumstances, the Pull mode is preferred

2. Send message

1. Ordinary news

Sync message(***)

After the synchronization message is sent, there will be a return value, which is a confirmation returned by the mq server after receiving the message. This method is very safe, but the performance is not so high, and in the mq cluster, it also has to wait until all slave machines The message will be returned only after all the messages have been copied, so you can choose this method for important messages.

Reliable synchronous transmission is widely used in various scenarios, such as important notification messages, short message notifications, etc.

Native dependency introduction:

        <!--  原生api,不是starter      -->
        <dependency>
            <groupId>org.apache.rocketmq</groupId>
            <artifactId>rocketmq-client</artifactId>
            <version>4.9.0</version>
        </dependency>

Synchronous message producer:

public class Producer {
    public static void main(String[] args) throws Exception {
            /*
             1. 谁来发？
             2. 发给谁？
             3. 怎么发？
             4. 发什么？
             5. 发的结果是什么？
             6. 关闭连接
             **/
            //1.创建一个发送消息的对象Producer，并指定生产者组名
            DefaultMQProducer producer = new DefaultMQProducer("sync-producer-group");
            //2.设定发送的命名服务器地址
            producer.setNamesrvAddr("ip:9876");
            producer.setSendMsgTimeout(1000000);

            //3.1启动发送的服务
            producer.start();
            //4.创建要发送的消息对象,指定topic，指定内容body
            Message msg = new Message("sync-topic", "hello-rocketmq".getBytes(StandardCharsets.UTF_8));
            //3.2发送消息
            SendResult result = producer.send(msg);
            System.out.println("返回结果：" + result);
            //5.关闭连接
            producer.shutdown();
    }
}

Synchronous message consumer:

public class Consumer {
    public static void main(String[] args) throws Exception {
        //1.创建一个接收消息的对象Consumer，并指定消费者组名
        //两种模式：①消费者定时拉取模式  ②建立长连接让Broker推送消息（选择第二种）
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("sync-producer-group");
        //2.设定接收的命名服务器地址
        consumer.setNamesrvAddr("ip:9876");
        //3.订阅一个主题，* 表示订阅这个主题的所有消息，后期会有消息过滤
        consumer.subscribe("sync-topic","*");
        //设置当前消费者的消费模式（默认模式：负载均衡）
        consumer.setMessageModel(MessageModel.CLUSTERING);
        //3.设置监听器，用于接收消息（一直监听，异步回调，异步线程）
        consumer.registerMessageListener(new MessageListenerConcurrently() {
            @Override
            //消费消息
            //消费上下文:consumeConcurrentlyContext
            public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> list, ConsumeConcurrentlyContext consumeConcurrentlyContext) {
                // 这个就是消费的方法 （业务处理）
                System.out.println("我是消费者");
                System.out.println(msgs.get(0).toString());
                System.out.println("消息内容:" + new String(msgs.get(0).getBody()));
                System.out.println("消费上下文:" + context);

                //签收消息，消息会从mq出队
                //如果返回 RECONSUME_LATER 或 null 或 产生异常 那么消息会重新 回到队列 过一会重新投递出来 ，给当前消费者或者其他消费者消费的
                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
            }
        });
        //4.启动接收消息的服务
        consumer.start();
        System.out.println("接受消息服务已经开启！");
        //5 不要关闭消费者！因为需要监听！
        //挂起
        System.in.read();
    }
}

The message MessageExt consists of: message header (basic attributes of the message) and message body

Asynchronous messages (***)

Asynchronous messages are usually used in business scenarios that are sensitive to response time , that is, the sender cannot tolerate waiting for a long time for the Broker's response. After sending, there will be an asynchronous message notification.

For example, after the video is uploaded, you will be notified to start the transcoding service, and after the transcoding is completed, you will be notified to push the transcoding results, etc.

Asynchronous message producer:

public class AsyncProducer {
    public static void main(String[] args) throws Exception {
        DefaultMQProducer producer = new DefaultMQProducer("async-producer-group");
        producer.setNamesrvAddr("ip:9876");
        producer.start();
        Message message = new Message("async-topic", "我是一个异步消息".getBytes());
        //没有返回值的
        producer.send(message, new SendCallback() {
            @Override
            public void onSuccess(SendResult sendResult) {
                System.out.println("发送成功");
            }

            @Override
            public void onException(Throwable e) {
                System.err.println("发送失败:" + e.getMessage());
            }
        });
        System.out.println("我先执行");
        //需要接收异步回调，这里需要挂起
        System.in.read();
    }
}

No special changes for consumers:

public class SimpleConsumer {
    public static void main(String[] args) throws Exception{
        // 创建一个消费者
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("async-producer-group");
        // 连接namesrv
        consumer.setNamesrvAddr("ip:9876");
        // 订阅一个主题  * 标识订阅这个主题中所有的消息  后期会有消息过滤
        consumer.subscribe("async-topic", "*");
        // 设置一个监听器 (一直监听的， 异步回调方式)
        consumer.registerMessageListener(new MessageListenerConcurrently() {
            @Override
            public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
                // 这个就是消费的方法 （业务处理）
                System.out.println("我是消费者");
                System.out.println(msgs.get(0).toString());
                System.out.println("消息内容:" + new String(msgs.get(0).getBody()));
                System.out.println("消费上下文:" + context);
                // 返回值  CONSUME_SUCCESS成功，消息会从mq出队
                // RECONSUME_LATER（报错/null） 失败 消息会重新回到队列 过一会重新投递出来 给当前消费者或者其他消费者消费的
                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
            }
        });
        // 启动
        consumer.start();
        // 挂起当前的jvm
        System.in.read();
    }
}

One-way message(*)

This method is mainly used in scenarios where you don’t care about sending results. This method has a high throughput, but there is a risk of message loss. It is generally used in scenarios where the results are not important , such as sending log information.

One-way message producer:

public class SingleWayProducer {
    public static void main(String[] args) throws Exception{
        // 创建默认的生产者
        DefaultMQProducer producer = new DefaultMQProducer("single-way-producer-group");
        // 设置nameServer地址
        producer.setNamesrvAddr("ip:9876");
        // 启动实例
        producer.start();
        Message msg = new Message("single-way-topic", ("单向消息").getBytes());
        // 发送单向消息
        producer.sendOneway(msg);
        // 关闭实例
        producer.shutdown();
    }

}

Ideas for writing log services

The service that generates logs uses MQ to send one-way messages without waiting for a reply, which greatly reduces the time for sending logs. The log-service writes them uniformly into the log table. And because the logs are too large, the hot and cold logs can be separated . The logs in the past month are hot data, and the logs in the past year are cold data (the actual situation depends on the business). The storage locations are different, and logs that are too old can be deleted. Lose

2. Delayed message(***)

After the message is put into MQ, it will take a while before it will be monitored and then consumed.

For example, in the order business, you can send a delayed message after submitting an order. Check the status of the order after 15 minutes. If payment is still not made, cancel the order and release inventory (order timeout ) .

In scenarios such as distributed scheduled scheduling triggering and task timeout processing, using RocketMQ's delayed messages can simplify the development logic of scheduled scheduling tasks and achieve high-performance, scalable, and highly reliable scheduled triggering capabilities.

Latency level

Delay message producer:

public class DelayProducer {
    public static void main(String[] args) throws Exception{
        // 创建默认的生产者
        DefaultMQProducer producer = new DefaultMQProducer("delay-producer-group");
        // 设置nameServer地址
        producer.setNamesrvAddr("ip:9876");
        // 启动实例
        producer.start();
        Message msg = new Message("delay-topic", ("延迟消息").getBytes());
        // 给这个消息设定一个延迟等级
        // messageDelayLevel = "1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h
        msg.setDelayTimeLevel(3);
        // 发送单向消息
        producer.send(msg);
        // 打印时间
        System.out.println(new Date());
        // 关闭实例
        producer.shutdown();
    }
}

Delayed message consumer (no special changes):

public class MSConsumer {
    public static void main(String[] args) throws Exception{
        // 创建一个消费者
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("delay-producer-group");
        // 连接namesrv
        consumer.setNamesrvAddr("ip:9876");
        // 订阅一个主题  * 标识订阅这个主题中所有的消息  后期会有消息过滤
        consumer.subscribe("delay-topic", "*");
        // 设置一个监听器 (一直监听的， 异步回调方式)
        consumer.registerMessageListener(new MessageListenerConcurrently() {
            @Override
            public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
                System.out.println(msgs.get(0).toString());
                System.out.println("消息内容:" + new String(msgs.get(0).getBody()));
                System.out.println("收到时间："+new Date());
                // 返回值  CONSUME_SUCCESS成功，消息会从mq出队
                // RECONSUME_LATER（报错/null） 失败 消息会重新回到队列 过一会重新投递出来 给当前消费者或者其他消费者消费的
                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
            }
        });
        // 启动
        consumer.start();
        // 挂起当前的jvm
        System.in.read();
    }
}

You can check it by printing the time difference (it is normal to have an error the first time)

3. Batch messages

Rocketmq can send a group of messages at one time, and this group of messages will be consumed as one message.

When there are certain requirements on the throughput rate , some messages can be grouped into a batch and sent later, which can increase the throughput rate and reduce the number of API and network calls.

Pack the message into Collection<Message> msgs and pass it into the method. It should be noted that the size of the batch message cannot exceed 1MiB (otherwise it needs to be divided by itself). Secondly, the topics in the same batch must be the same.

Batch message producer:

public class BatchProducer {
    public static void main(String[] args) throws Exception{
        // 创建默认的生产者
        DefaultMQProducer producer = new DefaultMQProducer("batch-producer-group");
        // 设置nameServer地址
        producer.setNamesrvAddr("ip:9876");
        // 启动实例
        producer.start();
        List<Message> msgs = Arrays.asList(
                //需要是同一种主题
                new Message("batch-topic", "我是一组消息的A消息".getBytes()),
                new Message("batch-topic", "我是一组消息的B消息".getBytes()),
                new Message("batch-topic", "我是一组消息的C消息".getBytes())

        );
        SendResult send = producer.send(msgs);
        System.out.println(send);
        // 关闭实例
        producer.shutdown();
    }
}

By looking at the panel, we can find that all batch messages are placed in one queue.

Batch message consumer (no special changes):

public class BatchConsumer {
    public static void main(String[] args) throws Exception{
        // 创建默认消费者组
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("batch-producer-group");
        // 设置nameServer地址
        consumer.setNamesrvAddr("ip:9876");
        // 订阅一个主题来消费   表达式，默认是*
        consumer.subscribe("batch-topic", "*");
        // 注册一个消费监听 MessageListenerConcurrently是并发消费
        // 默认是20个线程一起消费，可以参看 consumer.setConsumeThreadMax()
        consumer.registerMessageListener(new MessageListenerConcurrently() {
            @Override
            public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
                // 这里执行消费的代码 默认是多线程消费
                System.out.println("msgs.size():"+msgs.size());
                System.out.println(Thread.currentThread().getName() + "----" + new String(msgs.get(0).getBody()));
                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
            }
        });
        consumer.start();
        System.in.read();
    }
}

In addition, there is only one List<MessageExt> msgs parameter in the consumeMessage method at a time, because MessageListenerConcurrently defaults to concurrent consumption mode.

The print result is as follows:

4. Sequential messages (*)

By default, ordinary messages are sent in Round Robin polling mode to send messages to different queues (partition queues); when consuming messages, messages are pulled from multiple queues, which does not satisfy the production sequence and consumption. Sequentiality. For example, if an order is generated, paid and shipped, these three operations need to be executed sequentially. However, if it is a normal message, the message of order A may be polled and sent to different queues, and the messages of different queues will not be maintained. order.

Message ordering means that messages can be consumed in the order in which they are sent (FIFO) . Message ordering can be divided into: partition ordering or global ordering .

Global ordering: There is only one queue involved in sending and consuming. That is, the sequential messages sent are controlled to be sent to the same queue in sequence, and they are only pulled from this queue in sequence when consuming. To ensure global order, the number of read and write queues of the Topic needs to be set to 1, and then the concurrency setting of the producer and consumer is also 1, and multi-threading cannot be used. So in this case, high concurrency and high throughput functions are completely useless.
Partition order: There are multiple queues participating in sending and consuming. But relative to each queue, messages are ordered. For example, the generation, payment and delivery messages of order A are placed in the same queue (such as queue1) because the order number is the same (setting standard). The messages of the three operations of order B will also be in the same queue because of this standard. in, but not necessarily queue1, because we want them to be spread across different queues to improve throughput.

Program simulation, encapsulating entity class MsgModel:

@Data
@AllArgsConstructor
@NoArgsConstructor
public class MsgModel {

    private String orderSn;
    private Integer userId;
    private String desc; // 下单 付款 发货

}

Sequential message producer:

public class OrderProducer {
    private static final List<MsgModel> msgModels = Arrays.asList(
            new MsgModel("qwer", 1, "下单"),
            new MsgModel("qwer", 1, "付款"),
            new MsgModel("qwer", 1, "发货"),
            new MsgModel("zxcv", 2, "下单"),
            new MsgModel("zxcv", 2, "付款"),
            new MsgModel("zxcv", 2, "发货")
    );

    public static void main(String[] args) throws Exception{
        DefaultMQProducer producer = new DefaultMQProducer("orderly-producer-group");
        producer.setNamesrvAddr("ip:9876");
        producer.start();
        // 发送顺序消息  发送时要确保有序 并且要发到同一个队列下面去
        msgModels.forEach(msgModel -> {
            Message message = new Message("orderly-topic", msgModel.toString().getBytes());
            try {
                // 发 相同的订单号去相同的队列
                //MessageQueueSelector() 消息队列选择器
                producer.send(message, new MessageQueueSelector() {
                    @Override
                    //Object arg 即为 select的第三个参数 msgModel.getOrderSn()
                    //作为消息发送分区的分类标准
                    public MessageQueue select(List<MessageQueue> mqs, Message msg, Object arg) {
                        // 在这里 选择队列
                        // 保证 订单号 相同的消息在同一个 queue
                        int hashCode = arg.toString().hashCode();
                        // 周期性函数
                        int i = hashCode % mqs.size();
                        return mqs.get(i);
                    }
                }, msgModel.getOrderSn());

            } catch (Exception e) {
                e.printStackTrace();
            }
        });
        producer.shutdown();
        System.out.println("发送完成");
    }
}

Let's take a closer look at the interface of MessageQueueSelector:

public interface MessageQueueSelector {
    MessageQueue select(final List<MessageQueue> mqs, final Message msg, final Object arg);
}

Among them , mqs is the queue that can be sent, msg is the message, arg is the Object object (the third parameter) passed in the above send interface, and what is returned is the queue to which the message needs to be sent. In this example, orderSn (order number) is used as the partition classification standard, and the remainder of the number of all queues is used to send messages with the same orderId to the same queue.

In a production environment, it is recommended to choose the most granular partition key for splitting . For example, using order ID and user ID as partition key keywords can realize that messages from the same end user are processed in order, and messages from different users do not need to be guaranteed in order.

Sequential message consumer:

public class OrderConsumer {
    public static void main(String[] args) throws Exception{
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("orderly-consumer-group");
        consumer.setNamesrvAddr("ip:9876");
        consumer.subscribe("orderly-topic", "*");
        // MessageListenerConcurrently 并发模式 多线程的  重试16次 后会将其放入 死信队列
        // MessageListenerOrderly 顺序模式 单线程的   无限重试Integer.Max_Value
        consumer.registerMessageListener(new MessageListenerOrderly() {
            @Override
            public ConsumeOrderlyStatus consumeMessage(List<MessageExt> msgs, ConsumeOrderlyContext context) {
                System.out.println("线程id:" + Thread.currentThread().getId());
                System.out.println(new String(msgs.get(0).getBody()));
                //若返回 ConsumeOrderlyStatus.SUSPEND_CURRENT_QUEUE_A_MOMENT 则会被挂起，等待一段时间再重试
                return ConsumeOrderlyStatus.SUCCESS;
            }
        });
        consumer.start();
        System.in.read();
    }
}

MessageListenerConcurrently concurrent mode multi-threaded will be put into the dead letter queue after retrying 16 times.
MessageListenerOrderly sequential mode single-threaded infinite retries (Integer.Max_Value times)

If ConsumeOrderlyStatus.SUSPEND_CURRENT_QUEUE_A_MOMENT is returned , it will be suspended. Wait for a while and try again.

The print result is as follows:

3. Tag filtering

Rocketmq provides message filtering function, distinguished by tag or key

Tag message tag is used to further distinguish the message classification under a certain topic. The message queue RocketMQ allows consumers to filter messages according to Tag to ensure that the consumer only consumes the message types he cares about.

Practical operation: The producer will produce messages with two tags, vip1 and vip2, while consumer1 only subscribes to vip1 messages, and consumer2 subscribes to vip1 and vip2 messages. Although consumer1 and consumer2 subscribe to the same topic, they have different tags. In order to ensure the consistency of the subscription relationship, they should belong to different consumer groups.

TagProducer：

public class TagProducer {
    public static void main(String[] args) throws Exception{
        // 创建默认的生产者
        DefaultMQProducer producer = new DefaultMQProducer("tag-producer-group");
        // 设置nameServer地址
        producer.setNamesrvAddr("ip:9876");
        // 启动实例
        producer.start();
        Message msg1 = new Message("tag-topic","vip1", "vip1的消息".getBytes());
        Message msg2 = new Message("tag-topic","vip2", "vip2的消息".getBytes());
        producer.send(msg1);
        producer.send(msg2);
        System.out.println("发送成功");
        // 关闭实例
        producer.shutdown();
    }
}

TagConsumer1：

public class TagConsumer1 {
    public static void main(String[] args) throws Exception{
        // 订阅关系一致性
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("tag-consumer-group-a");
        consumer.setNamesrvAddr("ip:9876");
        //只能收到 vip1 的消息
        consumer.subscribe("tag-topic", "vip1");
        consumer.registerMessageListener(new MessageListenerConcurrently() {
            @Override
            public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
                System.out.println("我是vip1的消费者，我正在消费消息" + new String(msgs.get(0).getBody()));
                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
            }
        });
        consumer.start();
        System.in.read();
    }
}

TagConsumer2：

public class TagConsumer2 {
    public static void main(String[] args) throws Exception{
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("tag-consumer-group-b");
        consumer.setNamesrvAddr("ip:9876");
        consumer.subscribe("tag-topic", "vip1 || vip2");
        consumer.registerMessageListener(new MessageListenerConcurrently() {
            @Override
            public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
                System.out.println("我是vip2的消费者，我正在消费消息" + new String(msgs.get(0).getBody()));
                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
            }
        });
        consumer.start();
        System.in.read();
    }
}

Subscription relationship consistency

Subscription relationship: A consumer group subscribes to a certain Tag of a Topic. This record is called a subscription relationship.

Consistent subscription relationships: The topics and tags subscribed to by all consumer instances under the same consumer group must be exactly the same. If the subscription relationship (consumer group name-Topic-Tag) is inconsistent, consumption messages will be disordered or even lost.

①Subscribe to a Topic and subscribe to a Tag

As shown in the figure below, the three Consumer instances C1, C2 and C3 under the same Group ID are all subscribed to TopicA, and the tags subscribed to TopicA are all Tag1, which conforms to the principle of consistent subscription relationships.

②Subscribe to a Topic and multiple Tags

As shown in the figure below, the three Consumer instances C1, C2 and C3 under the same Group ID are all subscribed to TopicB. The tags subscribed to TopicB are also Tag2 and Tag3, which means that all messages in TopicB with Tags of Tag2 or Tag3 are subscribed, and The order is always Tag2||Tag3, which conforms to the consistency principle of the subscription relationship.

③Subscribe to multiple topics and subscribe to multiple tags

As shown in the figure below, the three Consumer instances C1, C2 and C3 under the same Group ID are subscribed to TopicA and TopicB respectively, and the subscribed TopicA does not specify a tag, that is, all the messages in TopicA are subscribed, and the tags of the subscribed TopicB are all are Tag2 and Tag3, indicating that all messages in TopicB with Tag 2 or Tag3 are subscribed, and the order is consistent and all are Tag2||Tag3, which conforms to the principle of consistent subscription relationships.

When should you use Topic and when should you use Tag?

Different businesses should use different Topics. If the same business has different table representations, then we need to use tags to distinguish them.

It can be judged from the following aspects:

Whether the message types are consistent : such as ordinary messages, transaction messages, scheduled (delayed) messages, and sequential messages. Different message types use different topics and cannot be distinguished by Tags.
Whether the business is related : Messages that are not directly related, such as Taobao transaction messages and JD Logistics messages, use different Topics to distinguish them; and for the same Tmall transaction messages, messages about electrical appliance orders, women's clothing orders, and cosmetics orders can use Tags Make a distinction.
Whether the message priority is consistent : For example, if it is a logistics message, Hema must deliver it within hours, Tmall Supermarket must deliver it within 24 hours, and Taobao Logistics will be relatively slower. Messages with different priorities are distinguished by different Topics.
Whether the message magnitudes are equivalent : Although some business messages are small in volume, they have high real-time requirements. If they use the same topic as some trillion-level messages, they may "starve to death" due to too long waiting time. This is It is necessary to split messages of different magnitudes and use different topics.

4. Key in the message

Message producer with key:

public class KeyProducer {
    public static void main(String[] args) throws Exception{
        DefaultMQProducer producer = new DefaultMQProducer("key-producer-group");
        producer.setNamesrvAddr("36.133.174.153:9876");
        producer.start();
        /*
         * 业务参数 我们自身要确保唯一
         * 为了查阅和去重
         * 我这里就用没有业务意义的UUID暂替一下
         */
        String key = UUID.randomUUID().toString();
        System.out.println("key:"+key);
        Message message = new Message("key-topic", "vip1", key, "我是vip1的文章".getBytes());
        producer.send(message);
        System.out.println("发送成功");
        producer.shutdown();
    }
}

Message consumer with key:

public class KeyConsumer {
    public static void main(String[] args) throws Exception{
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("key-consumer-group");
        consumer.setNamesrvAddr("ip:9876");
        consumer.subscribe("key-topic", "*");
        consumer.registerMessageListener(new MessageListenerConcurrently() {
            @Override
            public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
                MessageExt messageExt = msgs.get(0);
                System.out.println("我是vip1的消费者，我正在消费消息" + new String(messageExt.getBody()));
                System.out.println("我们业务的标识：" + messageExt.getKeys());//取自己定义的业务标识key
                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
            }
        });
        consumer.start();
        System.in.read();
    }
}

5. The problem of repeated consumption

Problem arises

Duplication caused by client re-investment mechanism

For example, the producer uses the retry mechanism when sending a message. After sending the message, it does not receive the response information from MQ due to network reasons, reports a timeout exception, and then resends the message.

But in fact, MQ has received the message and returned a response, but it timed out due to network reasons. In this case, a message will be sent twice.

When the consumer group is rebalanced

When the consumer group expands (or vice versa when a consumer goes offline ), it will be Reblanced and the queue will be reallocated. The new consumer corresponding to a queue needs to obtain the offset of the previous consumption (offset, that is, the point of message consumption). At this time, the previous consumer may have consumed a message, but did not submit the offset to the broker. , then the new consumer may consume again.

Concept introduction

Idempotence

The impact of multiple operations is the same as the impact of the first operation

New: May be non-idempotent operation, may be idempotent operation. If a unique index is set in the inserted field, it is an idempotent operation (because subsequent insertions will not take effect)

Modification: It may be a non-idempotent operation, it may be an idempotent operation

update goods set stock = 10 where id = 1 //幂等性
update goods set stock = stock -1 where id = 1 //非幂等性

Query: generally considered to be an idempotent operation
Delete: generally considered an idempotent operation

Solution steps

Step 1: The sender needs to bring a unique mark to the message (controlled by its own business)

Step 2: The consumer side performs deduplication processing at the business level .

Use a relational database (such as MySQL) to perform deduplication, design a deduplication table, and add a unique index to the unique key of the message . Every time a message is consumed, it is first inserted into the database. If it succeeds, the business logic will be executed (if an error occurs during the execution of the business logic, the deduplication record will be deleted); if the insertion fails, it means that the message has arrived and you can sign for it directly.
Use BloomFilter . A Bloom filter is actually a long binary vector and a series of random mapping functions . Bloom filters can be used to retrieve whether an element is in a collection . Its advantage is that space efficiency and query time are much better than ordinary algorithms. Its disadvantage is that it has a certain misrecognition rate and difficulty in deletion .
Through redis's setnx, the weight will be judged based on whether the setting is successful.

6. Message retry and dead letter messages

message retry

It is divided into producer retry and consumer retry. The main thing is consumer retry, so the focus here is on consumer retry.

If the consumer returns RECONSUME_LATER or null or an exception occurs during consumption, the message will be returned to the queue and re-delivered later to the current consumer or other consumers.

In the actual production process, we usually retry 3-5 times. If the consumption is not successful, we can sign the message and notify manual processing.

Retry interval (based on the delay interval)

10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h

The default retry is 16 times (concurrent mode). In sequential mode (int maximum times), you can customize the number of retries. If it still fails after retrying, it will be placed in a dead letter topic with the name of the topic: %DLQ% Consumer group name

dead letter message

When the consumption retry reaches the threshold, the message will not be delivered to the consumer, but will enter the dead letter queue. This type of message is called a dead letter message , and the special queue that stores dead letter messages is called a dead letter queue . The dead letter queue is a separate queue (only one queue) with the unique number of partitions under the dead letter topic . If a dead letter message is generated, the corresponding dead letter Topic name of the ConsumerGroup is %DLQ%ConsumerGroupName. Note: Dead letter topics can be subscribed and dead letter queues can also be monitored.

Solution ①: Directly monitor the messages on the dead letter topic

DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("retry-dead-consumer-group");
        consumer.setNamesrvAddr("ip:9876");
        consumer.subscribe("%DLQ%消费者组名", "*");
        consumer.registerMessageListener(new MessageListenerConcurrently() {
            @Override
            public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
                MessageExt messageExt = msgs.get(0);
                System.out.println(new Date());
                System.out.println(new String(messageExt.getBody()));
                System.out.println("记录到特别的位置 文件 mysql 通知人工处理");
                // 业务报错了 返回null 返回 RECONSUME_LATER 都会重试
                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
            }
        });
        consumer.start();
        System.in.read();

Disadvantages: Every kind of dead letter queue has to write a consumer group to monitor, which is very troublesome.

Solution ② (used more often)

DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("retry-consumer-group");
        consumer.setNamesrvAddr(MqConstant.NAME_SRV_ADDR);
        consumer.subscribe("ip:9876", "*");
        // 设定重试次数
        consumer.registerMessageListener(new MessageListenerConcurrently() {
            @Override
            public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
                MessageExt messageExt = msgs.get(0);
                System.out.println(new Date());
                // 业务处理
                try {
                   //注入的 service
                } catch (Exception e) {
                    // 重试
                    int reconsumeTimes = messageExt.getReconsumeTimes();
                    if (reconsumeTimes >= 3) {
                        // 不要重试了
                        System.out.println("记录到特别的位置 文件 mysql 通知人工处理");
                        return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
                    }
                    return ConsumeConcurrentlyStatus.RECONSUME_LATER;
                }
                // 业务报错了 返回null 返回 RECONSUME_LATER 都会重试
                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
            }
        });
        consumer.start();
        System.in.read();

7. Two consumption models

Rocketmq message consumption modes are divided into two types: cluster mode and broadcast mode

Cluster mode means that multiple consumers alternately consume messages in the same topic.

In cluster mode, the queue will be shared among consumers. The number of queues should preferably be >= the number of consumers, and the MQ server will record the consumption location for processing messages.

The broadcast mode means that each consumer consumes the messages of the subscribed topic once.

In broadcast mode, the message will be processed once by each consumer. The MQ server will not record the consumption point and will not retry.

8. Message backlog problem

A single queue can store up to 300,000 messages. It is generally considered that when the message difference of a single queue is >= 100,00, it is considered a stacking problem.

Cause of the problem:

①Production speed is too fast

solution:

Producers can limit business flow
Increase the number of consumers appropriately, but always keep the number of consumers <= the number of queues
You can appropriately set the maximum number of consumer threads for consumers. If the consumer is I/O-intensive (that is, mostly operates databases, files, or calls RPC, etc.) , you can set a maximum number of 2n threads; if the consumer is CPU-intensive (mostly does calculations) ) , you can set n+1 maximum threads. n is the number of logical processors in the server.
Dynamically expand the number of queues to increase the number of consumers (negotiate with operation and maintenance)
If the business does not have high data requirements, you can choose to discard unimportant messages.

②Consumer has a problem/hangs up

Troubleshoot consumer program issues

9. Message loss problem

We can analyze it from the perspective of producers, MQ and consumers respectively.

producer perspective

If the producer wants to ensure that the message is not lost as much as possible, he can choose to send it synchronously.
ACK confirmation mechanism. RocketMQ provides an ACK mechanism to ensure that messages can be consumed normally. In order for the sender to ensure that the message consumption is successful, RocketMQ will consider the message consumption to be successful only if the user explicitly states that the consumption is successful. If the power is cut off midway, an exception is thrown, etc., it will not be considered successful - which leads to a retry mechanism.
Retry mechanism
The producer can record the sending log before sending, and then modify the status of the log after the consumer has finished consuming. If the status of the production log has not changed for a long time using scheduled task query , the message can be reissued. Of course, during consumption The layer needs to be idempotent.

MQ angle

Provides strategies for synchronous disk brushing. After the message is delivered to the broker, it will be stored in the page cache first, and then the disk will be flushed immediately according to the flushing strategy set by the broker. That is, if the flushing strategy is asynchronous, the broker will not wait for the message to be dropped and will return to the producer successfully. That is to say, when the server where the broker is located suddenly goes down, some pages of information will be lost.
Provides master-slave mode, and master-slave supports synchronous dual writing. Even if the broker sets up synchronous disk flushing, if the main broker disk is damaged, messages will be lost. Therefore, you can specify a slave for the broker, set the master to SYNC_MASTER, and then set the slave to a synchronous disk brushing strategy. In this mode, every time the producer sends a message, it will wait until the message is successfully delivered to both the master and the slave, and then the broker will regard the message as successfully delivered, ensuring that the rest is not lost. However, this performance is lower, so the master-slave synchronization strategy also defaults to asynchronous synchronization.

consumer perspective

Submit ACK only after correct consumption and processing
Retry mechanism

RocketMQ sends message

1. Consumption pattern