Table of contents

1. How to ensure the reliability of the message

return mechanism

1.2. How to ensure that messages are not lost in the queue

1.3. Ensure that messages can be consumed reliably

2. Delay Queue

2.1.TTL

2.2. Dead letter queue

2.3. Delay Queue

3. How to prevent consumers from repeatedly consuming messages

1. How to ensure the reliability of the message

1.1. Reliable delivery of messages

In the production environment, due to some unknown reasons, rabbitmq restarts. During the restart of RabbitMQ, the delivery of producer messages fails, resulting in message loss, which requires manual processing and recovery. So, we started to think about how to deliver RabbitMQ messages reliably? Especially in such an extreme situation, when the RabbitMQ cluster is unavailable, how to deal with undeliverable messages?

When using RabbitMQ, as a message sender, you want to prevent any message loss or delivery failure scenarios. RabbitMQ provides us with two ways to control the reliability mode of message delivery.

confirm confirmation mode

return return mode

Message from producer to exchange will return a confirmCallback.

If the message fails to be delivered from exchange-->queue, a returnCallback will be returned.

By default, rabbitmq does not enable the above two modes

We will use these two callbacks to control the reliable delivery of messages

Implementation of confirm and return

Set the publisher-confirm-type: correlated of the ConnectionFactory to enable the confirmation mode.

Use rabbitTemplate.setConfirmCallback to set the callback function. Call back the confirm method when the message is sent to the exchange. Judge the ack in the method, if it is true, the sending is successful, if it is false, the sending fails and needs to be processed.

Set publisher-returns="true" of ConnectionFactory to enable return mode.

Use rabbitTemplate.setReturnCallback to set the return function, and execute the callback function returnedMessage when the message fails to be routed from the exchange to the queue.

confirmation mechanism

Demo: 4. springboot integrates RabbitMQ

(1). Turn on confirm in the configuration file

#开启confirm确认机制
spring.rabbitmq.publisher-confirm-type=correlated

(2) Set the confirmCallback callback function of rabbitTemplate

    /**
     * 使用confirm机制：
     * （1）需要开启confirm机制。-----配置文件中加入：spring.rabbitmq.publisher-confirm-type=correlated
     * (2)为rabbitTemplate指定setConfirmCallback回调函数
     */
    @Test
    void test001() {
        //只能保证消息从生产者到交换机的可靠性
        rabbitTemplate.setConfirmCallback(new RabbitTemplate.ConfirmCallback() {
            @Override
            public void confirm(CorrelationData correlationData, boolean ack, String cause) {
                //触发该方法
                if (ack == false) {
                    System.out.println("未来根据项目的需要，完成相应的操作");
                }
            }
        });
        rabbitTemplate.convertAndSend("Topics-exchange", "lazy.aaa", "Hello RabbitMQ...");
    }

return mechanism

(1). Turn on return in the configuration file

#开启return机制
spring.rabbitmq.publisher-returns=true

(2) Set the return callback function of rabbitTemplate

    /**
     * return机制：
     * （1）开启return机制---配置文件加入：spring.rabbitmq.publisher-returns=true
     * （2）为rabbitTemplate设置return的回调函数
     */
    @Test
    void test002() {
        //只有当消息无法从交换机到队列时才会触发
        rabbitTemplate.setReturnCallback(new RabbitTemplate.ReturnCallback() {
            @Override
            public void returnedMessage(Message message, int replyCode, String replyText, String exchange, String routingKey) {
                //为交换机到队列分发消息失败时会触发
                System.out.println("replyCode====" + replyCode);
                System.out.println("replyText====" + replyText);
            }
        });
        rabbitTemplate.convertAndSend("Topics-exchange", "lazy.aaa", "Hello RabbitMQ...");

    }

1.2. How to ensure that messages are not lost in the queue

(1) Set the queue as persistent

(2) Set the persistence of the message

1.3. Ensure that messages can be consumed reliably

ACK confirmation mechanism

Multiple consumers receive messages at the same time. Halfway through receiving messages, a consumer suddenly hangs up. To ensure that this message is not lost, an acknowledgment mechanism is required, that is, the consumer must notify the server after consumption, and the server will delete the data.

This solves the case that even if a consumer has a problem and there is no synchronous message to the server, there are other consumers to consume, ensuring that the message is not lost.

Implementation of ACK

ack refers to Acknowledge, confirmation. Indicates the confirmation method of the consumer after receiving the message.

There are three confirmation methods:

Automatic confirmation: acknowledge="none"

Manual confirmation: acknowledge="manual"

Confirm according to the abnormal situation: acknowledge="auto" (this method is troublesome to use and is not commonly used)

The automatic confirmation means that once the message is received by the Consumer, it will automatically confirm the receipt and remove the corresponding message from the RabbitMQ message queue. However, in actual business processing, it is very likely that the message is received and the business processing is abnormal, and the message will be lost.

If the manual confirmation method is set , you need to call channel.basicAck() to sign for the receipt manually after the business processing is successful. If there is an exception, call the channel.basicNack() method to let it automatically resend the message.

Consumer side:

(1). Modify the consumer side - manually confirm the message
#修改消费端----手动确认消息
spring.rabbitmq.listener.simple.acknowledge-mode=manual
(2).Modify the code
package com.wqg.listener;

import com.rabbitmq.client.Channel;
import org.springframework.amqp.rabbit.annotation.RabbitListener;
import org.springframework.stereotype.Component;
import org.springframework.amqp.core.Message;

import java.io.IOException;



/**
 * @ fileName:MyListener
 * @ description:
 * @ author:wqg
 * @ createTime:2023/7/12 18:57
 */
@Component
public class MyListener {

    //basicAck:确认消息----rabbit服务端删除
    //basicNack：服务继续发送消息
    @RabbitListener(queues = {"Topics-queue002"})//queues:表示你监听的队列名
    public void h(Message message , Channel channel) throws IOException {
        long deliveryTag = message.getMessageProperties().getDeliveryTag();
        //把监听到的消息封装到Message类对象中
        byte[] body = message.getBody();
        String s = new String(body);
        System.out.println("消息内容==="+s);
        try {
            //int a = 10/0; //模拟宕机
            System.out.println("核心业务的处理~~~");
            /**
             *long deliveryTag： 消息的标注
             * boolean multiple：是否把该消息之前未确认的消息一起确认掉
             */
            channel.basicAck(deliveryTag,true);//确认消息
        } catch (Exception e) {
            /**
             * long deliveryTag; boolean multiple;
             * boolean requeue：是否要求rabbitmq服务重新发送该消息
             */
            channel.basicNack(deliveryTag,true,true);
        }
    }
}

Summary: How to ensure the reliability of the message?

Guaranteed reliable delivery of messages: confirm mechanism and return mechanism

In the queue:---persistence

Use the ack mechanism to ensure the reliable consumption of consumers.

2. Delay Queue

2.1.TTL

The full name of TTL is Time To Live (time to live/expiration time).

When the message reaches the survival time, it will be automatically cleared if it has not been consumed.

RabbitMQ can set the expiration time for the message, or set the expiration time for the entire queue (Queue).

Demo:

Created using a graphical interface

producer test

result

The queue is set with an expiration time and the message is also set with an expiration time ----- execute according to the shortest time

summary:

To set the queue expiration time, use the parameter: x-message-ttl, unit: ms (milliseconds), which will uniformly expire the entire queue message.

Set the message expiration time using the parameter: expiration. Unit: ms (milliseconds), when the message is at the head of the queue (when consumed), it will be judged whether the message is expired or not.

If both are set, the shorter time will prevail.

2.2. Dead letter queue

Dead letter queue, English abbreviation: DLX. Dead Letter Exchange (dead letter exchange), when the message becomes Dead message, it can be resent to another exchange, this exchange is DLX.

What kind of messages become dead letter messages:

Queue message length reaches limit

The consumer rejects the consumption message, basicNack/basicReject, and does not put the message back into the original target queue, requeue=false

There is a message expiration setting in the original queue, and the message arrival timeout time has not been consumed

Queue binding dead letter exchange:

Demo:

Create using a GUI:

producer test

result

2.3. Delay Queue

Delay queue, that is, the message will not be consumed immediately after entering the queue, but will be consumed only after reaching the specified time.

need:

After placing the order, if the payment is not made within 30 minutes, the order will be canceled and the inventory will be rolled back.
7 days after the successful registration of a new user, send a text message greeting.

Method to realize:

Timer: Poor performance --- Database queries are performed at regular intervals.
delay queue

The function of delay queue is completed through message queue:

The delay queue function is not provided in RabbitMQ.
But you can use: TTL + dead letter queue combination to achieve the effect of delay queue.

Demo:

queue is empty

Create a springboot project

configuration file


#rabbitmq的配置
spring.rabbitmq.host=192.168.75.129
spring.rabbitmq.username=guest
spring.rabbitmq.password=guest
spring.rabbitmq.virtual-host=/

(1). Introduce dependencies

<dependencies>
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.83</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-amqp</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.amqp</groupId>
            <artifactId>spring-rabbit-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

(2). Create OrderController.java to simulate orders

package com.wqg.controller;

import com.alibaba.fastjson.JSON;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.HashMap;
import java.util.UUID;


/**
 * @ fileName:OrderController
 * @ description:订单
 * @ author:wqg
 * @ createTime:2023/7/13 16:24
 */
@RestController
@RequestMapping("/order")
public class OrderController {
    @Autowired
    private RabbitTemplate rabbitTemplate;

    @GetMapping("/saveOrder")
    public String save(Integer pid, Integer num) {
        //生成一个订单号
        String orderId = UUID.randomUUID().toString().replace("-", "");
        System.out.println("下单成功，订单号为===" + orderId);
        HashMap<Object, Object> map = new HashMap<>();
        map.put("orderId", orderId);
        map.put("pid", pid);
        map.put("num", num);
        rabbitTemplate.convertAndSend("pt_exchange", "qy165.aaa", JSON.toJSONString(map));
        return "下单成功";
    }
}

(3). Create MyListener.java to simulate listening

package com.wqg.listener;

import com.alibaba.fastjson.JSON;
import org.springframework.amqp.core.Message;
import org.springframework.amqp.rabbit.annotation.RabbitListener;
import org.springframework.stereotype.Component;

import java.util.HashMap;

/**
 * @ fileName:MyListener
 * @ description:
 * @ author:wqg
 * @ createTime:2023/7/13 16:35
 */
@Component
public class MyListener {

    @RabbitListener(queues = {"dead_queue"})
    public void hello(Message message){
        byte[] body = message.getBody();
        String s = new String(body);
        HashMap hashMap = JSON.parseObject(s, HashMap.class);
        System.out.println("message==="+hashMap);

        //取消订单
        System.out.println("取消订单号为==="+hashMap.get("orderId"));


    }
}

test

console

3. How to prevent consumers from repeatedly consuming messages

The idempotency of the message---no matter how many times the operation is performed, the result is the same.

Generate a global id, store it in redis or database, and check whether the message has been consumed before the consumer consumes the message.
If the message has been consumed, tell mq that the message has been consumed, and discard the message (manual ack).
If it has not been consumed, consume the message and write the consumption record into redis or database.

Briefly describe the requirements. If the order is completed, points need to be accumulated for the user, and it is necessary to ensure that the points will not be accumulated repeatedly. Then before mq consumes the message, first go to the database to check whether the message has been consumed, and if it has been consumed, then discard the message directly.

Demo:

producer

import com.alibaba.fastjson.JSONObject;
import com.xiaojie.score.entity.Score;
import lombok.extern.slf4j.Slf4j;
import org.springframework.amqp.rabbit.connection.CorrelationData;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.scheduling.annotation.Async;
import org.springframework.stereotype.Component;
 
import java.util.UUID;
 
/**
 * @author 
 * @version 1.0
 * @description:发送积分消息的生产者
 * @date
 */
@Component
@Slf4j
public class ScoreProducer implements RabbitTemplate.ConfirmCallback {
    @Autowired
    private RabbitTemplate rabbitTemplate;
    //定义交换机
    private static final String SCORE_EXCHANGE = "ykq_score_exchaneg";
    //定义路由键
    private static final String SCORE_ROUTINNGKEY = "score.add";
 
    /**
     * @description: 订单完成
     * @param:
     * @return: java.lang.String
     * @author xiaojie
     * @date: 
     */
    public String completeOrder() {
        String orderId = UUID.randomUUID().toString();
        System.out.println("订单已完成");
        //发送积分通知
        Score score = new Score();
        score.setScore(100);
        score.setOrderId(orderId);
        String jsonMSg = JSONObject.toJSONString(score);
        sendScoreMsg(jsonMSg, orderId);
        return orderId;
    }
 
    /**
     * @description: 发送积分消息
     * @param:
     * @param: message
     * @param: orderId
     * @return: void
     * @author 
     * @date:
     */
 
    @Async
    public void sendScoreMsg(String jsonMSg, String orderId) {
        this.rabbitTemplate.setConfirmCallback(this);
        rabbitTemplate.convertAndSend(SCORE_EXCHANGE, SCORE_ROUTINNGKEY, jsonMSg, message -> {
            //设置消息的id为唯一
            message.getMessageProperties().setMessageId(orderId);
            return message;
        });
    }
 
    @Override
    public void confirm(CorrelationData correlationData, boolean ack, String s) {
        if (ack) {
            log.info(">>>>>>>>消息发送成功:correlationData:{},ack:{},s:{}", correlationData, ack, s);
        } else {
            log.info(">>>>>>>消息发送失败{}", ack);
        }
    }
}

consumer

import com.alibaba.fastjson.JSONObject;
import com.rabbitmq.client.Channel;
import com.xiaojie.score.entity.Score;
import com.xiaojie.score.mapper.ScoreMapper;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.springframework.amqp.core.Message;
import org.springframework.amqp.rabbit.annotation.RabbitListener;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.messaging.handler.annotation.Headers;
import org.springframework.stereotype.Component;
 
import java.io.IOException;
import java.util.Map;
 
/**
 * @author 
 * @version 1.0
 * @description: 积分的消费者
 * @date 
 */
@Component
@Slf4j
public class ScoreConsumer {
    @Autowired
    private ScoreMapper scoreMapper;
 
    @RabbitListener(queues = {"ykq_score_queue"})
    public void onMessage(Message message, @Headers Map<String, Object> headers, Channel channel) throws IOException {
        String orderId = message.getMessageProperties().getMessageId();
        if (StringUtils.isBlank(orderId)) {
            return;
        }
        log.info(">>>>>>>>消息id是：{}", orderId);
        String msg = new String(message.getBody());
        Score score = JSONObject.parseObject(msg, Score.class);
        if (score == null) {
            return;
        }
        //执行前去数据库查询，是否存在该数据，存在说明已经消费成功，不存在就去添加数据，添加成功丢弃消息
        Score dbScore = scoreMapper.selectByOrderId(orderId);
        if (dbScore != null) {
            //证明已经消费消息，告诉mq已经消费,丢弃消息
            channel.basicAck(message.getMessageProperties().getDeliveryTag(), false);
            return;
        }
        Integer result = scoreMapper.save(score);
        if (result > 0) {
            //积分已经累加，删除消息
            channel.basicAck(message.getMessageProperties().getDeliveryTag(), false);
            return;
        } else {
            log.info("消费失败，采取相应的人工补偿");
        } 
    }
}

How to ensure message reliability + delay queue (TTL + dead letter queue + delay queue)

1. How to ensure the reliability of the message

1.1. Reliable delivery of messages

confirmation mechanism

return mechanism

1.2. How to ensure that messages are not lost in the queue

1.3. Ensure that messages can be consumed reliably

2. Delay Queue

2.1.TTL

2.2. Dead letter queue

2.3. Delay Queue

3. How to prevent consumers from repeatedly consuming messages

Guess you like