Detailed use of RabbitMq

Detailed use of message queue RabbitMQ

The article here is re-edited based on my notes from the Internet and my own understanding, mainly to facilitate my review of knowledge points in real time.

MQ related concepts

What is MQ

MQ (message queue), literally, is a queue. FIFO is first in, first out. However, the content stored in the queue is only messages. It is also a cross-process communication mechanism for upstream and downstream message transmission. In the Internet architecture, MQ is a very common upstream and downstream "logical decoupling + physical decoupling" message communication service. After using MQ, the upstream message sending only needs to rely on MQ and does not need to rely on other services.

Why use MQ

• Traffic peak reduction

For example, if the order system can handle up to 10,000 orders, this processing capacity is more than enough to handle orders during normal periods. During normal periods, we can return results in one second after placing orders. However, during the peak period, if there are 20,000 orders placed, the operating system cannot handle it. It can only limit the number of orders exceeding 10,000 and not allow users to place orders. Using the message queue as buffer, we can cancel this restriction and spread the orders placed within one second into a period of time for processing. At this time, some users may not receive the successful order operation more than ten seconds after placing the order, but it is better than not being able to place the order. The single experience is better.

• Application decoupling

Take e-commerce applications as an example. The applications include order systems, inventory systems, logistics systems, and payment systems. After the user creates an order, if the inventory system, logistics system, and payment system are coupled and called, if any subsystem fails, the order operation will be abnormal. When converted to a message queue-based approach, the problems of inter-system calls will be greatly reduced. For example, the logistics system will take a few minutes to repair due to a failure. During these few minutes, the memory to be processed by the logistics system is cached in the message queue, and the user's order operation can be completed normally. When the logistics system is restored, order information can continue to be processed. The mid-order user will not feel the failure of the logistics system, improving the availability of the system.

• Asynchronous processing

Some calls between services are asynchronous. For example, A calls B, and B takes a long time to execute, but A needs to know when B can be completed.

In the past, there were generally two ways. A would call B's query api query after a period of time. Or A provides a callback API, and after B completes the execution, it calls the API to notify A of the service. Neither approach is very elegant.

Using the message bus can easily solve this problem. After A calls service B, it only needs to monitor the message that B has completed. When B completes processing, it will send a message to MQ, and MQ will forward the message to service A. In this way, service A does not need to call B's query API in a loop, nor does it need to provide a callback API. Similarly, service B does not need to perform these operations. Service A can also get the message of successful asynchronous processing in a timely manner.

Classification of MQ

::: tip ActiveMQ

:::

Advantages: Single machine throughput of 10,000 levels, timeliness of ms level, high availability, high availability based on master-slave architecture, low probability of data loss due to low message reliability

Disadvantages: The official community now maintains less and less ActiveMQ 5.x, and it is rarely used in high-throughput scenarios.

::: tip Kafka

:::

The trump card of big data. When it comes to message transmission in the field of big data, Kafka cannot be avoided. This product is born forbig data's messaging middleware, famous for its million-level TPS throughput, has quickly become the darling of the big data field and plays a pivotal role in the process of data collection, transmission, and storage. It has been adopted by major companies such as LinkedIn, Uber, Twitter, and Netflix.

Advantages: Excellent performance, single-machine write TPS is about one million entries/second. The biggest advantage ishigh throughput a>. Timeliness and MS-level availability are very high. Kafka is distributed. There are multiple copies of one data. If a few machines go down, there will be no loss of data or unavailability. Consumers use the Pull method to obtain messages, and the messages are ordered and controlled through It can ensure that all messages are consumed and only consumed once; there is an excellent third-party Kafka Web management interface Kafka-Manager; it is relatively mature in the field of logs and is used by many companies and multiple open source projects; function support: The function is relatively simple, mainly Supports simple MQ functions, and is used on a large scale for real-time computing and log collection in the field of big data

Disadvantages: If a single Kafka machine has more than 64 queues/partitions, the load will obviously skyrocket. The more queues, the higher the load, and the response time for sending messages will become longer. Use Short polling method, real-time performance depends on the polling interval, consumption failure does not support retry; message sequence is supported, but when an agent goes down, messages will be out of order, Community Updates are slow;

::: tip RocketMQ

:::

RocketMQ is an open source product from Alibaba and is implemented in Java language. It referred to Kafka during design and made some improvements of its own. It is widely used by Alibaba in order, transaction, recharge, stream computing, message push, log streaming processing, binglog distribution and other scenarios.

Advantages:Single machine throughput of 100,000 levels, very high availability, distributed architecture, messages It can achieve 0 loss, MQ has relatively complete functions, is distributed, has good scalability, and supports 1 billion level message accumulation , there will be no performance degradation due to accumulation. The source code is java. We can read the source code ourselves and customize our own company's MQ

Disadvantages:There are not many supported client languages. Currently they are java and c++, of which c++ is immature; the community activity is average, no To implement interfaces such as JMS in the MQ core, some systems need to modify a lot of code to migrate

::: note RabbitMQ

:::

Released in 2007, it is a reusable enterprise messaging system based on AMQP (Advanced Message Queuing Protocol). is one of the most mainstream messaging middlewares currently. one.

Advantages: Due to the high concurrency characteristics of the erlang language, the performance is better; the throughput reaches 10,000 levels, the MQ function is relatively complete, robust, stable, easy to use, cross-platform, and supports multiple languages ​​​​such as: Python, Ruby, .NET, Java, JMS, C, PHP, ActionScript, XMPP, STOMP, etc., support AJAX and have complete documentation; the management interface provided by the open source is very good, easy to use, and the community is highly active; the update frequency is quite high

Official website update:https://www.rabbitmq.com/news.html

Disadvantages: The commercial version requires a fee and the learning cost is high

MQ choice

• Kafka

The main feature of Kafka is to handle message consumption based on the Pull mode and pursue high throughput. Its initial purpose is to use log collection and transmission, which is suitable for the data collection business of Internet services that generate large amounts of data. Large companies are recommended to use it. If there is a log collection function, kafka is definitely the first choice.

Shang Silicon Valley official website kafka video tutorial: http://www.gulixueyuan.com/course/330/tasks

• RocketMQ

Born forFinancial Internet field, for scenarios with high reliability requirements, especially order deductions in e-commerce, As well as business peak shaving, when a large number of transactions pour in, the backend may not be able to process it in time. RoketMQ may be more trustworthy in terms of stability. These business scenarios have been tested many times in Alibaba Double 11. If your business has the above concurrency scenarios, it is recommended to choose RocketMQ.

• RabbitMQ

Combined with the concurrency advantages of the Erlang language itself, the performance is good and the timeliness is within microseconds. The community activity is also relatively high. The management interface is very convenient to use. If your data volume is not So big, small and medium-sized companies give priority to RabbitMQ, which has relatively complete functions.

RabbitMQ

RabbitMQ concept

RabbitMQ is a messaging middleware: it accepts and forwards messages.

You can think of it as a courier site. When you want to send a package, you put your package at the courier station, and the courier will eventually deliver your package to the recipient. According to this logic, RabbitMQ is a courier. Station, a courier will help you deliver the express.

The main difference between RabbitMQ and express delivery stations is that it does not process express mail but receives, stores and forwards message data.

官网：https://www.rabbitmq.com/#features

Four core concepts

• producer

  The program that generates data and sends messages is the producer

• switch

  The switch is a very important component of RabbitMQ. On the one hand, it receives messages from producers, and on the other hand, it pushes messages to the queue. The switch must know exactly how to process the messages it receives, whether to push these messages to a specific queue or to multiple queues, or to discard the message, which depends on the type of switch.

• queue

  A queue is a data structure used internally by RabbitMQ, and although messages flow through RabbitMQ and the application, they can only be stored in queues. A queue is limited only by the memory and disk limitations of the host and is essentially a large message buffer. Many producers can send messages to a queue, and many consumers can try to receive data from a queue. This is how we use queues

• consumer

  Consumption and receiving have similar meanings. A consumer is mostly a program waiting to receive messages. Please note that producers, consumers and message middleware are often not on the same machine. The same application can be both a producer and a consumer.

Introduction to each noun

• Broker

  The application that receives and distributes messages, RabbitMQ Server is the Message Broker, usually sends the messages to consumers, and also provides reliable delivery of messages.

• Virtual host

  Designed for multi-tenancy and security reasons, the basic components of AMQP are divided into a virtual group, similar to the namespace concept in the network. It allows you to run multiple independent RabbitMQ instances on a single RabbitMQ server.

• Connection

  TCP connection between publisher/consumer and broker

• Channel

  If a Connection is established every time RabbitMQ is accessed, the overhead of establishing a TCP Connection will be huge and the efficiency will be low when the message volume is large. Channel is a logical connection established inside the connection. If the application supports multi-threading, each thread usually creates a separate channel for communication. The AMQP method includes the channel id to help the client and message broker identify the channel, so the channels are completely isolated. of. As a lightweight Connection, Channel greatly reduces the operating system's cost of establishing a TCP connection, eliminating the need to establish a connection every time.

• Exchange

  The message reaches the first stop of the broker. According to the distribution rules, it matches the routing key in the query table and distributes the message to the queue. Commonly used types are: direct (point-to-point), topic (publish-subscribe) and fanout (multicast)

• Queue

  The message is finally sent here to be picked up by the consumer

• Binding

  The virtual connection between the exchange and the queue. The binding can contain the routing key. The binding information is saved in the query table in the exchange and is used as the basis for message distribution.

Install RabbitMQ

1. Download

Official website download address:https://www.rabbitmq.com/download.html

Here we choose the version number (note the requirements for these two versions)

• rabbitmq-server-3.8.8-1.el7.noarch.rpm

  GitHub：https://github.com/rabbitmq/rabbitmq-server/releases/tag/v3.8.8

  Load download:https://packagecloud.io/rabbitmq/rabbitmq-server/packages/el/7/rabbitmq-server-3.8.8-1.el7.noarch .rpm

• erlang-21.3.8.21-1.el7.x86_64.rpm

  Official website: https://www.erlang-solutions.com/downloads/

  加速：https://packagecloud.io/rabbitmq/erlang/packages/el/7/erlang-21.3.8.21-1.el7.x86_64.rpm

Red Hat 8, CentOS 8 and modern Fedora versions, replace "el7" with "el8"

2. Installation

Upload to the /usr/local/software directory (if there is no software, you need to create it yourself)

rpm -ivh erlang-21.3.8.21-1.el7.x86_64.rpm
yum install socat -y
rpm -ivh rabbitmq-server-3.8.8-1.el7.noarch.rpm

3. Start

# 启动服务
systemctl start rabbitmq-server
# 查看服务状态
systemctl status rabbitmq-server
# 开机自启动
systemctl enable rabbitmq-server
# 停止服务
systemctl stop rabbitmq-server
# 重启服务
systemctl restart rabbitmq-server

Web management interface and authorization operations

1. Installation

By default, the web client plug-in is not installed and needs to be installed to take effect.

rabbitmq-plugins enable rabbitmq_management

After the installation is complete, restart the service

systemctl restart rabbitmq-server

Visit http://42.192.149.71:15672 and log in with the default account password (guest). A permission problem occurs.

By default, it can only be accessed on localhost, so you need to add a remote login user.

2. Add user

# 创建账号和密码
rabbitmqctl add_user admin 123456

# 设置用户角色
rabbitmqctl set_user_tags admin administrator

# 为用户添加资源权限
# set_permissions [-p <vhostpath>] <user> <conf> <write> <read>
rabbitmqctl set_permissions -p "/" admin ".*" ".*" ".*"
# 添加配置、写、读权限

user level:

1. administrator: You can log in to the console, view all information, and manage rabbitmq
2. monitoring: Monitor log in to the console to view all information
3. policymaker: Policy maker Log in to the console and specify the policy
4. managment: Ordinary administrator logs in to the console

Log in again as the admin user

::: tip reset command

:::

The command to close the application is: rabbitmqctl stop_app

The clear command is: rabbitmqctl reset

The restart command is: rabbitmqctl start_app

Docker installation

官网：https://registry.hub.docker.com/_/rabbitmq/

docker run -id --name myrabbit -e RABBITMQ_DEFAULT_USER=admin -e RABBITMQ_DEFAULT_PASS=123456 -p 15672:15672 rabbitmq:3-management

Hello world

We will write two programs in Java. A producer that sends a single message and a consumer that receives the message and prints it out

In the diagram below, "P" is our producer and "C" is our consumer. The middle box is a queue RabbitMQ holds a message buffer on behalf of the consumer

When connecting, port 5672 needs to be opened

• rely

pom.xml

<!--指定 jdk 编译版本-->
<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <configuration>
                <source>8</source>
                <target>8</target>
            </configuration>
        </plugin>
    </plugins>
</build>
<dependencies>
    <!--rabbitmq 依赖客户端-->
    <dependency>
        <groupId>com.rabbitmq</groupId>
        <artifactId>amqp-client</artifactId>
        <version>5.8.0</version>
    </dependency>
    <!--操作文件流的一个依赖-->
    <dependency>
        <groupId>commons-io</groupId>
        <artifactId>commons-io</artifactId>
        <version>2.6</version>
    </dependency>
</dependencies>

Simple example

The following is an example of using a channel to implement the simplest message queue:

• message producer

Send a message

package com.oddfar.one;

import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;

/**
 * 生产者通过信道发送消息
 */
public class Producer {
    
    
    private final static String QUEUE_NAME = "hello";

    public static void main(String[] args) throws Exception {
    
    
        //创建一个连接工厂，用来获取信道channel
        ConnectionFactory factory = new ConnectionFactory();
        factory.setHost("42.192.149.71");
        factory.setUsername("admin");
        factory.setPassword("123456");
        //channel 实现了自动 close 接口 自动关闭 不需要显示关闭
        //创建连接
        Connection connection = factory.newConnection();
        //获取信道
        Channel channel = connection.createChannel();
        /**
         * 生成一个队列
         * 1.队列名称
         * 2.队列里面的消息是否持久化 也就是是否用完就删除
         * 3.该队列是否只供一个消费者进行消费 是否进行共享 true 可以多个消费者消费
         * 4.是否自动删除 最后一个消费者端开连接以后 该队列是否自动删除 true 自动删除
         * 5.其他参数
         */
        channel.queueDeclare(QUEUE_NAME, false, false, false, null);
        String message = "hello world";
        /**
         * 发送一个消息
         * 1.发送到那个交换机
         * 2.路由的 key 是哪个
         * 3.其他的参数信息
         * 4.发送消息的消息体
         */
        channel.basicPublish("", QUEUE_NAME, null, message.getBytes());
        System.out.println("消息发送完毕");

    }
    
}

• message consumer

Get messages sent by "producer"

package com.oddfar.one;

import com.rabbitmq.client.*;

/**
 * 消费者通过信道消费消息
 */
public class Consumer {
    
    
    private final static String QUEUE_NAME = "hello";

    public static void main(String[] args) throws Exception {
    
    
        ConnectionFactory factory = new ConnectionFactory();
        factory.setHost("42.192.149.71");
        factory.setUsername("admin");
        factory.setPassword("123456");
        Connection connection = factory.newConnection();
        Channel channel = connection.createChannel();

        System.out.println("等待接收消息.........");

        //推送的消息如何进行消费的接口函数回调
        DeliverCallback deliverCallback = (consumerTag, delivery) -> {
    
    
            String message = new String(delivery.getBody());
            System.out.println(message);
        };
        //取消消费的一个函数回调接口 比如如在消费的时候队列被删除掉了
        CancelCallback cancelCallback = (consumerTag) -> {
    
    
            System.out.println("消息消费被中断");
        };
        /**
         * 消费者消费消息 - 接受消息
         * 1.消费哪个队列
         * 2.消费成功之后是否要自动应答 true 代表自动应答 false 手动应答
         * 3.消费者未成功消费的回调
         * 4.消息被取消时的回调
         */
        channel.basicConsume(QUEUE_NAME, true, deliverCallback, cancelCallback);
    }

}

Work Queues

Work Queues - The main idea of ​​a work queue (also known as a task queue) is to avoid executing a resource-intensive task immediately and having to wait for it to complete. Instead we schedule the task to be executed later. We encapsulate the task as a message and send it to the queue. A worker process running in the background will pop out the tasks and eventually execute the job. When there are multiple worker threads, these worker threads will process these tasks together.

Rotation training distribution message

In this case we will start two worker threads and one message sending thread. Let's take a look at how these two worker threads work.

1. Extraction tools

package com.oddfar.utils;

import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;

public class RabbitMqUtils {
    
    
    //得到一个连接的 channel
    public static Channel getChannel() throws Exception {
    
    
        //创建一个连接工厂
        ConnectionFactory factory = new ConnectionFactory();
        factory.setHost("42.192.149.71");
        factory.setUsername("admin");
        factory.setPassword("123456");
        Connection connection = factory.newConnection();
        Channel channel = connection.createChannel();
        return channel;
    }
}

2. Start two worker threads to receive messages

package com.oddfar.two;

import com.oddfar.utils.RabbitMqUtils;
import com.rabbitmq.client.CancelCallback;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.DeliverCallback;

/**
 * 这是一个工作线程，相当于之前的消费者
 *
 */
public class Worker01 {
    
    

    private static final String QUEUE_NAME = "hello";

    public static void main(String[] args) throws Exception {
    
    

        Channel channel = RabbitMqUtils.getChannel();

        //消息接受
        DeliverCallback deliverCallback = (consumerTag, delivery) -> {
    
    
            String receivedMessage = new String(delivery.getBody());
            System.out.println("接收到消息:" + receivedMessage);
        };
        //消息被取消
        CancelCallback cancelCallback = (consumerTag) -> {
    
    
            System.out.println(consumerTag + "消费者取消消费接口回调逻辑");

        };

        System.out.println("C1 消费者启动等待消费.................. ");
        channel.basicConsume(QUEUE_NAME, true, deliverCallback, cancelCallback);

    }
}

selectedAllow multiple instances

After startup

3. Start a message sending thread

public class Task01 {
    
    
    public static final String QUEUE_NAME = "hello";

    public static void main(String[] args) throws Exception {
    
    

        Channel channel = RabbitMqUtils.getChannel();

        Scanner scanner = new Scanner(System.in);
        while (scanner.hasNext()) {
    
    
            String message = scanner.next();
            channel.basicPublish("", QUEUE_NAME, null, message.getBytes());
            System.out.println("消息发送完成：" + message);
        }

    }
}

• Results display

Through program execution, it was found that the producer sent a total of 4 messages, and consumer 1 and consumer 2 received two messages each, and received the messages one at a time in order.

message reply

It may take a while for consumers to complete a task, what happens if one of the consumers is working on a long task and only partially completes it and suddenly it hangs. As soon as RabbitMQ delivers a message to a consumer, it immediately marks the message for deletion. In this case, if a consumer suddenly hangs up, we will lose the message being processed. and subsequent messages sent to the consumer because it cannot receive them.

In order to ensure that the message is not lost during the sending process, a message response mechanism is introduced. The message response is:After the consumer receives the message and processes the message, it tells rabbitmq that it has been processed. Now, rabbitmq can delete the message.

Auto answer

The message is considered to have been successfully delivered immediately after it is sent. This mode requires a trade-off betweenhigh throughput and data transmission security, Because in this mode, if a connection occurs or the channel is closed on the consumer side before the message is received, the message will be lost. Of course, on the other hand, in this mode, the consumer side can deliver overloaded messages, There is no limit on the number of messages delivered. Of course, this may cause the consumer to receive too many messages that have no time to process, leading to a backlog of these messages and ultimately consuming memory. Eventually, these consumer threads are killed by the operating system, so this mode is only suitable for use when the consumer can process these messages efficiently and at a certain rate.

Manual message response method

• Channel.basicAck (for positive confirmation)

• channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);//第二个参数表示不重新入队
  

  RabbitMQ already knows the message and successfully processed it, so it can discard it.

• Channel.basicNack (for negative confirmation)

• Channel.basicReject (for negative confirmation)

  Note:
  Channel.basicNack and Channel.basicReject are both used to negate confirmation messages. The difference is:

  • Channel.basicNack can reject multiple messages in batches, while Channel.basicReject can only reject one message at a time.
  • Channel.basicNack can specify whether to requeue, while Channel.basicReject does not support requeuing.

  Here are examples of usage of Channel.basicNack and Channel.basicReject:

  // 使用 Channel.basicNack 拒绝一条消息
  channel.basicNack(deliveryTag, false, false);
  
  // 使用 Channel.basicNack 拒绝多条消息
  channel.basicNack(deliveryTags, false, false);
  
  // 使用 Channel.basicReject 拒绝一条消息
  channel.basicReject(deliveryTag, false);
  

  Parameter Description

  • deliveryTag: The identifier of the message.
  • multiple: Whether to reject messages in batches. If true, rejects the message specified by deliveryTag and all previously unacknowledged messages. If false, only messages specified by deliveryTag will be rejected.
  • requeue: Whether to requeue. If true, the message is requeued for consumption by other consumers. If false, the message is discarded.

Multiple explanation:

The advantage of manual answering is that it can answer in batches and reduce network congestion

• true represents batch response to unanswered messages on the channel

  For example, if there are messages with tags 5, 6, 7, 8 on the channel and the current tag is 8, then the unanswered messages from 5 to 8 will be confirmed to receive the message response.

• False Compared with the above, it will only respond to messages with tag=8. These three messages 5, 6, and 7 will still not be acknowledged.

When using batch response, you need to pay attention to the following points:

• Avoid requeuing : Messages should only be requeued if necessary.
• Set the queue size appropriately: The size of the queue should be set according to the message processing capacity to avoid message backlog.
• Use message expiration mechanism: You can set the expiration time of messages to avoid message backlog for too long.

Messages are automatically requeued

If the consumer loses connection for some reason (its channel is closed, the connection is closed, or the TCP connection is lost) and the message is not sent an ACK, RabbitMQ will understand that the message was not fully processed and will requeue it. If another consumer can handle it at this time, it will quickly redistribute it to another consumer. This way, even if a consumer dies occasionally, you can be sure that no messages are lost.

Message manual response code

The default message uses automatic response. The automatic response may re-enter the queue due to server interruption and other reasons, resulting in repeated consumption of messages. Therefore, if we want to achieve reliable message delivery, we need to change the automatic response to a manual response. The manual response must be changed. Flexible, you can choose to delete messages after consumption.

The consumer added the following content based on the above code

channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);//第二个参数表示不重新入队

Message producer:

package com.oddfar.three;

import com.oddfar.utils.RabbitMqUtils;
import com.rabbitmq.client.Channel;

import java.util.Scanner;

/**
 * 消息生产者,消息在手动应答时是不丢失的，放回队列重新消费

 */
public class Task02 {
    
    
    private static final String TASK_QUEUE_NAME = "ack_queue";

    public static void main(String[] args) throws Exception {
    
    
        Channel channel = RabbitMqUtils.getChannel();
        //声明队列
        channel.queueDeclare(TASK_QUEUE_NAME, false, false, false, null);
        Scanner sc = new Scanner(System.in);
        System.out.println("请输入信息");
        while (sc.hasNext()) {
    
    
            String message = sc.nextLine();
            //发布消息
            channel.basicPublish("", TASK_QUEUE_NAME, null, message.getBytes("UTF-8"));
            System.out.println("生产者发出消息" + message);
        }
    }

}

Consumer 01:

package com.oddfar.three;

import com.oddfar.utils.RabbitMqUtils;
import com.rabbitmq.client.CancelCallback;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.DeliverCallback;

/**
 * 消费者01 手动接收消息
 */
public class Work03 {
    
    
  private static final String TASK_QUEUE_NAME = "ack_queue";

  public static void main(String[] args) throws Exception {
    
    
    Channel channel = RabbitMqUtils.getChannel();
    //采用手动应答
    boolean autoAck = false;

    //注册消费者，并开始消费消息
    channel.basicConsume(TASK_QUEUE_NAME, autoAck, deliverCallback, cancelCallback);

    System.out.println("C1 等待接收消息处理时间较 短");

    DeliverCallback deliverCallback = (consumerTag, delivery) -> {
    
    
      String message = new String(delivery.getBody());
      try {
    
    
        Thread.sleep(1000);
      } catch (InterruptedException e) {
    
    
        e.printStackTrace();
      }
      System.out.println("接收到消息:" + message);
      //手动确认消息，第二个参数表示消息不重新入队。
      channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);
    };

    CancelCallback cancelCallback = (s) -> {
    
    
      System.out.println(s + "消费者取消消费接口回调逻辑");
    };
  }
}

Consumer 02:

​ Change the time to 30 seconds

::: tip Demonstration of manual response effect

:::

Under normal circumstances, the message sender sends two messages, C1 and C2 receive and process the messages respectively.

After the sender sends message dd, the C2 consumer is stopped after sending the message. Logically speaking, C2 should process the message, but due to its long processing time, it has not been processed yet, which means that C2 has not executed the ack code. At this time, C2 is stopped. At this time, you will see that the message is received by C1, indicating that the message dd is re-enqueued and then assigned to C1 that can process the message.

reject message

If the consumer rejects the message, the basic.reject() or basic.nack() methods can be used.

• basic.reject()Method is used to reject a single message.
• basic.nack()Method used to reject multiple messages.

When the consumer uses the basic.reject() or basic.nack() method to reject the message, it can set requeue=true or < a i=4>. requeue=false

• requeue=trueIndicates that the message will be requeued and consumed by other consumers.
• requeue=falseIndicates that the message will be discarded.

Deny multiple messages

basic.nack() 方法的第一个参数是 deliveryTag 数组，表示要拒绝的消息的 deliveryTag。

第二个参数是 multiple，表示是否拒绝多条消息。如果 multiple=true，则表示拒绝所有 deliveryTag 数组中的值。如果 multiple=false，则表示仅拒绝 deliveryTag 数组中的第一个值。

第三个参数是 requeue，表示是否将消息重新入队。如果 requeue=true，则表示消息将重新入队，并由其他消费者进行消费。如果 requeue=false，则表示消息将被丢弃。

以下是 basic.nack() 方法的使用示例：

// 拒绝所有 `deliveryTag` 数组中的值
channel.basicNack(new int[]{
    
    1, 2, 3}, true, false);

// 仅拒绝 `deliveryTag` 数组中的第一个值
channel.basicNack(new int[]{
    
    1}, false, false);

Reject a single message

channel.basicReject(deliveryTag, false);

In the above code, we rejected the message using the basic.reject() method and set requeue=false to indicate that the message will be discarded.

RabbitMQ persistence

When the RabbitMQ service is stopped, how to ensure that the messages sent by the message producer are not lost? By default when RabbitMQ exits or crashes for some reason, it ignores queues and messages unless it is told not to. Two things need to be done to ensure that messages are not lost:We need to mark both the queue and the message as persistent.

::: tip How to achieve persistence in queues

The queues we created before are non-persistent. If rabbitmq is restarted, the queue will be deleted. If you want the queue to be durable, you need to set the durable parameter to persistent when declaring the queue.

//让队列持久化
boolean durable = true;
//声明队列
channel.queueDeclare(TASK_QUEUE_NAME, durable, false, false, null);

Note: If the previously declared queue is not persistent, you need to delete the original queue or re-create a persistent queue, otherwise an error will occur.

The following is the UI display area of ​​persistent and non-persistent queues in the console,

::: tip Maintain news results

Need to modify the code in the messageproducer and MessageProperties.PERSISTENT_TEXT_PLAIN add this attribute.

Marking a message as persistent does not completely guarantee that the message will not be lost. Although it tells RabbitMQ to save the message to disk, there is still a gap where the message is just about to be stored on disk but has not yet been stored and the message is still cached. Nothing is actually written to disk at this time. The durability guarantees are not strong, but for our simple task queue it's more than enough.

unfair distribution

At the beginning, we learned about the rotation distribution used by RabbitMQ to distribute messages, but this strategy is not very good in certain scenarios. For example, there are two consumers processing tasks, and one of them /span> The processing speed is very slow. At this time, we still use rotation training distribution. The consumer with the fast processing speed will be idle for a large part of the time, while the consumer with the slow processing speed has been working. This distribution The method is actually not very good in this case, but RabbitMQ does not know this situation and it still distributes it fairly. Consumer 2 processes tasks very quickly, while the otherConsumer 1

To avoid this situation, before consuming in the consumer, we can set parameters channel.basicQos(1);

//不公平分发
int prefetchCount = 1;
channel.basicQos(prefetchCount);
//采用手动应答
boolean autoAck = false;
channel.basicConsume(TASK_QUEUE_NAME, autoAck, deliverCallback, cancelCallback);

This means that if I haven't finished processing this task or I haven't responded to you yet, don't assign it to me yet. I can only handle one task at the moment, and then rabbitmq will assign the task to the idle consumer who is not so busy. Of course, if all consumers have not completed the tasks at hand and the queue continues to add new tasks, the queue may be full. At this time, you can only add new workers or change other storage Mission strategy.

Prefetch value distribution

Authorized message distribution

The message itself is sent asynchronously, so at any time, there must be more than one message on the channel. In addition, manual confirmation from the consumer is also asynchronous in nature. So there is an unacknowledged message buffer here, so I hope developers can limit the size of this buffer,To avoid the problem of unlimited unacknowledged messages in the buffer. This can be accomplished by setting the "prefetch count" value using the basic.qos method.

This value defines the maximum number of unacknowledged messages allowed on the channel. **Once the number reaches the configured number, RabbitMQ will stop delivering more messages on the channel unless at least one unprocessed message is acknowledged, for example, assuming there are unacknowledged messages 5, 6, 7, 8 on the channel, And the channel's prefetch count is set to 4. At this time, RabbitMQ will not deliver any messages on the channel unless at least one unanswered message is acked. For example, if the message with tag=6 has just been ACKed, RabbitMQ will sense this situation and send another message. Message acknowledgment and QoS prefetch values ​​have a significant impact on user throughput.

Generally, increasing prefetching will increase the speed of message delivery to consumers. Although the auto-answer transmission message rate is optimal, the number of delivered but not yet processed messages will also increase in this case, thereby increasing the consumer's RAM consumption a> (random access memory) should be careful to use automatic acknowledgment mode or manual acknowledgment mode with unlimited preprocessing. Consumers consume a large number of messages without acknowledgment, which will lead to memory consumption of consumer connection nodes. becomes larger, so finding the right prefetch value is a trial-and-error process that varies for different loads. Values ​​in the range of 100 to 300 usually provide the best throughput and do not impose any penalty on consumers. Too much risk.

A prefetch value of 1 is the most conservative. Of course, this will make the throughput very low, especially when the consumer connection delay is serious, especially in an environment where the consumer connection wait time is long. For most applications, a slightly higher value will be optimal.

Release confirmation

Release confirmation principle

The producer sets the channel to confirm mode. Once the channel enters confirm mode, all messages published on the channel will be assigned a unique ID (starting from 1). Once the message is delivered to all matching queues, the broker A confirmation will be sent to the producer (containing the unique ID of the message), which allows the producer to know that the message has correctly arrived at the destination queue. If the message and queue are durable, the confirmation message will write the message to disk. Afterwards, the delivery-tag field in the confirmation message sent back to the producer by the broker contains the sequence number of the confirmation message. In addition, the broker can also set the multiple field of basic.ack to indicate that all messages up to this sequence number have been received. deal with.

The biggest advantage of confirm mode is that it is asynchronous. Once a message is published, the producer application can continue to send the next message while waiting for the channel to return confirmation. When the message is finally confirmed, the producer application can use the callback method To process the confirmation message, if RabbitMQ causes the message to be lost due to its own internal error, a nack message will be sent. The producer application can also process the nack message in the callback method.

Release confirmed strategy

How to enable release confirmation:

Release confirmation is not enabled by default. If you want to enable it, you need to call the confirmSelect method. Whenever you want to use release confirmation, you need to call this method on the channel.

//开启发布确认
channel.confirmSelect();

Single confirmed release

This is a simple confirmation method. It is a synchronous confirmation publishing method, that is, after publishing a message, only it After being confirmed for publication, subsequent messages can continue to be published. waitForConfirmsOrDie(long) This method will only return when the message is confirmed. If the message is not confirmed within the specified time range, it will throw an exception.

One of the biggest disadvantages of this confirmation method is:The publishing speed is extremely slow, because the published message will be blocked if there is no confirmation Publishing of all subsequent messages, this approach provides a throughput of no more than a few hundred published messages per second. Of course for some applications this may be enough.

/**
 * 单个发送确认
 */
public static void publishMessageIndividually() throws Exception {
    
    
    Channel channel = RabbitMqUtils.getChannel();
    //队列声明
    String queueName = UUID.randomUUID().toString();
    channel.queueDeclare(queueName, true, false, false, null);
    //开启发布确认
    channel.confirmSelect();

    long begin = System.currentTimeMillis();

    for (int i = 0; i < MESSAGE_COUNT; i++) {
    
    
        String message = i + "";
        channel.basicPublish("", queueName, null, message.getBytes());
        //同步确认发布的方式：服务端返回 false 或超时时间内未返回，生产者可以消息重发
        // 等待所有未应答的消息都被确认
       try {
    
    
           Boolean flag = channel.waitForConfirmsOrDie(10000);
           if (flag) {
    
    
               System.out.println("消息发送成功");
            }
        } catch (TimeoutException e) {
    
    
            e.printStackTrace();
        }

    }

    long end = System.currentTimeMillis();
    System.out.println("发布" + MESSAGE_COUNT + "个单独确认消息,耗时" + (end - begin) + "ms");

}

Batch confirmation release

The above method is very slow. Compared with waiting for a single confirmation message, publishing a batch of messages first and then confirming them together can greatly improve the throughput. Of course, the disadvantage of this method is: when a failure occurs, publishing problems will occur. When we don't know which message has the problem, we have to save the entire batch in memory to record important information and then republish the message. Of course, this solution is still synchronous and blocks the release of messages.

/**
 * 批量
 */
public static void publishMessageBatch() throws Exception {
    
    
    Channel channel = RabbitMqUtils.getChannel();
    //队列声明
    String queueName = UUID.randomUUID().toString();
    channel.queueDeclare(queueName, true, false, false, null);
    //开启发布确认
    channel.confirmSelect();
    //批量确认消息大小
    int batchSize = 100;
    //未确认消息个数
    int outstandingMessageCount = 0;
    long begin = System.currentTimeMillis();

    for (int i = 0; i < MESSAGE_COUNT; i++) {
    
    
        String message = i + "";
        channel.basicPublish("", queueName, null, message.getBytes());
        outstandingMessageCount++;
        if (outstandingMessageCount == batchSize) {
    
    
            //进行批量确认
            channel.waitForConfirms();
            outstandingMessageCount = 0;
        }
    }
    //为了确保还有剩余没有确认消息 再次确认
    if (outstandingMessageCount > 0) {
    
    
        channel.waitForConfirms();
    }
    long end = System.currentTimeMillis();
    System.out.println("发布" + MESSAGE_COUNT + "个批量确认消息,耗时" + (end - begin) + "ms");
}

Asynchronous confirmation release

Although the programming logic of asynchronous confirmation is more complicated than the previous two, it is the most cost-effective, both in terms of reliability and efficiency. It uses callback function to achieve reliable message delivery. This middleware also uses function callbacks to ensure whether the delivery is successful. Let us go into details below. Explain how asynchronous confirmation is implemented.

How to handle asynchronous unacknowledged messages?

Unconfirmed messages can be called back a certain number of times in the message callback and logged.

Comparison of the above 3 release confirmation speeds:

• Post messages individually

  Synchronous waiting for confirmation, simple, but very limited throughput.

• Publish messages in batches

  Batch synchronization waits for confirmation, simple, reasonable throughput. Once a problem occurs, it is difficult to infer which message has the problem.

• Asynchronous processing

  Optimum performance and resource usage, well controlled in case of errors, but slightly harder to achieve

switch

Exchanges

The core idea of ​​RabbitMQ messaging model is: Messages produced by producers are never sent directly to queues. In fact, often the producer doesn't even know which queues these messages are being delivered to.

In contrast,The producer can only send messages to the exchange (exchange). The work of the exchange is very simple. On the one hand, it receives messages from Producers on the other hand push them into queues. The switch must know exactly what to do with the received message. Should these messages be put into a specific queue, put them into many queues, or should they be discarded. This depends on the type of switch.

Types of Exchanges:

​ Direct, topic, headers, fanout

Nameless exchange:

​ In the previous part we didn’t know anything about exchange, but we were still able to send messages to the queue. The reason why this was possible before was because we were using the default exchange, and we identified it with an empty string ("").

The first parameter is the name of the switch. An empty string indicates a default or unnamed switch: the message can be routed to the queue specified by the routingKey(bindingkey) binding key, if it exists

temporary queue

In previous chapters we used queues with specific names (remember hello and ack_queue?). The name of the queue is very important to us. We need to specify which queue our consumer will consume messages from.

Every time we connect to RabbitMQ we need a brand new empty queue, for this we can create a queue with a random name, or even better, let the server choose a random queue name for us. Secondly, once we disconnect the consumer, the queue will be automatically deleted.

Createtemporary queue as follows:

String queueName = channel.queueDeclare().getQueue();

bindings bindings

What is binding? Binding is actually the bridge between exchange and queue. It tells us that exchange is bound to that queue. For example, the picture below tells us that X is bound to Q1 and Q2 (generally, the message sent will have a routing key, and the queue bound to the switch will also have a routing key. If the two routing keys match, the message will sent to the specified queue)

Fanout exchange

Introduction to Fanout

Fanout This type is very simple. As you might guess from the name, it broadcasts all messages it receives to all queues it knows about. For Fanout type switches, each consumer will consume the full amount of messages. Moreover, the fanout switch does not require a routing key, and the routing key has no effect on this switch.

Fanout actual combat

The binding relationship between Logs and temporary queue is as shown below

To illustrate this pattern, we will build a simple logging system. It will consist of two programs: the first program will emit log messages, and the second program will be the consumer. We will start two consumers. One consumer will store the log on the disk after receiving the message, and the other consumer will print to the console.

ReceiveLogs01 prints received messages to the console

package com.oddfar.five;

import com.oddfar.utils.RabbitMqUtils;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.DeliverCallback;

/**
 * 消费之把消息打印到控制台
 */
public class ReceiveLogs01 {
    
    
    private static final String EXCHANGE_NAME = "logs";

    public static void main(String[] args) throws Exception {
    
    

        Channel channel = RabbitMqUtils.getChannel();
        //设置交换机类型为fanout
        channel.exchangeDeclare(EXCHANGE_NAME, "fanout");
        /**
         * 生成一个临时的队列 队列的名称是随机的
         * 当消费者断开和该队列的连接时 队列自动删除
         */
        String queueName = channel.queueDeclare().getQueue();
        //把该临时队列绑定我们的 exchange 其中 routingkey(也称之为 binding key)为空字符串
        channel.queueBind(queueName, EXCHANGE_NAME, "");
        System.out.println("等待接收消息,把接收到的消息打印在屏幕........... ");

        //发送回调
        DeliverCallback deliverCallback = (consumerTag, delivery) -> {
    
    
            String message = new String(delivery.getBody(), "UTF-8");
            System.out.println("控制台打印接收到的消息" + message);
        };
        //注册消费者，并且第二个参数设置了 autoAck 属性为 true，这意味着消费者在消费消息后，会自动向 RabbitMQ 确认消息已经被消费。
        channel.basicConsume(queueName, true, deliverCallback, consumerTag -> {
    
    });

    }
}

ReceiveLogs02 writes messages to a file

public class ReceiveLogs02 {
    
    
    private static final String EXCHANGE_NAME = "logs";

    public static void main(String[] args) throws Exception {
    
    

        Channel channel = RabbitMqUtils.getChannel();
        channel.exchangeDeclare(EXCHANGE_NAME, "fanout");
        /**
         * 生成一个临时的队列 队列的名称是随机的
         * 当消费者断开和该队列的连接时 队列自动删除
         */
        String queueName = channel.queueDeclare().getQueue();
        //把该临时队列绑定我们的 exchange 其中 routingkey(也称之为 binding key)为空字符串
        channel.queueBind(queueName, EXCHANGE_NAME, "");
        System.out.println("等待接收消息,把接收到的消息写到文件........... ");

        //发送回调
        DeliverCallback deliverCallback = (consumerTag, delivery) -> {
    
    
            String message = new String(delivery.getBody(), "UTF-8");
            File file = new File("D:\\test\\rabbitmq_info.txt");
            FileUtils.writeStringToFile(file,message,"UTF-8");
            System.out.println("数据写入文件成功");
        };
        //注册消费者
        channel.basicConsume(queueName, true, deliverCallback, consumerTag -> {
    
    });

    }
}

EmitLog sends messages to two consumers for reception:

public class EmitLog {
    
    
    private static final String EXCHANGE_NAME = "logs";

    public static void main(String[] args) throws Exception {
    
    
        Channel channel = RabbitMqUtils.getChannel();

        /**
         * 声明一个 exchange
         * 1.exchange 的名称
         * 2.exchange 的类型为fanout
         */
        channel.exchangeDeclare(EXCHANGE_NAME, "fanout");
        Scanner sc = new Scanner(System.in);
        System.out.println("请输入信息");
        while (sc.hasNext()) {
    
    
            String message = sc.nextLine();
            channel.basicPublish(EXCHANGE_NAME, "", null, message.getBytes("UTF-8"));
            System.out.println("生产者发出消息" + message);
        }
    }
    
}

Direct exchange

In the previous section, we built a simple logging system. We are able to broadcast log messages to many receivers. In this section we will add some special functionality to it - let a consumer subscribe to some of the published messages. For example, we only redirect critical error messages to the log file (to save disk space), while still being able to print all log messages on the console.

Let’s review what bindings are again. Binding is the bridge relationship between switches and queues. It can also be understood this way: The queue is only interested in messages from the switch it is bound to.. Binding is represented by parameter: routingKey, which can also be called binding key. To create binding, we use code: channel.queueBind(queueName, EXCHANGE_NAME, “routingKey”);

The meaning after binding is determined by its exchange type.

Introduction to Direct

A point-to-point switch determines which queue the message is sent to based on the bound routing key.

Our logging system from the previous section broadcasts all messages to all consumers, we want to make some changes to this, for example we want the program that writes log messages to disk to only receive critical errors (errros) and not store warnings (warning) or information (info) log messages to avoid wasting disk space. The Fanout exchange type does not give us a lot of flexibility - it only does unconscious broadcasts. Here we will use the direct type to replace it. The way this type works is that the message only goes to The routingKey it is bound to goes to the queue.

In the picture above, we can see that X is bound to two queues, and the binding type is direct. The binding key of queue Q1 is orange, and the binding key of queue Q2 is two: one binding key is black, and the other binding key is green.

In this binding case, the producer publishes messages to the exchange, and messages with the binding key orange will be published to queue Q1. Messages with binding keys black and green will be published to queue Q2, and messages of other message types will be discarded.

multiple bindings

Of course, if the binding type of exchange is direct,, but if the keys of multiple queues it is bound to are the same, in this case In this case, although the binding type is direct , it behaves a bit like fanout , almost like broadcast, as shown in the figure above.

Direct actual combat

relation:

switch:

c2: Bind disk, routingKey is error

c1: Bind console, routingKey is info, warning

1、

package com.oddfar.six;

import com.oddfar.utils.RabbitMqUtils;
import com.rabbitmq.client.BuiltinExchangeType;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.DeliverCallback;

/**
接受路由键为error的消息
 */
public class ReceiveLogsDirect01 {
    
    
    private static final String EXCHANGE_NAME = "direct_logs";

    public static void main(String[] args) throws Exception {
    
    
        Channel channel = RabbitMqUtils.getChannel();
        //设置交换机类型为DIRECT
        channel.exchangeDeclare(EXCHANGE_NAME, BuiltinExchangeType.DIRECT);

        String queueName = "disk";
        //队列声明
        channel.queueDeclare(queueName, false, false, false, null);
        //队列绑定设定路由键为error
        channel.queueBind(queueName, EXCHANGE_NAME, "error");
        System.out.println("等待接收消息...");

        //发送回调
        DeliverCallback deliverCallback = (consumerTag, delivery) -> {
    
    
            String message = new String(delivery.getBody(), "UTF-8");
            message = "接收绑定键:" + delivery.getEnvelope().getRoutingKey() + ",消息:" + message;
            System.out.println("error 消息已经接收：\n" + message);
        };
        channel.basicConsume(queueName, true, deliverCallback, consumerTag -> {
    
    
        });
    }
}

2、

/**
**接受路由键为info和warning的消息
*/
public class ReceiveLogsDirect02 {
    
    
    private static final String EXCHANGE_NAME = "direct_logs";

    public static void main(String[] args) throws Exception {
    
    
        Channel channel = RabbitMqUtils.getChannel();
        channel.exchangeDeclare(EXCHANGE_NAME, BuiltinExchangeType.DIRECT);

        String queueName = "console";
        //队列声明
        channel.queueDeclare(queueName, false, false, false, null);
        //队列绑定
        channel.queueBind(queueName, EXCHANGE_NAME, "info");
        channel.queueBind(queueName, EXCHANGE_NAME, "warning");

        System.out.println("等待接收消息...");

        //发送回调
        DeliverCallback deliverCallback = (consumerTag, delivery) -> {
    
    
            String message = new String(delivery.getBody(), "UTF-8");
            message = "接收绑定键:" + delivery.getEnvelope().getRoutingKey() + ",消息:" + message;
            System.out.println("info和warning 消息已经接收：\n" + message);
        };
        channel.basicConsume(queueName, true, deliverCallback, consumerTag -> {
    
    
        });
    }
}

3、

public class EmitLogDirect {
    
    
    private static final String EXCHANGE_NAME = "direct_logs";

    public static void main(String[] args) throws Exception {
    
    
        Channel channel = RabbitMqUtils.getChannel();
        channel.exchangeDeclare(EXCHANGE_NAME, BuiltinExchangeType.DIRECT);

        //创建多个 bindingKey
        Map<String, String> bindingKeyMap = new HashMap<>();
        bindingKeyMap.put("info", "普通 info 信息");
        bindingKeyMap.put("warning", "警告 warning 信息");
        bindingKeyMap.put("error", "错误 error 信息");
        //debug 没有消费这接收这个消息 所有就丢失了
        bindingKeyMap.put("debug", "调试 debug 信息");

        for (Map.Entry<String, String> bindingKeyEntry : bindingKeyMap.entrySet()) {
    
    
            //获取 key value 发送不同路由键类型的消息
            String bindingKey = bindingKeyEntry.getKey();
            String message = bindingKeyEntry.getValue();

            channel.basicPublish(EXCHANGE_NAME, bindingKey, null, message.getBytes("UTF-8"));
            System.out.println("生产者发出消息:" + message);
        }
    }
}

Topics exchange

Introduction to Topic

In the previous section, we improved the logging system. Instead of using a fanout switch that can only do random broadcasts, we use a direct switch to selectively receive logs.

Although using the direct switch has improved our system, it still has limitations - for example, the log types we want to receive are info.base and info.advantage, and a certain queue only wants to fuzz the routing key. If the match accepts the message of info.base, direct will not be able to do it at this time. At this time, you can only use the topic type

::: Requirements for tip Topic

:::

The routing_key of the message sent to the topic switch cannot be written arbitrarily and must meet certain requirements. It must bea word list, separated by dots. These words can be any words

For example: "stock.usd.nyse", "nyse.vmw", "quick.orange.rabbit". This type.

Of course, this word list cannot exceed 255 bytes.

In this rule list, there are two replacement characters that everyone needs to pay attention to:

• *(asterisk) can replace a word
• #(pound sign) can replace zero or more words

Topic matching case

The binding relationship in the figure below is as follows

• Q1–>Bound is

  • String with 3 words in middle orange(*.orange.*)

• Q2–>Bound is

  • 3 words with the last word being rabbit(*.*.rabbit)
  • Multiple words where the first word is lazy(lazy.#)

The picture above is a queue binding relationship diagram. Let’s take a look at the data reception situation between them.

example	illustrate
quick.orange.rabbit	Received by queue Q1Q2
his.orange.elephant	Received by queue Q1Q2
quick.orange.fox	Received by queue Q1
lazy.brown.fox	Received by queue Q2
lazy.pink.rabbit	Although two bindings are satisfied, it is only received once by queue Q2.
quick.brown.fox	Not matching any binding will not be received by any queue and will be discarded
quick.orange.male.rabbit	Any binding that does not match any of the four words will be discarded
lazy.orange.male.rabbit	is four words but matches Q2

Notice:

• When a queue binding key is #, then the queue will receive all data, a bit like fanout
• If # and * do not appear in the queue binding key, then the queue binding type is direct.

Topic actual combat

code show as below:

package com.oddfar.seven;

import com.oddfar.utils.RabbitMqUtils;
import com.rabbitmq.client.BuiltinExchangeType;
import com.rabbitmq.client.Channel;

import java.util.HashMap;
import java.util.Map;

/**
 * 发送端 测试topic类型交换机  *(星号)可以代替一个单词 #(井号)可以替代零个或多个单词
  **                       
 */
public class EmitLogTopic {
    
    
    private static final String EXCHANGE_NAME = "topic_logs";

    public static void main(String[] args) throws Exception {
    
    
        Channel channel = RabbitMqUtils.getChannel();
        channel.exchangeDeclare(EXCHANGE_NAME, BuiltinExchangeType.TOPIC);

        /**
         * Q1-->绑定的是
         *      中间带 orange 带 3 个单词的字符串(*.orange.*)
         * Q2-->绑定的是
         *      最后一个单词是 rabbit 的 3 个单词(*.*.rabbit)
         *      第一个单词是 lazy 的多个单词(lazy.#)
         *
         */
        Map<String, String> bindingKeyMap = new HashMap<>();
        bindingKeyMap.put("quick.orange.rabbit", "被队列 Q1Q2 接收到");
        bindingKeyMap.put("lazy.orange.elephant", "被队列 Q1Q2 接收到");
        bindingKeyMap.put("quick.orange.fox", "被队列 Q1 接收到");
        bindingKeyMap.put("lazy.brown.fox", "被队列 Q2 接收到");
        bindingKeyMap.put("lazy.pink.rabbit", "虽然满足两个绑定但只被队列 Q2 接收一次");
        bindingKeyMap.put("quick.brown.fox", "不匹配任何绑定不会被任何队列接收到会被丢弃");
        bindingKeyMap.put("quick.orange.male.rabbit", "是四个单词不匹配任何绑定会被丢弃");
        bindingKeyMap.put("lazy.orange.male.rabbit", "是四个单词但匹配 Q2");
        for (Map.Entry<String, String> bindingKeyEntry : bindingKeyMap.entrySet()) {
    
    
            String bindingKey = bindingKeyEntry.getKey();
            String message = bindingKeyEntry.getValue();

            channel.basicPublish(EXCHANGE_NAME, bindingKey, null, message.getBytes("UTF-8"));
            System.out.println("生产者发出消息：" + message);
        }
    }
}

/**
 * 接收端 测试topic类型交换机   *(星号)可以代替一个单词 #(井号)可以替代零个或多个单词
 */
public class ReceiveLogsTopic01 {
    
    
    private static final String EXCHANGE_NAME = "topic_logs";

    public static void main(String[] args) throws Exception {
    
    
        Channel channel = RabbitMqUtils.getChannel();
        channel.exchangeDeclare(EXCHANGE_NAME, BuiltinExchangeType.TOPIC);
        //声明 Q1 队列与绑定关系
        String queueName = "Q1";
        //声明
        channel.queueDeclare(queueName, false, false, false, null);
        //绑定
        channel.queueBind(queueName, EXCHANGE_NAME, "*.orange.*");
        System.out.println("等待接收消息........... ");

        DeliverCallback deliverCallback = (consumerTag, delivery) -> {
    
    
            String message = new String(delivery.getBody(), "UTF-8");
            System.out.println(" 接收队列:" + queueName + " 绑定键:" + delivery.getEnvelope().getRoutingKey() + ",消息:" + message);
        };
        channel.basicConsume(queueName, true, deliverCallback, consumerTag -> {
    
    
        });
    }
}

/**
 * 接收端 测试topic类型交换机   *(星号)可以代替一个单词 #(井号)可以替代零个或多个单词
 */
public class ReceiveLogsTopic02 {
    
    
    private static final String EXCHANGE_NAME = "topic_logs";

    public static void main(String[] args) throws Exception {
    
    
        Channel channel = RabbitMqUtils.getChannel();
        channel.exchangeDeclare(EXCHANGE_NAME, BuiltinExchangeType.TOPIC);
        //声明 Q2 队列与绑定关系
        String queueName = "Q2";
        //声明
        channel.queueDeclare(queueName, false, false, false, null);
        //绑定
        channel.queueBind(queueName, EXCHANGE_NAME, "*.*.rabbit");
        channel.queueBind(queueName, EXCHANGE_NAME, "lazy.#");

        System.out.println("等待接收消息........... ");

        DeliverCallback deliverCallback = (consumerTag, delivery) -> {
    
    
            String message = new String(delivery.getBody(), "UTF-8");
            System.out.println(" 接收队列:" + queueName + " 绑定键:" + delivery.getEnvelope().getRoutingKey() + ",消息:" + message);
        };
        channel.basicConsume(queueName, true, deliverCallback, consumerTag -> {
    
    
        });
    }
}

dead letter queue

dead letter concept

Let’s first clarify this definition from a conceptual explanation. Dead letters, as the name suggests, are messages that cannot be consumed. The literal meaning can be understood like this. Generally speaking, the producer delivers the message to the broker or directly to the queue. Consumer Messages are taken out from the queue for consumption, but sometimes due to specific reasons some messages in the queue cannot be consumed. If such messages are not followed up If it is processed, it becomes a dead letter. If there is a dead letter, there will naturally be a dead letter queue.

Application scenario: In order to ensure that the message data of the order business is not lost, the dead letter queue mechanism of RabbitMQ needs to be used. When an exception occurs in message consumption, the message is put into the dead letter queue. Another example is: after the user successfully places an order in the mall and clicks to pay, it will automatically expire if the payment is not made within the specified time, achieving the effect of a delayed queue.

Source of dead letters

• Message TTL expired

  TTL is the abbreviation of Time To Live, which is the survival expiration time.

• Queue reaches maximum length

  The queue is full and no more data can be added to mq

• Message rejected

  (basic.reject or basic.nack), single or multiple rejections, and requeue=false, no requeue.

Dead letter practice

Death Faith TTL

Consumer C1 code:

/**
 * 死信队列 - 消费者01
 */
public class Consumer01 {
    
    

    //普通交换机名称
    private static final String NORMAL_EXCHANGE = "normal_exchange";
    //死信交换机名称
    private static final String DEAD_EXCHANGE = "dead_exchange";

    public static void main(String[] args) throws Exception {
    
    
        Channel channel = RabbitMqUtils.getChannel();

        //声明死信和普通交换机 类型为 direct
        channel.exchangeDeclare(NORMAL_EXCHANGE, BuiltinExchangeType.DIRECT);
        channel.exchangeDeclare(DEAD_EXCHANGE, BuiltinExchangeType.DIRECT);

        //声明死信队列绑定死信交换机-路由键为lisi
        String deadQueue = "dead-queue";
        channel.queueDeclare(deadQueue, false, false, false, null);
        //死信队列绑定：队列、交换机、路由键（routingKey）
        channel.queueBind(deadQueue, DEAD_EXCHANGE, "lisi");

        //再把正常队列通过路由键lisi绑定死信交换机，把消息存入死信队列，后续再用消费者消费死信消息实现延时队列
        Map<String, Object> params = new HashMap<>();
        //正常队列设置死信交换机 参数 key 是固定值
        params.put("x-dead-letter-exchange", DEAD_EXCHANGE);
        //正常队列设置死信 routing-key 参数 key 是固定值
        params.put("x-dead-letter-routing-key", "lisi");
        //正常队列
        String normalQueue = "normal-queue";
        channel.queueDeclare(normalQueue, false, false, false, params);
        //正常队列接受路由键为zhangsan的消息，通过上面的死信的map参数再转发到死信队列（生产者给消息设置死信时间）
        channel.queueBind(normalQueue, NORMAL_EXCHANGE, "zhangsan");

        System.out.println("等待接收消息........... ");
        DeliverCallback deliverCallback = (consumerTag, delivery) -> {
    
    
            String message = new String(delivery.getBody(), "UTF-8");
            System.out.println("Consumer01 接收到消息" + message);
        };
        channel.basicConsume(normalQueue, true, deliverCallback, consumerTag -> {
    
    
        });
    }

}

producer code

/**
**生产者给消息设置死信时间
**/
public class Producer {
    
    
    private static final String NORMAL_EXCHANGE = "normal_exchange";

    public static void main(String[] argv) throws Exception {
    
    
        Channel channel = RabbitMqUtils.getChannel();

        channel.exchangeDeclare(NORMAL_EXCHANGE, BuiltinExchangeType.DIRECT);
        //设置消息的 TTL 时间 10s
        AMQP.BasicProperties properties = new AMQP.BasicProperties().builder().expiration("10000").build();
        //该信息是用作演示队列个数限制
        for (int i = 1; i < 11; i++) {
    
    
            String message = "info" + i;
            channel.basicPublish(NORMAL_EXCHANGE, "zhangsan", properties, message.getBytes());
            System.out.println("生产者发送消息:" + message);
        }

    }
}

Start C1 and then close the consumer to simulate that it cannot receive messages. Restart Producer

Consumer C2 code:

After the above steps are completed, start the C2 consumer, which consumes the messages in the dead letter queue and implements a 10-second delay in the delay queue.

public class Consumer02 {
    
    
    //死信交换机名称
    private static final String DEAD_EXCHANGE = "dead_exchange";

    public static void main(String[] args) throws Exception {
    
    
        Channel channel = RabbitMqUtils.getChannel();

        //声明交换机
        channel.exchangeDeclare(DEAD_EXCHANGE, BuiltinExchangeType.DIRECT);
        //声明队列
        String deadQueue = "dead-queue";
        channel.queueDeclare(deadQueue, false, false, false, null);
        channel.queueBind(deadQueue, DEAD_EXCHANGE, "lisi");

        System.out.println("等待接收死信消息........... ");
        DeliverCallback deliverCallback = (consumerTag, delivery) -> {
    
    
            String message = new String(delivery.getBody(), "UTF-8");
            System.out.println("Consumer02 接收到消息" + message);
        };
        channel.basicConsume(deadQueue, true, deliverCallback, consumerTag -> {
    
    
        });
    }
}

Maximum length of dead letter

1. Remove the TTL attribute from the message producer code

2. The C1 consumer modifies the following code** (closes the consumer after startup to simulate that it cannot receive messages), then here the message is entered into the dead letter queue by setting the normal queue length. **

//设置正常队列的长度限制，例如发10个，4个则为死信
params.put("x-max-length",6);

Note that the original queue needs to be deleted at this time because the parameters have changed.

3. C2 consumer code remains unchanged (start C2 consumer)

Dead letter message rejected

1. The code of the message producer is the same as that of the producer above.

2. C1 consumer code (close the consumer after startup to simulate that it cannot receive messages)

Reject message "info5"

/**
**通过拒绝消息不重新入队，让消息变为死信进入死信队列
**/
public class Consumer01 {
    
    
    //普通交换机名称
    private static final String NORMAL_EXCHANGE = "normal_exchange";
    //死信交换机名称
    private static final String DEAD_EXCHANGE = "dead_exchange";

    public static void main(String[] args) throws Exception {
    
    
        Channel channel = RabbitMqUtils.getChannel();

        //声明死信和普通交换机 类型为 direct
        channel.exchangeDeclare(NORMAL_EXCHANGE, BuiltinExchangeType.DIRECT);
        channel.exchangeDeclare(DEAD_EXCHANGE, BuiltinExchangeType.DIRECT);

        //声明死信队列
        String deadQueue = "dead-queue";
        channel.queueDeclare(deadQueue, false, false, false, null);
        //死信队列绑定：队列、交换机、路由键（routingKey）
        channel.queueBind(deadQueue, DEAD_EXCHANGE, "lisi");
        
        //正常队列绑定死信队列信息
        Map<String, Object> params = new HashMap<>();
        //正常队列设置死信交换机 参数 key 是固定值
        params.put("x-dead-letter-exchange", DEAD_EXCHANGE);
        //正常队列设置死信 routing-key 参数 key 是固定值
        params.put("x-dead-letter-routing-key", "lisi");
//        //设置正常队列的长度限制，例如发10个，4个则为死信
//        params.put("x-max-length",6);
        
        //正常队列
        String normalQueue = "normal-queue";
        channel.queueDeclare(normalQueue, false, false, false, params);
        channel.queueBind(normalQueue, NORMAL_EXCHANGE, "zhangsan");

        System.out.println("等待接收消息........... ");
        
        DeliverCallback deliverCallback = (consumerTag, delivery) -> {
    
    
            String message = new String(delivery.getBody(), "UTF-8");
            if (message.equals("info5")) {
    
    
                System.out.println("Consumer01 接收到消息" + message + "并拒绝签收该消息");
                //requeue 设置为 false 代表拒绝重新入队 该队列如果配置了死信交换机将发送到死信队列中
                channel.basicReject(delivery.getEnvelope().getDeliveryTag(), false);
            } else {
    
    
                System.out.println("Consumer01 接收到消息" + message);
                channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);
            }

        };
        //开启手动应答
        channel.basicConsume(normalQueue, false, deliverCallback, consumerTag -> {
    
    
        });
    }

}

3. C2 consumer code remains unchanged

Start consumer 1 and then consumer 2

delay queue

Introduction to delay queue

• Delay queue concept:

Delay queue, the queue is internally ordered, and its most important feature is reflected in its delay attribute. The elements in the delay queue are expected to be taken out and processed after or before the specified time. Simply put, delay A queue is a queue used to store elements that need to be processed at a specified time.

• Delay queue usage scenarios:

1. If the order is not paid within ten minutes, it will be automatically canceled
2. If the newly created store has not uploaded products within ten days, a message reminder will be automatically sent.
3. After the user successfully registers, if the user does not log in within three days, a text message reminder will be sent.
4. The user initiates a refund. If the refund is not processed within three days, the relevant operations personnel will be notified.
5. After booking a meeting, each participant needs to be notified ten minutes before the scheduled time to attend the meeting

These scenarios all have a characteristic that requires completing a certain task at a specified time point after an event occurs or before, such as: When an order generation event occurs, check the payment status of the order after ten minutes, and then process the unpaid order. Close; then we keep polling the data, check it once every second, take out the data that needs to be processed, and then process it, isn't it done?

If the amount of data is relatively small, this can indeed be done. For example, for a requirement such as "automatic settlement if the bill is not paid within a week", if the time is not strictly limited, but a week in a loose sense, then run a A scheduled task to check all unpaid bills is indeed a feasible solution. However, for scenarios with a relatively large amount of data and strong timeliness, such as: "The order will be closed if it is not paid within ten minutes." There may be a lot of unpaid order data in the short term, and even reach millions or even tens of millions during the event. Level, it is obviously not advisable to still use the polling method for such a huge amount of data. It is likely that all orders cannot be completed within one second. At the same time, it will put a lot of pressure on the database, which cannot meet business requirements and has low performance. .

TTL in RabbitMQ

What is TTL? TTL is a property of a message or queue in RabbitMQ, indicating the maximum survival time of a message or all messages in the queue, in milliseconds.

In other words, if a message has the TTL attribute set or enters the queue where the TTL attribute is set, then if the message is not consumed within the time set by the TTL, it will become a "dead letter". If the TTL of the queue and the TTL of the message are configured at the same time, then the smaller value will be used. There are two ways to set the TTL.

• Queue setting TTL

Set the "x-message-ttl" attribute of the queue when creating the queue

• Message settings TTL

Set TTL for each message

::: tip The difference between the two

:::

If the TTL attribute of the queue is set, then once the message expires, it will be discarded by the queue (if the dead letter queue is configured, it will be thrown into the dead letter queue). In the second method, even if the message expires, it may not be immediately discarded. Discard, because whether the message has expired is determined before it is delivered to the consumer. If the current queue has a serious message backlog, the expired message may still survive for a long time;

In addition, one thing to note is thatIf TTL is not set, it means that the message will never expire. If TTL is set to 0, it means that it can be delivered directly unless it is at this time. The message goes to the consumer, otherwise the message will be discarded.

Integrate springboot

We introduced the dead letter queue in the previous section, and just introduced TTL. At this point, the two major elements of using RabbitMQ to implement delay queues have been gathered. Next, we only need to fuse them, add a little bit of seasoning, and delay queues. It will be freshly baked. Think about it, delay queue, doesn't it mean how long the message is delayed to be processed? TTL is just how long the message will be delayed to become a dead letter. On the other hand, the message that becomes a dead letter will be delivered to the dead letter queue. , so that the consumer only needs to keep consuming the messages in the dead letter queue, because the messages inside are all messages that are expected to be processed immediately.

1. Create an empty project:

2. Add dependencies:

<dependencies>
   <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter</artifactId>
    </dependency>
    <!--RabbitMQ 依赖-->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-amqp</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>com.alibaba</groupId>
        <artifactId>fastjson</artifactId>
        <version>1.2.47</version>
    </dependency>
    <dependency>
        <groupId>org.projectlombok</groupId>
        <artifactId>lombok</artifactId>
    </dependency>
    <!--swagger-->
    <dependency>
        <groupId>io.springfox</groupId>
        <artifactId>springfox-swagger2</artifactId>
        <version>3.0.0</version>
    </dependency>
    <dependency>
        <groupId>io.springfox</groupId>
        <artifactId>springfox-swagger-ui</artifactId>
        <version>3.0.0</version>
    </dependency>
    <!--RabbitMQ 测试依赖-->
    <dependency>
        <groupId>org.springframework.amqp</groupId>
        <artifactId>spring-rabbit-test</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>

3. Modify the configuration file

spring.rabbitmq.host=42.192.149.71
spring.rabbitmq.port=5672
spring.rabbitmq.username=admin
spring.rabbitmq.password=123456

4. Add Swagger configuration class

package com.oddfar.config;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import springfox.documentation.builders.ApiInfoBuilder;
import springfox.documentation.service.ApiInfo;
import springfox.documentation.service.Contact;
import springfox.documentation.spi.DocumentationType;
import springfox.documentation.spring.web.plugins.Docket;
import springfox.documentation.swagger2.annotations.EnableSwagger2;


@Configuration
@EnableSwagger2
public class SwaggerConfig {
    
    

    @Bean
    public Docket webApiConfig() {
    
    
        return new Docket(DocumentationType.SWAGGER_2)
                .groupName("webApi")
                .apiInfo(webApiInfo())
                .select()
                .build();
    }

    private ApiInfo webApiInfo() {
    
    
        return new ApiInfoBuilder()
                .title("rabbitmq 接口文档")
                .description("本文档描述了 rabbitmq 微服务接口定义")
                .version("1.0")
                .contact(new Contact("zhiyuan", "http://oddfar.com", "[email protected]"))
                .build();
    }

}

Queue TTL

• Code architecture diagram

Create two queues QA and QB. Set the TTL of the two queues to 10S and 40S respectively. Then create a switch xExchange and a dead letter switch yExchange (the circle in the figure below is the switch). Their types are direct. Create a dead letter switch. Letter queue QD, their binding relationship is as follows:

Originally, the configuration queue information was written in the producer and consumer codes. Now it can be written in the configuration class. The producer only sends messages and the consumer only receives messages.

1. Configuration file class code:

package com.oddfar.config;

import org.springframework.amqp.core.*;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.util.HashMap;
import java.util.Map;

/**
 * 配置MQ元信息
 */
@Configuration
public class TtlQueueConfig {
    
    
    //普通交换机
    public static final String X_EXCHANGE = "X";
    //两个不同队列
    public static final String QUEUE_A = "QA";
    public static final String QUEUE_B = "QB";
    //死信交换机
    public static final String Y_DEAD_LETTER_EXCHANGE = "Y";
    //死信队列
    public static final String DEAD_LETTER_QUEUE = "QD";

    // 声明 xExchange普通direct类型交换机
    @Bean("xExchange")
    public DirectExchange xExchange() {
    
    
        return new DirectExchange(X_EXCHANGE);
    }

    // 声明 y死信队列交换机
    @Bean("yExchange")
    public DirectExchange yExchange() {
    
    
        return new DirectExchange(Y_DEAD_LETTER_EXCHANGE);
    }

    // 声明队列 A 绑定 X 普通交换机 并指定路由键为XA
    @Bean
    public Binding queueaBindingX(@Qualifier("queueA") Queue queueA,
                                  @Qualifier("xExchange") DirectExchange xExchange) {
    
    
        return BindingBuilder.bind(queueA).to(xExchange).with("XA");
    }
   //声明队列 B 绑定 X普通交换机 并指定路由键为XB
    @Bean
    public Binding queuebBindingX(@Qualifier("queueB") Queue queue1B,
                                  @Qualifier("xExchange") DirectExchange xExchange) {
    
    
        return BindingBuilder.bind(queue1B).to(xExchange).with("XB");
    }
   //声明队列 A ttl 为 10s 并绑定到对应的死信交换机
    @Bean("queueA")
    public Queue queueA() {
    
    
        Map<String, Object> args = new HashMap<>(3);
        //声明当前队列绑定的死信交换机
        args.put("x-dead-letter-exchange", Y_DEAD_LETTER_EXCHANGE);
        //声明当前队列的死信路由 key
        args.put("x-dead-letter-routing-key", "YD");
        //声明队列的 TTL 过期时间
        args.put("x-message-ttl", 10000);
        return QueueBuilder.durable(QUEUE_A).withArguments(args).build();
    }
    //声明队列 B ttl 为 40s 并绑定到对应的死信交换机
    @Bean("queueB")
    public Queue queueB() {
    
    
        Map<String, Object> args = new HashMap<>(3);
        //声明当前队列绑定的死信交换机
        args.put("x-dead-letter-exchange", Y_DEAD_LETTER_EXCHANGE);
        //声明当前队列的死信路由 key
        args.put("x-dead-letter-routing-key", "YD");
        //声明队列的 TTL 过期时间
        args.put("x-message-ttl", 40000);
        return QueueBuilder.durable(QUEUE_B).withArguments(args).build();
    }

    //声明死信队列 QD
    @Bean("queueD")
    public Queue queueD() {
    
    
        return new Queue(DEAD_LETTER_QUEUE);
    }

    //声明死信队列 QD 绑定关系，让消息进入QD死信队列
    @Bean
    public Binding deadLetterBindingQAD(@Qualifier("queueD") Queue queueD,
                                        @Qualifier("yExchange") DirectExchange yExchange) {
    
    
        return BindingBuilder.bind(queueD).to(yExchange).with("YD");
    }

}

2. Message producer code

package com.oddfar.contorller;

import lombok.extern.slf4j.Slf4j;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.Date;

/**
 * 生产者向两个普通队列发送消息
 */
@Slf4j
@RequestMapping("ttl")
@RestController
public class SendMsgController {
    
    
    @Autowired
    private RabbitTemplate rabbitTemplate;

    @GetMapping("sendMsg/{message}")
    public void sendMsg(@PathVariable String message) {
    
    
        log.info("当前时间：{},发送一条信息给两个 TTL 队列:{}", new Date(), message);
        //convertAndSend() 方法的第一个参数是交换机名称。第二个参数是路由键。第三个参数是消息内容。
        rabbitTemplate.convertAndSend("X", "XA", "消息来自 ttl 为 10S 的队列: " + message);
        rabbitTemplate.convertAndSend("X", "XB", "消息来自 ttl 为 40S 的队列: " + message);
    }
    
}

3. Message consumer code

package com.oddfar.consumer;

import com.rabbitmq.client.Channel;
import lombok.extern.slf4j.Slf4j;
import org.springframework.amqp.core.Message;
import org.springframework.amqp.rabbit.annotation.RabbitListener;
import org.springframework.stereotype.Component;

import java.io.IOException;
import java.util.Date;

/**
 * 消费者 - 死信队列，消费死信队列的消息，实现延时效果
 */
@Slf4j
@Component
public class DeadLetterQueueConsumer {
    
    

    @RabbitListener(queues = "QD")
    public void receiveD(Message message, Channel channel) throws IOException {
    
    
        String msg = new String(message.getBody());
        log.info("当前时间：{},收到死信队列信息{}", new Date().toString(), msg);
    }

}

Initiate a request http://localhost:8080/ttl/sendMsg/heeheehee

The first message becomes a dead letter message after 10S and is then consumed by the consumer. The second message becomes a dead letter message after 40S and is then consumed. In this way, a delay queue is completed.

However, if used in this way, wouldn't it mean that every time a new time requirement is added, a new queue must be added. There are only two time options of 10S and 40S. If it needs to be processed after one hour, then the TTL needs to be increased to one hour. Queue, if it is a scenario like booking a conference room and notifying it in advance, wouldn’t it be necessary to add countless queues to meet the demand?

Delay queue TTL optimization

A new queue QC is added here. The binding relationship is as follows. The queue does not set a TTL time.

Configuration file class code:

@Configuration
public class MsgTtlQueueConfig {
    
    
    public static final String Y_DEAD_LETTER_EXCHANGE = "Y";
    public static final String QUEUE_C = "QC";

    //声明普通队列 C ，而是给消息设置过期时间，进入死信队列，消息过期时间设置为参数由前端传递
    @Bean("queueC")
    public Queue queueB() {
    
    
        Map<String, Object> args = new HashMap<>(3);
        //声明当前队列绑定的死信交换机
        args.put("x-dead-letter-exchange", Y_DEAD_LETTER_EXCHANGE);
        //声明当前队列的死信路由 key
        args.put("x-dead-letter-routing-key", "YD");
        //没有声明 TTL 属性
        return QueueBuilder.durable(QUEUE_C).withArguments(args).build();
    }

    //声明队列 B 绑定 X 交换机
    @Bean
    public Binding queuecBindingX(@Qualifier("queueC") Queue queueC,
                                  @Qualifier("xExchange") DirectExchange xExchange) {
    
    
        return BindingBuilder.bind(queueC).to(xExchange).with("XC");
    }
}

Producer code:

/**
 * 延时队列优化  设置消息的过期时间，绑定了死信队列之后，消息到了过期时间就会进入死信队列
 * @param message 消息
 * @param ttlTime 延时的毫秒
 */
@GetMapping("sendExpirationMsg/{message}/{ttlTime}")
public void sendMsg(@PathVariable String message, @PathVariable String ttlTime) {
    
    
    rabbitTemplate.convertAndSend("X", "XC", message, correlationData -> {
    
    
        correlationData.getMessageProperties().setExpiration(ttlTime);
        return correlationData;
    });
    log.info("当前时间：{},发送一条时长{}毫秒 TTL 信息给队列 C:{}", new Date(), ttlTime, message);
}

Make a request

http://localhost:8080/ttl/sendExpirationMsg/Hello1/20000

http://localhost:8080/ttl/sendExpirationMsg/Hello2/2000

There seems to be no problem, but at the beginning, it was introduced that if you use the method of setting TTL on the message properties, the message may not "die" on time.

Because RabbitMQ will only check whether the first message has expired, and if it expires, it will be thrown into the dead letter queue. If the delay time of the first message is very long, and the delay time of the second message is very short, the second message will not Will not be prioritized for execution.

This is why the second one is delayed for 2 seconds and then executed.

Rabbitmq plug-in implements delay queue

The problem mentioned above is indeed a problem. If TTL cannot be implemented at message granularity and it dies in time at the set TTL time, it cannot be designed as a general delay queue. So how to solve it? Next we will solve this problem.

::: tip Install delay queue plug-in

You can download the rabbitmq_delayed_message_exchange plug-in from the official website and place it in the RabbitMQ plug-in directory.

Enter the plgins directory under the RabbitMQ installation directory, execute the following command to make the plug-in take effect, and then restart RabbitMQ

[root@VM-0-6-centos software]# ls
erlang-21.3.8.21-1.el7.x86_64.rpm  rabbitmq_delayed_message_exchange-3.8.0.ez  rabbitmq-server-3.8.8-1.el7.noarch.rpm
#移动
cp rabbitmq_delayed_message_exchange-3.8.0.ez /usr/lib/rabbitmq/lib/rabbitmq_server-3.8.8/plugins
#安装
rabbitmq-plugins enable rabbitmq_delayed_message_exchange
#重启服务
systemctl restart rabbitmq-server

::: tip code

A new queue delayed.queue and a custom switch delayed.exchange are added here. The binding relationship is as follows:

1. Configuration file code:

In our custom switch, this is a new exchange type. This type of message supports a delayed delivery mechanism. After the message is delivered, it will not be delivered to the target queue immediately, but will be stored in the mnesia (a distributed data system) table. When the delivery time is reached, it will be delivered to the target queue.

@Configuration
public class DelayedQueueConfig {
    
    
    public static final String DELAYED_QUEUE_NAME = "delayed.queue";
    public static final String DELAYED_EXCHANGE_NAME = "delayed.exchange";
    public static final String DELAYED_ROUTING_KEY = "delayed.routingkey";

    @Bean
    public Queue delayedQueue() {
    
    
        return new Queue(DELAYED_QUEUE_NAME);
    }

    //自定义交换机 我们在这里定义的是一个延迟交换机
    @Bean
    public CustomExchange delayedExchange() {
    
    
        Map<String, Object> args = new HashMap<>();
        //自定义交换机的类型
        args.put("x-delayed-type", "direct");
        return new CustomExchange(DELAYED_EXCHANGE_NAME, "x-delayed-message", true, false, args);
    }

    @Bean
    public Binding bindingDelayedQueue(@Qualifier("delayedQueue") Queue queue,
                                       @Qualifier("delayedExchange") CustomExchange delayedExchange) {
    
    
        return BindingBuilder.bind(queue).to(delayedExchange).with(DELAYED_ROUTING_KEY).noargs();
    }

}

2. Producer code

@GetMapping("sendDelayMsg/{message}/{delayTime}")
public void sendMsg(@PathVariable String message, @PathVariable Integer delayTime) {
    
    
    rabbitTemplate.convertAndSend(DELAYED_EXCHANGE_NAME, DELAYED_ROUTING_KEY, message,
            correlationData -> {
    
    
                //设置消息进入mnesia(一个分布式数据系统)表，到达延时时间后进入延时队列
                correlationData.getMessageProperties().setDelay(delayTime);
                return correlationData;
            });
    log.info(" 当 前 时 间 ： {}, 发 送 一 条 延 迟 {} 毫秒的信息给队列 delayed.queue:{}", new Date(), delayTime, message);
}

3. Consumer code

/**
 * 消费者 - 基于插件的延时队列
 */
@Slf4j
@Component
public class DelayQueueConsumer {
    
    

    public static final String DELAYED_QUEUE_NAME = "delayed.queue";

    @RabbitListener(queues = DELAYED_QUEUE_NAME)
    public void receiveDelayedQueue(Message message) {
    
    
        String msg = new String(message.getBody());
        log.info("当前时间：{},收到延时队列的消息：{}", new Date().toString(), msg);
    }
}

send request:

• http://localhost:8080/ttl/sendDelayMsg/hello1/20000
• http://localhost:8080/ttl/sendDelayMsg/hello2/2000

The second message was consumed first, in line with expectations.

Delay queue summary

Delay queues are very useful in scenarios that require delayed processing. Using RabbitMQ to implement delay queues can make good use of RabbitMQ's features, such as: reliable message sending, reliable message delivery, and dead letter queues to ensure that messages are consumed at least once. And messages that are not processed correctly are not discarded. In addition, through the characteristics of RabbitMQ cluster, the problem of single point of failure can be well solved, and the delay queue will not be unavailable or messages will be lost due to the failure of a single node.

Of course, there are many other options for delay queues, such as using Java's DelayQueue, using Redis's zset, using Quartz or using kafka's time wheel. Each of these methods has its own characteristics, depending on the applicable scenario.

Release Confirmation Advanced

In the production environment, due to some unknown reasons, RabbitMQ restarts. During the restart of RabbitMQ, the producer message delivery fails, resulting in message loss, which requires manual processing and recovery. So, we started thinking, how can we achieve reliable message delivery in RabbitMQ? Create a backup of the message and store it in the cache, send the message in the cache regularly, and clear the cache if it is successfully consumed.

Two callback functions are used here to ensure reliable delivery of messages

ConfirmCallback: The message will be executed when it reaches the broker (MQ server);

ReturnsCallback: The message will be executed if it does not arrive in the queue normally;

ConfirmCallback-confirmation message

Confirmation mechanism plan:

Code architecture diagram:

Need to be added to the configuration file

spring.rabbitmq.publisher-confirm-type=correlated

• NONEThe value is to disable release confirmation mode, which is the default value

• CORRELATEDThe value is that the callback method will be triggered after the message is successfully published to the exchange.

• SIMPLEThe value has been tested and has two effects. The first effect is the same as the CORRELATED value and will trigger the callback method. The second effect is to use rabbitTemplate to call the waitForConfirms or waitForConfirmsOrDie method after the message is successfully published, waiting for the broker node to return the sending result, and determine the next logic based on the return result. , the point to note is that if the waitForConfirmsOrDie method returns false, the channel will be closed, and messages will not be sent to the broker;

::: tip code

1. Add configuration class:

@Configuration
public class ConfirmConfig {
    
    
    public static final String CONFIRM_EXCHANGE_NAME = "confirm.exchange";
    public static final String CONFIRM_QUEUE_NAME = "confirm.queue";

    //声明业务 Exchange
    @Bean("confirmExchange")
    public DirectExchange confirmExchange() {
    
    
        return new DirectExchange(CONFIRM_EXCHANGE_NAME);
    }

    // 声明确认队列
    @Bean("confirmQueue")
    public Queue confirmQueue() {
    
    
        return QueueBuilder.durable(CONFIRM_QUEUE_NAME).build();
    }

    // 声明确认队列绑定关系路由键为key1
    @Bean
    public Binding queueBinding(@Qualifier("confirmQueue") Queue queue,
                                @Qualifier("confirmExchange") DirectExchange exchange) {
    
    
        return BindingBuilder.bind(queue).to(exchange).with("key1");
    }
}

2. Callback interface of message producer

@Component
@Slf4j
/**
**消息抵达broker就会执行
**/
public class MyCallBack implements RabbitTemplate.ConfirmCallback {
    
    
    /**
     * 交换机不管是否收到消息的一个回调方法
     * @param correlationData 消息相关数据
     * @param ack             交换机是否收到消息
     * @param cause           为收到消息的原因
     */
    @Override
    public void confirm(CorrelationData correlationData, boolean ack, String cause) {
    
    
        String id = correlationData != null ? correlationData.getId() : "";
        if (ack) {
    
    
            log.info("交换机已经收到 id 为:{}的消息", id);
        } else {
    
    
            log.info("交换机还未收到 id 为:{}消息，原因:{}", id, cause);
        }
    }

}

3. Message producer

/**
**生产者发送两条消息，并指定自定义的消息回调函数
**/
@RestController
@RequestMapping("/confirm")
@Slf4j
public class ProducerController {
    
    
    public static final String CONFIRM_EXCHANGE_NAME = "confirm.exchange";
    @Autowired
    private RabbitTemplate rabbitTemplate;
    @Autowired
    private MyCallBack myCallBack;

    //依赖注入 rabbitTemplate 之后再设置它的回调对象(上面配置的自定义回调函数)
    @PostConstruct
    public void init() {
    
    
        rabbitTemplate.setConfirmCallback(myCallBack);
    }
    
    /**
     * 消息回调和退回
     * @param message
     */
    @GetMapping("sendMessage/{message}")
    public void sendMessage(@PathVariable String message) {
    
    

        //指定消息 id 为 1
        CorrelationData correlationData1 = new CorrelationData("1");
        String routingKey = "key1";
        //手动拒绝消息
        rabbitTemplate.convertAndSend(CONFIRM_EXCHANGE_NAME, routingKey, message + routingKey, correlationData1);
        log.info(routingKey + "发送消息内容:{}", message + routingKey);

        CorrelationData correlationData2 = new CorrelationData("2");
        routingKey = "key2";
        rabbitTemplate.convertAndSend(CONFIRM_EXCHANGE_NAME, routingKey, message + routingKey, correlationData2);
        log.info(routingKey + "发送消息内容:{}", message + routingKey);

    }

}

4. Message consumers

@Component
@Slf4j
public class ConfirmConsumer {
    
    
    public static final String CONFIRM_QUEUE_NAME = "confirm.queue";

    @RabbitListener(queues = CONFIRM_QUEUE_NAME)
    public void receiveMsg(Message message) {
    
    
        String msg = new String(message.getBody());
        log.info("接受到队列 confirm.queue 消息:{}", msg);
    }

}

Visit: http://localhost:8080/confirm/sendMessage/Hello

Result analysis:

It can be seen that two messages were sent, the RoutingKey of the first message is "key1", and the RoutingKey of the second message is "key2". Both messages were successfully received by the switch, and a confirmation callback from the switch was also received, but The consumer only received one message because the RoutingKey of the second message was inconsistent with the BindingKey of the queue and no other queue could receive the message. All the second messages were discarded directly.

The dropped message is unknown to the exchange and needs to be resolved by telling the producer that the message delivery failed

ReturnsCallback-return message

Mandatory parameters

rabbitTemplate.setReturnsCallback(myCallBack);

When only the producer confirmation mechanism is enabled, after receiving the message, the switch will directly send a confirmation message to the message producer. If the message is found to be unroutable, the message will be directly discarded. At this time, the producer does not know the message. This event was discarded.

So how can I help me find a way to handle messages that cannot be routed? At least let me know so I can handle it myself. By setting the mandatory parameter, the message can be returned to the producer when the destination cannot be reached during message delivery.

1. Modify configuration

#消息退回
spring.rabbitmq.publisher-returns=true

2. Modify the callback interface

@Component
@Slf4j
public class MyCallBack implements RabbitTemplate.ConfirmCallback, RabbitTemplate.ReturnsCallback {
    
    

    /**
     * 交换机不管是否收到消息的一个回调方法
     *
     * @param correlationData 消息相关数据
     * @param ack             交换机是否收到消息
     * @param cause           为收到消息的原因
     */
    @Override
    public void confirm(CorrelationData correlationData, boolean ack, String cause) {
    
    
        String id = correlationData != null ? correlationData.getId() : "";
        if (ack) {
    
    
            log.info("交换机已经收到 id 为:{}的消息", id);
        } else {
    
    
            log.info("交换机还未收到 id 为:{}消息，原因:{}", id, cause);
        }
    }

    //当消息无法路由的时候的回调方法
    @Override
    public void returnedMessage(ReturnedMessage returned) {
    
    

        log.error("消息：{}，被交换机 {} 退回，原因：{}，路由key：{},code:{}",
                new String(returned.getMessage().getBody()), returned.getExchange(),
                returned.getReplyText(), returned.getRoutingKey(),
                returned.getReplyCode());

    }
}

Lower versions may not have it RabbitTemplate.ReturnsCallback Please use RabbitTemplate.ReturnCallback

@Override
public void returnedMessage(Message message, int replyCode, String replyText, String
exchange, String routingKey) {
    
    
	log.info("消息:{}被服务器退回，退回原因:{}, 交换机是:{}, 路由 key:{}",new String(message.getBody()),replyText, exchange, routingKey);
}

3. Modify the sender ProducerController

//依赖注入 rabbitTemplate 之后再设置它的回调对象
@PostConstruct
public void init() {
    
    
    //消息回调
    rabbitTemplate.setConfirmCallback(myCallBack);
    /**
     * true：交换机无法将消息进行路由时，会将该消息返回给生产者
     * false：如果发现消息无法进行路由，则直接丢弃
     */
    rabbitTemplate.setMandatory(true);
    //设置回退消息交给谁处理
    rabbitTemplate.setReturnsCallback(myCallBack);

}

Visit: http://localhost:8080/confirm/sendMessage/Hello

Result analysis:

Backup switch

With the mandatory parameter and fallback message, we gain the ability to perceive undeliverable messages and discover and handle when the producer's message cannot be delivered. But sometimes, we don't know how to deal with these unroutable messages. At most, we can log them, trigger an alarm, and then handle them manually. It is very inelegant to handle these unroutable messages through logs, especially when the service where the producer is located has multiple machines. Manually copying logs is more cumbersome and error-prone. Moreover, setting the mandatory parameter will increase the complexity of the producer, and logic needs to be added to handle these returned messages. What should you do if you don’t want to lose messages but don’t want to increase the complexity of the producer?

In the previous article on setting up a dead letter queue, we mentioned that a dead letter switch can be set up for the queue to store messages that failed to be processed. However, these non-routable messages have no chance to enter the queue, so the dead letter queue cannot be used to save messages. . In RabbitMQ, there is a backup switch mechanism that can deal with this problem very well.

What isa backup switch? The backup switch can be understood as the "spare tire" of the switch in RabbitMQ. When we declare a corresponding backup switch for a certain switch, we create a spare tire for it. **When the switch receives an unroutable message, it will Forward this message to the backup switch, and the backup switch will forward and process it. Usually the type of backup switch is Fanout. **In this way, all messages can be delivered to the queue bound to it, and then we will use the backup switch to Bind a queue so that all messages that cannot be routed by the original switch will enter this queue. Of course, we can also create an alarm queue and use independent consumers to monitor and alarm.

• Code architecture diagram

1. Modify configuration class

@Configuration
public class ConfirmConfig {
    
    
    public static final String CONFIRM_EXCHANGE_NAME = "confirm.exchange";
    public static final String CONFIRM_QUEUE_NAME = "confirm.queue";
    //关于备份的
    public static final String BACKUP_EXCHANGE_NAME = "backup.exchange";
    public static final String BACKUP_QUEUE_NAME = "backup.queue";
    public static final String WARNING_QUEUE_NAME = "warning.queue";


    /*
    //声明业务 Exchange
    @Bean("confirmExchange")
    public DirectExchange confirmExchange() {
        return new DirectExchange(CONFIRM_EXCHANGE_NAME);
    }
    */

    // 声明确认队列
    @Bean("confirmQueue")
    public Queue confirmQueue() {
    
    
        return QueueBuilder.durable(CONFIRM_QUEUE_NAME).build();
    }

    // 声明确认队列绑定关系
    @Bean
    public Binding queueBinding(@Qualifier("confirmQueue") Queue queue,
                                @Qualifier("confirmExchange") DirectExchange exchange) {
    
    
        return BindingBuilder.bind(queue).to(exchange).with("key1");
    }

    //************************以下是关于备份的******************************

    //声明备份 Exchange
    @Bean("backupExchange")
    public FanoutExchange backupExchange() {
    
    
        return new FanoutExchange(BACKUP_EXCHANGE_NAME);
    }

    //声明确认 业务Exchange 交换机的备份交换机，再通过备份交换机绑定警告队列和备份队列
    @Bean("confirmExchange")
    public DirectExchange confirmExchange() {
    
    
        ExchangeBuilder exchangeBuilder = ExchangeBuilder.directExchange(CONFIRM_EXCHANGE_NAME)
                .durable(true) //设置消息持久化
                //设置该交换机的备份交换机
                .withArgument("alternate-exchange", BACKUP_EXCHANGE_NAME);
        return exchangeBuilder.build();
    }

    // 声明警告队列
    @Bean("warningQueue")
    public Queue warningQueue() {
    
    
        return QueueBuilder.durable(WARNING_QUEUE_NAME).build();
    }

    // 声明报警队列绑定关系
    @Bean
    public Binding warningBinding(@Qualifier("warningQueue") Queue queue,
                                  @Qualifier("backupExchange") FanoutExchange backupExchange) {
    
    
        return BindingBuilder.bind(queue).to(backupExchange);
    }

    // 声明备份队列
    @Bean("backQueue")
    public Queue backQueue() {
    
    
        return QueueBuilder.durable(BACKUP_QUEUE_NAME).build();
    }

    // 声明备份队列绑定关系
    @Bean
    public Binding backupBinding(@Qualifier("backQueue") Queue queue,
                                 @Qualifier("backupExchange") FanoutExchange backupExchange) {
    
    
        return BindingBuilder.bind(queue).to(backupExchange);
    }
}

2. Alarm consumers

@Component
@Slf4j
public class WarningConsumer {
    
    
    public static final String WARNING_QUEUE_NAME = "warning.queue";

    @RabbitListener(queues = WARNING_QUEUE_NAME)
    public void receiveWarningMsg(Message message) {
    
    
        String msg = new String(message.getBody());
        log.error("报警发现不可路由消息：{}", msg);
    }
}

Has been written before confirm.exchange Switch needs to be deleted due to configuration changes, otherwise an error will be reported

• Visit: http://localhost:8080/confirm/sendMessage/Hello

When the mandatory parameter and the backup switch can be used together, if both are enabled at the same time, where will the message go? Who has the higher priority? The above results show that the answer is:The backup switch has the highest priority.

priority queue

• scenes to be used

There is an order reminder scenario in our system. When our customers place an order on Tmall, Taobao will push the order to us in time. If payment is not made within the time set by the user, a text message reminder will be pushed to the user. , a very simple function, right?

However, for us, tmall merchants must be divided into large customers and small customers, right? For example, large merchants like Apple and Xiaomi can create a lot of profits for us at least a year, so of course, their orders must be Get priority processing, and our back-end system used to use redis to store regular polling. Everyone knows that redis can only use List to make a simple message queue, and cannot realize a priority scenario, so the order volume After it gets bigger, use RabbitMQ for transformation and optimization. If it is found that the order is from a large customer, it will be given a relatively high priority, otherwise it will be the default priority.

You can set priorities for queues or messages, and those with higher priorities will be executed first.

• How to add?

a. Add console page

b. Add priority to the code in the queue

Map<String, Object> params = new HashMap();
params.put("x-max-priority", 10);
channel.queueDeclare("hello", true, false, false, params);

c. Add priority to the code in the message

AMQP.BasicProperties properties = new AMQP.BasicProperties().builder().priority(10).build();

Precautions:

The things that need to be done in order for the queue to implement priority are the following: the queue needs to be set as a priority queue, the message needs to set the priority of the message, and the consumer needs to wait for the message to be sent to the queue before consuming it because, in this way, there is a chance to Messages are sorted

::: tip in practice

Producer:

/**
**设置消息的优先级
**/
public class PriorityProducer {
    
    
    private static final String QUEUE_NAME = "hello";

    public static void main(String[] args) throws Exception {
    
    
        Channel channel = RabbitMqUtils.getChannel();

        //给消息赋予一个 priority 属性
        AMQP.BasicProperties properties = new AMQP.BasicProperties().builder().priority(10).build();

        for (int i = 1; i < 11; i++) {
    
    
            String message = "info" + i;
            if (i == 5) {
    
    
                //这里为5的消息优先级更高会先执行
                channel.basicPublish("", QUEUE_NAME, properties, message.getBytes());
            } else {
    
    
                channel.basicPublish("", QUEUE_NAME, null, message.getBytes());
            }
            System.out.println("发送消息完成:" + message);
        }
    }

}

consumer:

public class PriorityConsumer {
    
    
    private final static String QUEUE_NAME = "hello";

    public static void main(String[] args) throws Exception {
    
    
        Channel channel = RabbitMqUtils.getChannel();

        //设置队列的最大优先级 最大可以设置到 255 官网推荐 1-10 如果设置太高比较吃内存和 CPU
        Map<String, Object> params = new HashMap();
        params.put("x-max-priority", 10);
        channel.queueDeclare(QUEUE_NAME, true, false, false, params);

        //消费的接口回调
        DeliverCallback deliverCallback = (consumerTag, delivery) -> {
    
    
            String message = new String(delivery.getBody());
            System.out.println(message);
        };
        //取消消费的一个回调接口 如在消费的时候队列被删除掉了
        CancelCallback cancelCallback = (consumerTag) -> {
    
    
            System.out.println("消息消费被中断");
        };
        //创建消费者，这里没有设置手动接收消息就是采用的默认接收接收消息的方式
        channel.basicConsume(QUEUE_NAME, true, deliverCallback, cancelCallback);
    }

}

lazy queue

• scenes to be used

RabbitMQ introduced the concept of lazy queue starting from version 3.6.0. **The lazy queue will store messages on disk as much as possible, and will only be loaded into memory when the consumer consumes the corresponding message.An important design goal of it is Able tosupport longer queues, that is, support more message storage. **When consumers are unable to consume messages for a long time due to various reasons (such as consumers going offline, downtime, or shutting down due to maintenance, etc.), lazy queues are necessary.

By default, when a producer sends a message to RabbitMQ, the message in the queue will be stored in memory as much as possible, so that the message can be sent to the consumer faster. Even persistent messages retain a copy in memory while being written to disk. When RabbitMQ needs to release memory, it will page the messages in the memory to the disk. This operation will take a long time and will also block the queue operation, making it impossible to receive new messages. Although RabbitMQ developers have been upgrading related algorithms, the results are still not ideal, especially when the message volume is particularly large.

• Two modes

The queue has two modes: default and lazy. The default mode is default, and no changes are required for versions prior to 3.6.0. The lazy mode is the mode of the lazy queue. It can be set in the parameters when calling the channel.queueDeclare method, or it can be set through the Policy. If a queue is set using both methods at the same time, the Policy method has higher priority. If you want to change the mode of an existing queue through declaration, you can only delete the queue first and then re-declare a new one.

When declaring the queue, you can set the mode of the queue through the "x-queue-mode" parameter. The values ​​are "default" and "lazy". The following example demonstrates the declaration details of a lazy queue:

Map<String, Object> args = new HashMap<String, Object>();
args.put("x-queue-mode", "lazy");
channel.queueDeclare("myqueue", false, false, false, args);

• Memory overhead comparison

When 1 million messages are sent and each message occupies approximately 1KB, the memory occupied by the ordinary queue is 1.2GB, while the lazy queue only occupies 1.5MB

Extensions:

1. The difference between message response @RabbitListener and @RabbitHandler

1. The @RabbitListener annotation specifies a method as a message consumption method, such as listening to messages in a Queue.

2. @RabbitListener is marked on the method and directly listens to the specified queue. At this time, the parameters received need to be consistent with the sending city type.

@Component
public class PointConsumer {
    
    
	//监听的队列名
    @RabbitListener(queues = "point.to.point")
    public void processOne(String name) {
    
    
        System.out.println("point.to.point：" + name);
    }
}

3.@RabbitListener can be marked on the class and needs to be used together with the @RabbitHandler annotation
@RabbitListener is marked on the class to indicate that when a message is received, it will be handed over to @ The method processing of RabbitHandler enters the specific method according to the accepted parameter type.

@Component
@RabbitListener(queues = "consumer_queue")
public class Receiver {
    
    

@RabbitHandler
public void processMessage1(String message) {
    
    
    System.out.println(message);
}

@RabbitHandler
public void processMessage2(byte[] message) {
    
    
    System.out.println(new String(message));
}

2. Retry of message consumption

You can refer to the following articles:

RabbitMQ (3) Message retries_retries exhausted for message-CSDN Blog

Generally, the failed message can be re-queued after manual unack, but this will cause problems. The failed message will be placed at the head of the queue, resulting in repeated attempts of failed messages, which can easily cause memory problems. The correct approach is to use spring-retry to retry failed messages for a certain number of times before putting them into the dead letter queue, and then process the dead letter queue, report logs, manually intervene, etc.

Here's an example of retrying:

1. pom dependency

<dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
 
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-amqp</artifactId>
        </dependency>
 
       
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <scope>compile</scope>
        </dependency>
 
</dependencies>

When an exception occurs in message consumption, use the retry mechanism provided by springboot to retry. Because spring-retry is used, the exception must be thrown in the method, otherwise spring-retry will not be triggered! ! !

2. Configuration file application.properties:

Spring.rabbitmq.listener.simple.acknowledge-mode=manual is commented out here, so that when message consumption fails, it will automatically go to the dead letter queue. If the manual confirmation mechanism is enabled, chanel.basicNack(tag,false,false) must be called ) the message will enter the dead letter queue! ! !

# 应用名称
spring.application.name=rabbitmq
server.port=8080
server.servlet.context-path=/

spring.rabbitmq.host=localhost
spring.rabbitmq.port=5672
# 指定连接的虚拟主机，可以在rabbitMQ控制台查看对应的虚拟主机的名字
spring.rabbitmq.virtual-host=my_vhost
spring.rabbitmq.username=admin
spring.rabbitmq.password=admin

#开启消息单条消费（方便测试）
spring.rabbitmq.listener.simple.prefetch=1

# 开启 publish-comfirm 机制和消息路由匹配失败退回机制（两回调函数）
spring.rabbitmq.publisher-returns=true
spring.rabbitmq.publisher-confirm-type=correlated
# 注释掉消费者手动 ack 机制，改为自动
# spring.rabbitmq.listener.simple.acknowledge-mode=manual
# 开启spring提供的retry
spring.rabbitmq.listener.simple.retry.enabled=true
spring.rabbitmq.listener.simple.retry.max-attempts=3
spring.rabbitmq.listener.simple.retry.initial-interval=3000

3、 RabbitConfig

Configure MQ meta information, mainly when the program starts, make the following settings:

• Create a dead letter queue and a dead letter exchanger, and bind the dead letter queue to the dead letter exchanger.

• Create an ordinary queue and an ordinary exchanger, bind the ordinary queue to the ordinary exchanger, and associate the dead letter queue with the ordinary queue, so that when message consumption fails, the message will enter the dead letter queue (automatic ack mode is used).

package com.fmi110.rabbitmq.config;

import lombok.extern.slf4j.Slf4j;
import org.springframework.amqp.core.*;
import org.springframework.amqp.rabbit.retry.MessageRecoverer;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.util.HashMap;


/**
 * @description rabbitMQ 配置类
 */
@Configuration
@Slf4j
public class RabbitConfig {
    
    

    String dlQueueName  = "my-queue-dl"; // 普通队列名称
    String dlExchangeName = "my-exchange-dl"; // 死信交换器名称
    String dlRoutingKey   = "rabbit.test";

    String queueName = "retry-queue";
    String exchangeName = "my-exchange"; // 普通交换器名称

    /**
     * 创建死信队列
     *
     * @return
     */
    @Bean
    public Queue queueDL() {
    
    

        return QueueBuilder
                .durable(dlQueueName) // 持久化队列
                .build();
    }

    /**
     * 创建死信交换机
     *
     * @return
     */
    @Bean
    public TopicExchange exchangeDL() {
    
    
        return new TopicExchange(dlExchangeName, true, false);
    }

    /**
     * 绑定操作
     */
    @Bean
    public Binding bindQueueDL2ExchangeDL(Queue queueDL, TopicExchange exchangeDL) {
    
    
        log.info(">>>> 队列与交换器绑定");
        return BindingBuilder.bind(queueDL).to(exchangeDL).with(dlRoutingKey);
    }

    /**
     * 创建持久化队列,同时绑定死信交换器
     *
     * @return
     */
    @Bean
    public Queue queue() {
    
    
        log.info(">>>> 创建队列 retry-queue");
        HashMap<String, Object> params = new HashMap<>();
        params.put("x-dead-letter-exchange", dlExchangeName);
        params.put("x-dead-letter-routing-key", dlRoutingKey);

        return QueueBuilder
                .durable(queueName) // 持久化队列
                .withArguments(params) // 关联死信交换器
                .build();
    }


    /**
     * 创建交换机
     *
     * @return
     */
    @Bean
    public TopicExchange exchange() {
    
    
        log.info(">>>> 创建交换器 my-exchange");
        boolean durable    = true; // 持久化
        boolean autoDelete = false; // 消费者全部解绑时不自动删除
        return new TopicExchange(exchangeName, durable, autoDelete);
    }

    /**
     * 绑定队列到交换机
     *
     * @param queue
     * @param exchange
     * @return
     */
    @Bean
    public Binding bindQueue2Exchange(Queue queue, TopicExchange exchange) {
    
    
        log.info(">>>> 队列与交换器绑定");
        return BindingBuilder.bind(queue).to(exchange).with("rabbit.test");
    }

//    /**
//     * spring-retry重试机制：当重试次数达到最大，消息仍然消费失败时回调。
//     * 如果开启这个类，则死信队列失效，消息消费失败，即使配置了死信队列，消息也不会进入死信队列。
//     * 重试失败回调和死信队列只能二选一！！！spring 提供回调实现类有如下几个：
//     * RejectAndDontRequeueRecoverer ：消费失败，并且消息不再入列，spring默认使用。
//     * ImmediateRequeueMessageRecoverer ：将消息重新入列
//     * RepublishMessageRecoverer：转发消息到指定的队列，
//     * @return
//     */
//    @Bean
//    public MessageRecoverer messageRecoverer(){
    
    
//        return new MessageRecoverer() {
    
    
//            @Override
//            public void recover(Message message, Throwable cause) {
    
    
//                log.info(message.toString());
//                log.info("spring-retry重试次数达到最大，消息仍然失败的回调");
//                // TODO: 记录错误信息并上报
//            }
//        };
//    }
}

4. Message producer RabbitProducer

In order to ensure reliable message delivery, the confirm message confirmation mechanism and callback function are configured here.

package com.fmi110.rabbitmq;
import com.rabbitmq.client.AMQP;
import lombok.extern.slf4j.Slf4j;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import javax.annotation.PostConstruct;

/**
 * @description 消息生产者
 */
@Component
@Slf4j
public class RabbitProducer {
    
    
    @Autowired
    RabbitTemplate rabbitTemplate;

    /**
     * 1 设置 confirm 回调，消息发送到 exchange 时回调
     * 2 设置 return callback ,当路由规则无法匹配到消息队列时，回调
     *
     * correlationData：消息发送时，传递的参数，里边只有一个id属性，标识消息用
     */
    @PostConstruct
    public void enableConfirmCallback(){
    
    
        // #1
        /**
         * 连接不上 exchange或exchange不存在时回调
         */
        rabbitTemplate.setConfirmCallback((correlationData,ack,cause)->{
    
    
            if (!ack) {
    
    
                log.error("消息发送失败");
                // TODO 记录日志，发送通知等逻辑
            }
        });

        // #2
        /**
         * 消息投递到队列失败时，才会回调该方法
         * message:发送的消息
         * exchange:消息发往的交换器的名称
         * routingKey:消息携带的路由关键字信息
         */
        rabbitTemplate.setReturnCallback((message,replyCode,replyText,exchange,routingKey) ->{
    
    
            log.error("消息路由失败");
            // TODO 路由失败后续处理逻辑
        });
    }

    public void send(String msg){
    
    
        String exchangeName = "my-exchange";
        // String routingKey   = "aaa.xxx";
        String routingKey   = "rabbit.test";
        rabbitTemplate.convertAndSend(exchangeName, routingKey, msg);
    }
 }

5. Message consumer RabbitConsumer

package com.fmi110.rabbitmq;

import com.rabbitmq.client.Channel;
import lombok.extern.slf4j.Slf4j;
import org.springframework.amqp.rabbit.annotation.RabbitListener;
import org.springframework.amqp.support.AmqpHeaders;
import org.springframework.messaging.handler.annotation.Header;
import org.springframework.stereotype.Component;
import java.util.concurrent.atomic.AtomicInteger;


/**
 * @description 消息消费者
 */
@Component
@Slf4j
public class RabbitConsumer {
    
    

    AtomicInteger count = new AtomicInteger();

    /**
     * 普通队列消费者
     * @param data
     * @param channel
     * @param tag
     * @throws Exception
     */
    @RabbitListener(queues="retry-queue")
    public void consumer(String data, Channel channel, @Header(AmqpHeaders.DELIVERY_TAG) long tag) throws Exception{
    
    

        log.info(">>>> consumer 消费 tag = {},次数count={},消息内容 : {}",tag, count.incrementAndGet(),data);
        // TODO 消息处理逻辑
        throw new RuntimeException("抛出异常，模拟消费失败，触发spring-retry");
    }

    /**
     * 死信队列消费者
     * @param data
     * @param channel
     * @param tag
     * @throws Exception
     */
    @RabbitListener(queues="my-queue-dl")
    public void consumeDL(String data, Channel channel, @Header(AmqpHeaders.DELIVERY_TAG) long tag) throws Exception{
    
    
        log.info(">>>> 死信队列消费 tag = {},消息内容 : {}",tag,data);
//        channel.basicNack(tag, false, false);
    }
}

6、Controller

Used to trigger sending messages

package com.fmi110.rabbitmq.controller;

import com.fmi110.rabbitmq.RabbitProducer;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.ResponseBody;
import org.springframework.web.bind.annotation.RestController;
import java.util.HashMap;

@RestController
public class TestController {
    
    
    @Autowired
    RabbitProducer rabbitProducer;

    @GetMapping("/test")
    public Object test() {
    
    

        rabbitProducer.send("this is a message");

        HashMap<String, Object> result = new HashMap<>();
        result.put("code", 0);
        result.put("msg", "success");
        return result;
    }
}

7. Operation results

The running log is as follows:

: >>>> consumer 消费 tag = 1,次数count=1,消息内容 : this is a message
: >>>> consumer 消费 tag = 1,次数count=2,消息内容 : this is a message
: >>>> consumer 消费 tag = 1,次数count=3,消息内容 : this is a message
o.s.a.r.r.RejectAndDontRequeueRecoverer  : Retries exhausted for message 
(Body:'this is a message' MessageProperties 
[headers={
    
    spring_listener_return_correlation=2840e95b-8544-4ed8-b3ed-8ba02aee2729}, 
contentType=text/plain, contentEncoding=UTF-8, contentLength=0, 
receivedDeliveryMode=PERSISTENT, priority=0, redelivered=false, 
receivedExchange=my-exchange, receivedRoutingKey=rabbit.test, 
deliveryTag=1, consumerTag=amq.ctag-a5AZEb9AYpOzL6mQJQIvaQ, 
consumerQueue=retry-queue])

...
Caused by: java.lang.RuntimeException: 抛出异常，模拟消费失败，触发spring-retry
    at com.fmi110.rabbitmq.RabbitConsumer.consumer(RabbitConsumer.java:36) ~[classes/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_181]
....

: >>>> 死信队列消费 tag = 1,消息内容 : this is a message

It can be seen from the log that the consumer of the ordinary queue failed to consume three times in total. Finally, the callback spring provided RejectAndDontRequeueRecoverer, and then the message entered the dead letter queue and was consumed.

Summary of frequently asked questions:

Message loss, message idempotence, message sequential consumption, message delayed consumption, repeated consumption, message backlog

• Message loss: In RabbitMq, two function callbacks will be performed when the message reaches the Broker and when the message fails to reach the queue.

  • On the producer side, as long as the message arrives at the Broker and no error is reported in another callback function, the message is persisted to disk.
  • On the consumer side, messages need to be received manually to prevent message loss, because messages are received automatically by default. There may be problems with the business at this time, but the message is still confirmed and it will be lost. However, if it is received manually, the message will be lost. When the business fails, a retry mechanism can be adopted, logging can be performed, the database can be scanned regularly, and the failed message can be sent again. (If you feel that adding logs to the callback function is troublesome, you can also set up a backup teaching switch to implement alarm queues, backup queues, etc.)
  • Transaction messages can also be sent, but this will be less efficient.

  Message idempotence: This must be considered based on the actual business,

  • You can use a unique identifier to add a unique identifier to the message, such as message ID, business serial number, etc. When consuming a message, first check whether the message has been consumed. If it has already been consumed, skip the message;
  • Token verification can also be implemented in conjunction with redis. For example, when submitting an order business, you can first create a token and set the expiration time in redis and return the token to the front end. When the front end submits the order, it passes the token to the back end. The token is verified. If the verification is successful, the submission business is executed and the redis token is deleted.

  Message sequential consumption:

  • Use partitioning: Partition messages according to business logic, and then send messages in the same partition to the same queue. In this way, consumers can consume messages in partition order.
  • Use sequential messages Some message queue systems support sequential messages, which can ensure that messages are consumed in the order they are sent.
  • Use distributed locksWhen consumers consume messages, they use distributed locks to ensure that messages in the same queue can only be consumed by one consumer. In this way, the sequential consumption of messages can be guaranteed.

  Message delayed consumption: Just use a delayed queue.

  Message duplication: It means that the consumer consumes the message and the message is interrupted before he can manually reply. Then after the server reconnects, the message changes from unAcked state to ready state and is handed over to the consumer. If the user consumes again, the solution is similar to idempotence. Create an anti-duplication table or a unique identifier, such as to lock the inventory. Only the locked inventory will be unlocked. Once unlocked, the status will not change. Unlocked again.

  Message backlog: There are too many messages sent and the consumer’s messaging capabilities are insufficient. First check the log code, and then limit the message flow on the producer side and bring more consumers online.