Series Article Directory
Teach you
how to build local RabbitMQ service ( windows ) How to choose RabbitMQ consumption mode
What is a dead letter, and how to use the dead letter mechanism of RabbitMQ?
foreword
我们在上次讨论RabbitMQ的消息可靠性时，已经提到了死信队列（详见系列文章《RabbitMQ 能保证消息可靠性吗》），死信概念是RabbitMQ的重要特性，官网也有该特性的介绍，那么这种设计有什么用，我们又该怎么使用死信呢？一起开始本次的学习吧
1. Dead letter and AMQP
A dead letter refers to an email or message that cannot be normally delivered to the target address due to some reasons, and under the semantics of MQ, it is an MQ message that cannot be consumed.
From the original AMQP specification ( AMQP0-9 version protocol document ), we can also see the instructions related to dead letters:
The server SHOULD track the number of times a message has been delivered to clients and when a message is redelivered a certain number of times eg 5 times without being acknowledged, the server SHOULD consider the message to be unprocessable (possibly causing client applications to abort
) , and move the message to a dead letter queue.
The server SHOULD track the number of times a message has been delivered to clients and when a message is redelivered a certain number of times eg 5 times without being acknowledged, the server SHOULD consider the message to be unprocessable (possibly causing client
applications to abort), and move the message to a dead letter queue.
The server should keep track of how many times the message has been delivered to the client, when the message has been re-delivered a certain number of times (say 5) without being acknowledged , the server should consider the message unprocessable (possibly causing the client's application to abort), and move the message to the dead letter queue.
Therefore, it is not difficult to see that RabbitMQ has a dead letter design, which mainly complies with the AMQP specification.Its purpose is to avoid the infinite loop of some messages that cannot be processed temporarily, and at the same time ensure that these MQ messages will not be lost because they cannot be processed temporarily, which can help developers better control the message processing flow and improve the reliability of the system sex and stability.
2. The scene of dead letter
1. Consumption failed
That is, the consumer cannot process the message, or the processing fails, and finally returns a negative Ack to the rabbitMQ server, and asks not to re-enter the queue. Generally, there are two methods
// reject a single message
void basicReject(long deliveryTag, boolean requeue);
// reject one or more messages
void basicNack(long deliveryTag, boolean multiple, boolean requeue);
Let's take a piece of basicReject code as an example:
// 新建消费者
Consumer consumer = new DefaultConsumer(channel) {
@Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
String message = new String(body, "UTF-8");
System.out.println("Received message: " + message);
// 模拟处理消息失败
boolean messageProcessedSuccessfully = false;
if (!messageProcessedSuccessfully) {
System.out.println("Message processing failed, rejecting message...");
// 注意第二个参数设置为为false
channel.basicReject(envelope.getDeliveryTag(), false);
System.out.println("Message rejected");
}
}
};
// 推模式，消费者监听队列
channel.basicConsume(QUEUE_NAME, true, consumer);
It should be noted that if the Ack is not sent to the RabbitMQ server due to network reasons, the message will not be set as a dead letter. Similarly, if the parameter of Ack is requeue = true , the message will not be set as a dead letter, but will be resent to the tail of the queue (the tail is not necessarily the last)
2. timeout
Timeout (Expiration) must be because there is a time limit (Time-To-Live) , and in rabbitMQ, we can set its TTL for queues and messages. It should be noted that setting the TTL for the queue does not mean the effective duration of the queue itself, but the effective duration of the message that is distributed into the queue. When the message enters the queue longer than this duration, the message will time out .
Message TTL
The effective duration setting of the message is AMQP.BasicPropertiesset by
AMQP.BasicProperties properties = new AMQP.BasicProperties().builder()
.deliveryMode(2) // 消息持久化
.expiration("60000") // 有效时长60秒
.build();
channel.basicPublish("", queueName, properties, message.getBytes("UTF-8"));
As in the above code, if the message is not consumed within 60 seconds, it will be automatically removed from the queue
Queue TTL
To set the duration for the queue, you need to add parameters when declaring the queuex-message-ttl
Map<String, Object> args = new HashMap<>();
args.put("x-message-ttl", 60000);
channel.queueDeclare(queueName, false, false, false, args);
Similarly, if the messages in the queue are not consumed within 60 seconds, they will be automatically removed from the queue.When the queue TTL and message TTL are set at the same time, take the smaller value as the message validity period
3. Queue saturation
Same as setting the effective time, we can also set a message upper limit (the number of messages or the size of the data) for the queue. The parameters used are x-max-lengthandx-max-length-bytes
Map<String, Object> args = new HashMap<String, Object>();
args.put("x-max-length", 10); // 最大10条
args.put("x-max-length-bytes", 1000); // 最大1000长度的 byte数组
channel.queueDeclare("myqueue", false, false, false, args);
If the limit is exceeded, rabbitMQ will remove the message from the head of the queue by default (usually the oldest one in the queue). Of course, there are other options for this overflow handling strategy, such as setting the parameter x-overflowto drop-head (默认), reject-publishorreject-publish-dlx
channel.queueDeclare(QUEUE_NAME, true, false, false,
new java.util.HashMap<String, Object>() {
{
put("x-overflow", "reject-publish");
}});
The meanings of the three strategies are as follows:
- drop-head : drop the head of the queue
- reject-publish : Reject new messages from entering the queue, if the message has "publish confirmation" enabled, send nack to the message publisher
- reject-publish-dlx : Same as reject-publish , but if a dead-letter switch is specified, the message will be forwarded to the dead-letter switch
3. Handling of dead letters
1. DLX dead letter switch
For the processing of dead letters, you can choose to discard and dead letter switches (when a dead letter switch is configured) , we are naturally discussing the latter, here is a flow chart of a dead letter switch,
and it is not difficult to find that the dead letter switch The phenomenon of letters always occurs in the queue, so we can set a "dead letter switch" for the queue . When a dead letter occurs, the queue can forward the dead letter to the dead letter switch. Its code is as follows
channel.exchangeDeclare("some.exchange.name", "direct");
Map<String, Object> args = new HashMap<String, Object>();
args.put("x-dead-letter-exchange", "some.exchange.name");
// 为队列 myqueue 设置一个死信交换机，该死信交换机的名字为some.exchange.name
channel.queueDeclare("myqueue", false, false, false, args);
It must be noted that,The function of the so-called "dead letter switch" is actually no different from that of a normal switch. It is only called a "dead letter switch" because of its special usage.。
When a dead letter exchange is set for a queue, we can see the DLX flag of the queue on the management panel.
2. Dead letter queue
The queue bound to the dead letter switch is called the dead letter queue , because the switch itself does not store messages, so the dead letter is finally stored in the dead letter queue. Of course, the dead letter queue itself has nothing to do with the normal queue. the difference.
3. Some detailed logic
(1) Dead letter routing problem
The dead letter is forwarded from the queue to the dead letter exchange, also with a routing key, if we do not set it specially, then the routing key is the routing key of the message itself. If we do the following settings, then all the messages sent by the queue to the dead letter exchange will have the routing key changed to messageDead.dl
Map<String, Object> args = new HashMap<String, Object>();
// 设置一个死信交换机
args.put("x-dead-letter-exchange", dlxName);
// 设置死信路由键，此处将所有死信的路由键设置为"messageDead.dl"
args.put("x-dead-letter-routing-key", "messageDead.dl");
channel.queueDeclare(QUEUE_NAME, false, false, false, args);
(2) The circulation problem of dead letters
The general message process is distributed to the queue by the switch, but the dead letter is sent from the queue to the switch. Therefore, it is not difficult to imagine whether it is possible to build a ring structure, so that a dead letter can go through the dead letter switch and finally return to the same queue? We don't hold back, the ring can be built, but the message will not be repeated , and the official document is directly excerpted as follows:
It is possible to form a cycle of message dead-lettering. For instance, this can happen when a queue dead-letters messages to the default exchange without specifying a dead-letter routing key. Messages in such cycles (ie messages that reach the same queue twice) will be dropped if there was no rejections in the entire cycle.
It is possible to form a cycle of message dead letters. This can happen, for example, when dead-letter messages are sent to the default exchange without specifying a dead-letter routing key. Messages in such cycles (that is, messages that arrive on the same queue twice) are discarded if there are no rejections throughout the cycle.
4. Dead letter function demo
After learning the above content, let's actually run a demo to test whether the dead letter function is as described above.
In the following code, we set up a dead-letter switch for a queue with a length of 20, and then send 30 MQ messages with the routing key "messageAlive" to the queue, and the dead-letter queue bound to the dead-letter switch is listening Route for "messageDead.#"
public class AsyncPublisher {
private final static String QUEUE_NAME = "message_queue";
private static final int MESSAGE_COUNT = 30;
private static ConcurrentNavigableMap<Long, String> outstandingConfirms = new ConcurrentSkipListMap<>();
public static void main(String[] argv) throws Exception {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
String message = "Hello, RabbitMQ!";
// 声明个容量20条的，带死信交换机的队列
channel.exchangeDeclare("myExchange", "topic");
declareQueueWithDLX(channel);
channel.queueBind(QUEUE_NAME,"myExchange","messageAlive");
// 异步发布确认
channel.confirmSelect();
channel.addConfirmListener(new ConfirmCallback() {
@Override
public void handle(long deliveryTag, boolean multiple) throws IOException {
System.err.println("Sucess to publish message.");
if (multiple) {
ConcurrentNavigableMap<Long, String> confirmed = outstandingConfirms.headMap(deliveryTag, true);
confirmed.clear();
} else {
outstandingConfirms.remove(deliveryTag);
}
}
}, new ConfirmCallback() {
@Override
public void handle(long deliveryTag, boolean multiple) throws IOException {
System.err.println("Failed to publish message.");
}
});
for (int i = 0; i < MESSAGE_COUNT; i++) {
long nextSeqNo = channel.getNextPublishSeqNo();
channel.basicPublish("myExchange", "messageAlive", null, message.getBytes());
outstandingConfirms.put(nextSeqNo, message);
}
System.out.println("All messages published successfully.");
channel.close();
connection.close();
}
static void declareQueueWithDLX(Channel channel) throws IOException {
String dlxName = "some.exchange.name";
String dlqName = "some.exchange.queue";
// 声明个交换机，作为死信交换机，类型为topic
channel.exchangeDeclare(dlxName, "topic");
// 声明个死信队列
channel.queueDeclare(dlqName, false, false, false, null);
// 将死信队列与死信交换机绑定，此处设定路由
channel.queueBind(dlqName, dlxName, "messageDead.#");
Map<String, Object> args = new HashMap<String, Object>();
// 设置一个死信交换机
args.put("x-dead-letter-exchange", dlxName);
// 设置队列最大消息量为 20
args.put("x-max-length", 20);
//args.put("x-dead-letter-routing-key", "messageDead.dl");
channel.queueDeclare(QUEUE_NAME, false, false, false, args);
}
}
Predict the final result: the normal queue is full of 20 messages, and the dead letter queue is set with routing and focuses on "messageDead.#". When the dead letter enters the dead letter switch with the original routing key messageAlive, it cannot be Distribute to any queue, so a message in the dead letter queue will not be distributed, let's see the result:
The result is as expected, if we put the above declareQueueWithDLXmethod in
args.put("x-dead-letter-routing-key", "messageDead.dl");
Uncomment, that is, let the dead letter routing key take effect, then all messages sent to the dead letter exchange will use the routing key " messageDead.dl " instead of the routing key of the original message. We delete the queue and run it again, predicting that there will be ten dead letters in the dead letter queue.
The result is as expected, and DLKthe flag of the queue isx-dead-letter-routing-key
5. Application of Dead Letter
Let's go back to the beginning and introduce the dead letter part. We mentioned the purpose of the dead letter mechanism:It is aimed at some messages that cannot be processed temporarily, while avoiding its infinite loop, it can also ensure that these MQ messages will not be lost because they cannot be processed temporarily. It can help developers better control the message processing flow, improve system reliability and stability.
So in practice, how do we use the dead letter queue to accomplish the above goals?
1. Message accumulation alarm
We can use a fixed-length queue. If the consumption capacity of the consumer is less than the production capacity of the producer for a long time, a large number of messages will be accumulated in the MQ queue. At this time, the fixed-length queue can enter the dead letter switch when the message overflows. ->Dead letter queue, as long as we create a consumer for the dead letter queue, we can get the message accumulation situation in time and send out an alarm
2. Exception message check
When some messages are rejected by the consumer because they cannot be processed temporarily, you can set dead letter forwarding for the queue at this time, which can prevent the message from being repeatedly enqueued and obtained in a loop, and also allow the abnormal message to be backed up into the dead letter queue in, so as not to be lost. These messages can be retrieved and reprocessed later, or the cause of the exception can be analyzed
3. Delay consumption
You can set TTL for the queue, and cooperate with the dead letter mechanism to achieve the effect of delayed queue. For some messages that need to be delayed, you can send them to the TTL queue, wait for a certain period of time, and then deliver them to the dead letter queue , and then consumed by the dead letter queue consumer.