1. How to deal with the message of delivery failure
First of all, the delivery fails in the following two situations
- The exchange cannot be matched to the queue according to its own type and routing key
- When the exchange routed the message to the queue, it found that there were no consumers on the queue
solution:
When the producer delivers the message, specify the mandatory or immediate parameter as true, and RabbitMQ will return the undeliverable message to the producer through the Basic.Return command. At this time, the producer needs to call channel.addReturnListener to add the ReturnListener monitor to realize monitoring Delivery failed message
If you set the above two parameters, you need to add ReturnListener logic, which makes the logic of the producer more complicated. The backup switch in RabbitMQ can also handle this problem
This is achieved by adding the alternate-exchange parameter when declaring the exchange (calling the channel.exchangeDeclare method)
For backup switches, use includes several special cases:
- If the set backup switch does not exist , neither the client nor the RabbitMQ server will be abnormal, and the message will be lost at this time
- If the backup switch is not bound to any queue , neither the client nor the RabbitMQ server will be abnormal, and the message will be lost at this time
- If the backup switch does not have any matching queues , neither the client nor the RabbitMQ server will be abnormal, and the message will be lost at this time
If a backup switch is used with the mandatory parameter, the mandatory parameter is invalid
2. How to set the expiration time of the message
- Set the queue property, all messages in the queue have the same expiration time
- Set the message itself individually, the TTL of each message can be different
If the two methods are used together, the TTL of the message shall be the smaller value between the two
3. How to implement a delay queue
Answer: use a dead letter switch
Post the message to a queue without consumers, and specify the corresponding dead letter queue for this queue. When the message reaches the set expiration time and has not been consumed, it will be published to the dead letter queue, and the consumer subscription is dead The message queue directly consumes a delayed message
Delayed message plugin
Using delayed queues requires adding additional plugins for RabbitMQ
Delay plug-in rabbitmq_delayed_message_exchange download addresshttps://www.rabbitmq.com/community-plugins.html
Put the downloaded plugin into the following directory
/usr/lib/rabbitmq/lib/rabbitmq_server3.6.4/plugins
Start the plugin
rabbitmq-plugins enable rabbitmq_delayed_message_exchange
Declare the switch type as x-delayed-message to mark this switch as a delayed switch
When sending a message, add the x-delay parameter in the header to control the delay time of the message
4. How to specify the priority of the message
Set the max priority parameter of the queue, the priority of the message in RabbitMQ is 0 by default, and the maximum value is 10
5. How to realize the persistence of messages
The persistence of RabbitMQ is divided into: the persistence of the exchange, the persistence of the queue and the persistence of the message
The persistence of exchanges and queues is achieved by setting the durable parameter to true when declaring
The persistence of the message is achieved by specifying deliveryMode as 2 when sending the message
6. How to ensure that messages are not lost
The producer starts the transaction or the confirmation mechanism of the sender , all switches, queues and messages are set to be persistent, and the consumer starts the confirmation mechanism of the consumer
It can be seen from the figure that there are three possible situations in which messages may be lost:
1: The producer loses the message.
When the producer sends the data to MQ, the delivery of the message may fail due to reasons such as the network.
2: MQ itself loses messages.
The persistence of RabbitMQ is not enabled, the data is stored in memory, and the queue data is lost after the service hangs up; the
RabbitMQ persistence is enabled, and the message will be persisted to the disk after writing, but it hangs up when it is dropped. Yes, but the probability of this is very small
3: The consumer lost the message.
The consumer just received the message and has not processed it yet, so the consumer hangs up.
For the above three situations, each situation has a corresponding processing method:
1"The solution to the producer losing the message
Method 1: Open the transaction of RabbitMQ
Rabbitmq provides commands related to three transactions: select to open a transaction, commit to submit a transaction, and rollback to roll back a transaction.
Using this method due to the transaction mechanism will lead to a decrease in throughput and consume too much performance.
Method 2: Turn on the confirm mode
When using springboot, do the following configuration in the application.yml configuration file
spring:
rabbitmq:
addresses: 127.0.0.1
port: 5672
username: guest
password: guest
# 发送者开启 confirm 确认机制
publisher-confirm-type: correlated实现confirm回调接口
@Slf4j
public class ConfirmCallbackService implements RabbitTemplate.ConfirmCallback {
@Override
public void confirm(CorrelationData correlationData, boolean ack, String cause) {
if (!ack) {
log.error("消息发送异常!");
//可以进行重发等操作
} else {
log.info("发送者已经收到确认，correlationData={} ,ack={}, cause={}", correlationData, ack, cause);
}
}
}
生产者发送消息时设置confirm回调
@Slf4j
@Configuration
public class RabbitMqConfig {
@Bean
public ConfirmCallbackService confirmCallbackService() {
return new ConfirmCallbackService();
}
@Bean
public RabbitTemplate rabbitTemplate(@Autowired CachingConnectionFactory factory) {
RabbitTemplate rabbitTemplate = new RabbitTemplate(factory);
/**
* 消费者确认收到消息后，手动ack回执回调处理
*/
rabbitTemplate.setConfirmCallback(confirmCallbackService());
return rabbitTemplate;
}
//Other configuration code
...
Summary: The biggest difference between the transaction mechanism and the confirm mechanism is that the transaction mechanism is synchronous. After you submit a transaction, it will be blocked there, but the confirm mechanism is asynchronous. After you send a message, you can send the next message, and RabbitMQ receives it. Afterwards, the confirm interface will be called back asynchronously to notify you that the message has been received. Generally, to avoid data loss on the producer side, it is recommended to use the confirm mechanism.
2"The solution when MQ itself loses messages
Use persistent queues, and persist to disk processing when sending messages.
After setting the queue and message persistence at the same time, if RabbitMQ hangs and restarts again, it will also restart and restore the queue from the disk, and restore the data in the queue to ensure that the data will not be lost.
But even if the persistence mechanism is turned on, the service may hang up when the message is dropped. At this time, you can consider combining the producer's confirm mechanism to deal with it. After the persistence mechanism is enabled, the message will be notified through the confirm callback only when the message is successfully placed. Therefore, you can consider that the producer maintains a message waiting for message sending confirmation when producing messages. If the queue has not received the corresponding message feedback from the confirm after a certain period of time, it will automatically resend it.
3 "The solution when the consumer loses the message
Turn off the automatic ACK and use the manual ACK. There is an ACK mechanism in RabbitMQ. By default, when the consumer receives the message, RabbitMQ will automatically submit the ACK, and then the message will not be sent to the consumer. We can change to the manual ACK mode, and manually ack it after each message is processed. However, in this way, consumers may hang up without manual ack confirmation just after processing, resulting in repeated consumption of messages. However, we only need to ensure idempotence, and repeated consumption will not cause problems.
Modify the application.yml configuration file in springboot to change to manual ack mode
spring:
rabbitmq:
addresses: 127.0.0.1
port: 5672
username: guest
password: guest
# The sender opens the confirm confirmation mechanism
publisher-confirm-type: correlated
# The sender opens the return confirmation mechanism
publisher-returns: true
listener:
simple:
concurrency: 10
max-concurrency: 10
prefetch: 1
auto-startup: true
default-requeue-rejected: true
# Set the manual ack of the consumer side
acknowledge-mode: manual
# Whether to support retry
retry:
enabled: true
Consumer manual ack reference code:
@RabbitHandler
public void handlerMq(String msg, Channel channel, Message message) throws IOException {
try {
//业务处理代码
......
//手动ACK
channel.basicAck(message.getMessageProperties().getDeliveryTag(), false);
} catch (Exception e) {
if (message.getMessageProperties().getRedelivered()) {
log.error("消息已重复处理失败,拒绝再次接收...", e);
channel.basicReject(message.getMessageProperties().getDeliveryTag(), false); // 拒绝消息
} else {
log.error("消息即将再次返回队列处理...", e);
channel.basicNack(message.getMessageProperties().getDeliveryTag(), false, true);
}
}
}6. The problem of repeated consumption of messages
In case of repeated consumption of messages:
Reason
1. After the producer sends to the message queue, the message queue will respond to the producer. However, during this process, the message queue has a problem and does not receive the message, then the producer will repeatedly generate messages, and at this time a repeated message.
2. The producer sends a message to the message queue. The message queue is delayed due to the large number, and the producer waits for the response to time out. At this time, the producer will send a message to the message queue again.
3. The producer and message queue are caused by network problems, and the producer will initiate a retry. This also produces duplicate messages.
4. In fact, the main reason is that the message successfully entered the message queue, but due to various reasons, the message queue did not give the producer a successful return value, and the producer has a retry mechanism. In this case, duplicate messages will be generated.
Reason 2
: 1. The message queue is pushed to the consumer, and there is a consumption problem in the process of processing the message by the consumer. The message queue does not know the result of the consumer's processing, so it will deliver it next time.
2. After the consumer finishes processing, there is a problem with the network. At this time, no result is returned to the middleware message queue, and the message queue will deliver the message to the consumer next time.
3. The consumer processing timeout exceeds the timeout period of the message queue, and the message queue will be delivered again at this time.
4. The consumer returns the processing result to the message middleware, but there is a problem with the message middleware, and the processing result is lost. After restarting, the internal inspection of the message middleware finds that the message has not been processed and will be delivered to the consumer next time.
To solve this problem, idempotent processing is generally done on the consumer side.
7. How to ensure the idempotence of message queue consumption
This part should still be combined with the business to choose the appropriate method. There are the following solutions:
In order to simply write the consumption data into the database, you can first check whether the data already exists according to the primary key. If it already exists, there is no need to insert it. Or direct insertion is fine, because the uniqueness of the primary key can be used to ensure that data will not be inserted repeatedly. Repeated insertion will only report an error, but no dirty data will appear.
Consumption data is only to be cached in redis. In this case, the value is directly set in redis, which is naturally idempotent.
For complex business situations, you can add a globally unique ID to each message when producing messages. When consumers consume messages, they can use this ID to check whether they have been consumed before in redis. If it has not been consumed, it will be consumed and the ID of the message will be written into redis. If you have already consumed it, there is no need to consume it again.
8. Message guaranteed sequential consumption
When messages are put into the queue, there is an order. If only a single consumer processes the corresponding single queue, there will be no problem of message confusion. However, when consuming, it is possible for multiple consumers to consume the same queue. Because each consumer processes messages at different times, the messages cannot be processed in the expected order. In fact, the fundamental problem is how to ensure that the messages are processed in the expected order.
There is a situation where the order of consumption is out of order
In order to improve processing efficiency, there are multiple consumers in a queue
There is only one consumer in a queue, but in order to improve processing efficiency, multi-threading is used in the consumer for processing
The method to ensure the sequence of messages
is to split the original queue into multiple queues, and each queue has its own consumer. The core of this solution is that the producer sends the same type of data (data of the same order) that needs to be in order according to the key value of the business data (such as the order ID hash value modulo the number of order queues) when delivering the message to the same queue.
One queue is one consumer, and multiple memory queues are maintained in the consumer, and messages are added to different memory queues according to the key value of business data (such as the order ID hash value modulo the number of memory queues), and then multiple memory queues are actually responsible for processing The thread of the message goes to the corresponding memory queue to obtain the message for consumption.
RabbitMQ guarantees message sequence summary:
the core idea is to divide the business data into multiple message sets according to the key value, and the message data in each message set is in order, and each message set has its own independent consumer. The existence of multiple message sets ensures the efficiency of message consumption, and each ordered message set corresponds to a single consumer, which also ensures the orderliness of message consumption.