1. Reasons for message loss
1. When the sender sends the message, the message is lost. It seems that the message has been sent out
. 2. MQ has received the message, but the consumer has not received it when consuming
In p->b e->q and q->c three places to ensure reliable arrival; (picture from Shang Silicon Valley)
2. About the callback function and configuration on the production side
#开启发送端确认
spring.rabbitmq.publisher-confirm-type=correlated
#开启发送端消息抵达队列的确认
spring.rabbitmq.publisher-returns=true
#只要抵达队列 以异步方式优先回调returnconfirm
spring.rabbitmq.template.mandatory=true
@Slf4j
@Configuration
public class MyRabbitConfig {
@Autowired
RabbitTemplate rabbitTemplate;
@Bean
public MessageConverter messageConverter(){
return new Jackson2JsonMessageConverter();
}
/**
*定制RabbitTemplate
* 1.服务收到消息就回调
* 1）spring.rabbitmq.publisher-confirms=true
* 2)设置确认回调confirmCallback
* 2.消息正确抵达队列进行回调
* 1）spring.rabbitmq.publisher-returns=true
* 2) spring.rabbitmq.template.mandatory=true
* 设置确认回调ReturnCallback
* 3.消费端确认（保证每个消费被正确消费，此时才可以broker删除这个消息）
* 1）默认是自动确认的，只要消息接收到，客户端会自动确认，服务端就会移除这个消息
* 问题：
* 我们收到很多消息，自动回复服务器ack，只有一个消息处理成功，宕机了，就会发生消息丢失。
* 消费者手动确认模式，只要我们没有明确告诉MQ，货物被签收，没有ACK
* 消息就一直是unacked状态，即使Consumer宕机。消息不会丢失，会重新变成ready
* 2)如何签收：
* channel.basicAck(deliveryTag,false);签收获取
* channel.basicNack(deliveryTag,false,true);拒签
*
*/
@PostConstruct //再配置类对象创建完成以后，执行这个方法
public void initRabbitTemplate(){
RabbitTemplate.ConfirmCallback confirmCallback = new RabbitTemplate.ConfirmCallback() {
@Override
public void confirm(CorrelationData correlationData, boolean ack, String cause) {
if (ack){
System.out.println("发送成功");
}else {
System.out.println("发送失败");
}
}
};
rabbitTemplate.setConfirmCallback(confirmCallback);
rabbitTemplate.setReturnCallback(new RabbitTemplate.ReturnCallback() {
@Override
public void returnedMessage(Message message, int replyCode, String replyText, String exchange, String routingKey) {
System.out.println(message);
System.out.println(replyCode);
System.out.println(replyText);
}
});
}
}
(The above code is from Shang Silicon Valley)
3. Consumer side ack mode and chat related code
springboot-rabbit provides three message confirmation modes:
AcknowledgeMode.NONE: No confirmation mode (regardless of whether the program is abnormal or not, as long as the monitoring method is executed, the message will be consumed. It is equivalent to automatic confirmation in rabbitmq, and this method is not recommended)
AcknowledgeMode .AUTO: Automatic confirmation mode (by default, the consumer will automatically confirm if there is no exception, and will not confirm if there is an exception, infinite retry, resulting in an infinite loop of the program. Do not confuse it with the automatic confirmation in rabbit) AcknowledgeMode.MANUAL: Manual confirmation mode (
required Manually call channel.basicAck to confirm, you can catch exceptions to control the number of retries, and even control the handling of failure messages)
It is generally recommended to use the manual confirmation mode.
Examples are as follows:
springboot configuration
acknowledge-mode: none, auto, manual
none: automatic confirmation; auto: according to the situation; manual: manual confirmation
spring:
rabbitmq:
host: localhost
username: guest
password: guest
virtual-host: /
listener:
simple:
acknowledge-mode: manual // 开启手动确认
retry:
enabled: true // 开启重试
max-attempts: 3 //最大重试次数
initial-interval: 2000ms //重试间隔时间
@RestController
@RequestMapping("/mq")
//@Api(tags = "mq相关接口" , description = "MqController | 消息模块")
public class MqController {
private static final Logger logger = LoggerFactory.getLogger(MqController.class);
@Autowired
private ProdServer prodServer;
@Autowired
WebSocketServer webSocketServer;
@RabbitListener(queues = {
FanoutRabbitConfig.DEFAULT_BOOK_QUEUE})
public void listenerAutoAck(String text, Message message, Channel channel) {
// TODO 如果手动ACK,消息会被监听消费,但是消息在队列中依旧存在,如果 未配置 acknowledge-mode 默认是会在消费完毕后自动ACK掉
final long deliveryTag = message.getMessageProperties().getDeliveryTag();
try {
logger.info("[消费者一监听的消息] - [{}]", text);
ChatDto.MqChat chatDto = JSONObject.parseObject(text, ChatDto.MqChat.class);
Integer rs = WebSocketServer.sendInfo(chatDto.getMessage(), chatDto.getToUserId());
// yanUserChatService.saveChat(chatDto.getOpenid(),chatDto,rs);
// webSocketServer.sendMqMessage(text);
// TODO 通知 MQ 消息已被成功消费,可以ACK了
channel.basicAck(deliveryTag, false);
} catch (IOException e) {
try {
// TODO 处理失败,重新压入MQ
channel.basicRecover();
} catch (IOException e1) {
e1.printStackTrace();
}
}
}
}
@ApiOperation("聊天核心方法 ")
public void defaultMessage(@RequestBody ChatDto chatDto,
HttpServletRequest request) throws IOException{
String message = chatDto.getContent();
String toUserId = chatDto.getEmail();
String openid = Util.fromRequestToOpenid(request);
Map<String, String> map = new HashMap<String, String>();
map.put("message",message);
map.put("toUserId",toUserId);
map.put("openid",openid);
rabbitTemplate.convertAndSend("fanoutExchange", "", JSONObject.toJSONString(map));
}
Related code of websocket
public static Integer sendInfo(String message,@PathParam("userId") String userId) throws IOException {
log.info("发送消息到:"+userId+"，报文:"+message);
if(StringUtils.isNotBlank(userId)&&webSocketMap.containsKey(userId)){
//这个地方最好封装一下报文请求 要不然前端不好处理
JSONObject jsonObject =new JSONObject();
//追加发送人(防止串改)
jsonObject.put("type","receive");
jsonObject.put("content",message);
webSocketMap.get(userId).sendMessage(jsonObject.toJSONString());
log.info("发送消息到成功");
return 1;
}else{
log.error("用户"+userId+",不在线！");
return 0;
}
}
The application of rabbitmq under the chatting situation; it decouples the business logic after sending the message, and improves the performance during the chatting process.
4. How to ensure message reliability
1. Message lost
When the message is sent out, it does not reach the server due to network problems
. Make a fault-tolerant method (try-catch). The network failure may occur when sending a message. After the failure, there must be a retry mechanism , which can be recorded in the database, and it should be done
by regular scanning and resending.
Log records, whether the status of each message is received by the server should be recorded
. Do regular resend. If the message is not sent successfully, regularly go to the database to scan the unsuccessful message
for reaches the Broker, the Broker will write the message to Disk (persistence) is considered a success. At this time, Broker has not yet completed persistence and is down.
The publisher must also add a confirmation callback mechanism to confirm the successful message and modify the status of the database message.
In the state of automatic ACK. The consumer receives the message, but does not have time to receive the message and then
shuts down. Manual ACK must be enabled, and the consumer will be removed only if the consumption is successful. If it fails or has not had time to process it, it will noAck and re-enter the queue.
2. Repeated messages
The message consumption is successful, the transaction has been submitted, and when the ack occurs, the machine is down. As a result, there is no ack success, and the broker's message changes from unack to ready again, and sends it to other consumers.
The message consumption fails. Due to the retry mechanism, the message is automatically sent out for
successful consumption. When ack is down, the message changes from unack to ready , Broker resends
the consumer's business consumption interface should be designed to be idempotent. For example, if there is a status flag of a work order in the inventory
, use the anti-duplication table (redis/mysql), and each sent message has a unique identifier for the business. After processing, there is no need to process each
message of rabbitMQ. Each message has a redelivered field, which can be used to obtain whether it has been delivered. Reposted, not the first time
3. News backlog
Backlog of consumer downtime
Consumers’ consumption capacity is insufficient.
The sender sends too much traffic.
More consumers go online for normal consumption. Go online with a dedicated queue consumption service ,
fetch messages in batches first, record them in the database, and process them slowly offline——
——————————————
The fourth part of the original text link: https://blog.csdn.net/weixin_40566934/article/details/119643740
5. The use of rabbitmq dead letter queue and delay queue
————————————————
Original link: https://blog.csdn.net/fu_huo_1993/article/details/88350188