How to ensure the reliability, consistency, idempotence, and real-time nature of messages
For information on the development of instant messaging IM, please refer toTencent Cloud Instant Messaging IM. We can refer to its server and client APIs, which also provide us with some solutions and design ideas. However, how to ensure the real-time, reliability, idempotence, and consistency of messages depends on the specific development code, and it does not provide a solution. In this blog, let’s talk about how to ensure these characteristics of messages~
1. Real-time nature of messages
The real-time nature of messages is mainly for user experience.
A message is sent to the server, and the server quickly processes the message and sends the message result to the client, which can also be said to be distributed to the message recipient. How to make the message quickly received by the receiver is the real-time nature mentioned here.
A message is sent to the server, and the server needs to undergo the following processing:
- Verification message (for example, whether the other party has been blocked or something, this is optional depending on the specific business)
- Message storage
- Give the sender an ACK response, indicating that the verification has been passed and the message was sent successfully.
- Send the message to the receiver. If multi-terminal is supported, multi-terminal synchronization must be ensured.
- If verification fails, a failed ACK must be returned.
Next, let’s talk about how to ensure the real-time nature of the message. There is no absolute real-time, we can only try to optimize it.
There are three optimizations in total:
- Communication services and business processing services are generally separated. Communication services generally hand over messages to business services for processing. In order to improve the user experience, the verification message can be placed in the communication service layer for processing, and then respond immediately if it fails, and then go to the business layer to handle the remaining business if it succeeds.
Message verification is handled in the business layer because it is related to some classes in the business layer. How to deal with it? UseOpen Feignto handle remote calls. As follows:
public interface FeignMessageService {
@Headers({
"Content-Type: application/json","Accept: application/json"})
@RequestLine("POST /message/checkSend")
ResponseVO checkSendMessage(CheckSendMessageReq o);
}
// TODO 1. 调用校验消息发送方的接口.
ResponseVO responseVO = feignMessageService.checkSendMessage(req);
// 如果成功投递到 mq，交给业务服务去处理
// 去存储，去分发
if (responseVO.isOk()) {
MqMessageProducer.sendMessage(message, command);
}
// 失败则直接ack
- Use the thread pool to handle post-verification processing. Because these operations are serial and do not affect the original program, it is appropriate to hand it over to other background threads for processing. It is also an optimization. The core code is as follows:
private final ThreadPoolExecutor threadPoolExecutor;
{
AtomicInteger num = new AtomicInteger(0);
threadPoolExecutor = new ThreadPoolExecutor(
6, 8, 60, TimeUnit.SECONDS,
new LinkedBlockingQueue<>(),
new ThreadFactory() {
@Override
public Thread newThread(Runnable r) {
Thread thread = new Thread(r);
thread.setName("message-process-thread-" + num.getAndIncrement());
thread.setDaemon(true);
return thread;
}
}
);
}
// 使用线程池处理校验后的处理
threadPoolExecutor.execute(() -> {
messageStoreService.storeP2PMessage(messageContent);
// 1. 回 ack 给自己
ack(messageContent, ResponseVO.successResponse());
// 2. 发消息给同步在线端
syncToSender(messageContent, new ClientInfo(messageContent.getAppId(), messageContent.getClientType(), messageContent.getImei()));
// 3. 发消息给对方在线端
dispatchMessage(messageContent);
});
- Since the storage of messages is an IO operation, the process is very slow compared to the CPU processing program, so we should also find ways to optimize it. How to optimize it? You can use the message queue to put the messages to be processed into the queue, and then process and store them by consumers monitoring the queue. This process is asynchronous and can improve the speed of the entire message processing. I am using RabbitMQ here. You can take a look at the following code:
public void storeP2PMessage(MessageContent messageContent) {
// TODO 发送消息
rabbitTemplate.convertAndSend(Constants.RabbitConstants.StoreP2PMessage,
"",
JSONObject.toJSONString(dto));
}
// 监听消息，然后消费消息，存储消息
@Service
@Slf4j
public class StoreP2PMessageReceiver {
@Autowired
StoreMessageService storeMessageService;
@RabbitListener(
bindings = @QueueBinding(
value = @Queue(value = Constants.RabbitConstants.StoreP2PMessage, durable = "true"),
exchange = @Exchange(value = Constants.RabbitConstants.StoreP2PMessage, durable = "true")
), concurrency = "1"
)
public void storeP2PMessage(@Payload Message message,
@Headers Map<String, Object> headers,
Channel channel) throws IOException {
String msg = new String(message.getBody(), "utf-8");
log.info("CHAT MSG FORM QUEUE ::: {}", msg);
Long deliveryTag = (Long) headers.get(AmqpHeaders.DELIVERY_TAG);
try {
JSONObject jsonObject = JSON.parseObject(msg);
DoStoreP2PMessageDto dto = jsonObject.toJavaObject(DoStoreP2PMessageDto.class);
ImMessageBody messageBody = jsonObject.getObject("messageBodyDto", ImMessageBody.class);
dto.setMessageBody(messageBody);
storeMessageService.doStoreP2PMessage(dto);
channel.basicAck(deliveryTag, false);
} catch (Exception e) {
log.error("处理消息出现异常：{}", e.getMessage());
log.error("RMQ_CHAT_TRAN_ERROR", e);
log.error("NACK_MSG:{}", msg);
//第一个false 表示不批量拒绝，第二个false表示不重回队列
channel.basicNack(deliveryTag, false, false);
}
}
}
2. Reliability of the message
Everyone knows that TCP itself is reliable, but it can only guarantee the reliability of the transport layer from one application to another, but the reliability between application layers cannot be guaranteed (for example, the message reaches the transport layer, and then the application crashes , naturally the message is not processed at the application layer, and the message is naturally not stored, thus losing reliability). Message transmission through TCP protocol is shown in the figure below:
The essence of IM development is to process messages through the im service, so it is necessary to ensure the reliability between the application and the im service. As shown below:
How to ensure the reliability between applications? The guarantee of reliability is to let the sender know that the receiver has received the message, which means that the message has been successfully delivered. As shown in the figure below, the receiver needs to receive two acks to indicate that the message was successfully sent. One is to tell the sender that the message has arrived at the im server and the message has been stored, and the other is to tell the sender that the message has arrived at the im server and the message has been stored. The message is successfully delivered to the recipient.
There are two types of ack responses from the receiver. One is that the receiver sends the ack to the sender after receiving the message; the other is that the receiver is not online, and the server sends the sender ack to indicate that the message has been distributed.
- The receiver receives the message online and sends an ack
if (command == MessageCommand.MSG_RECIVE_ACK.getCommand()) {
// 接收方收到消息回 ack
MessageReciveAckContent messageContent = o.toJavaObject(MessageReciveAckContent.class);
messageSyncService.receiveMark(messageContent);
}
@Autowired
MessageProducer messageProducer;
public void receiveMark(MessageReciveAckContent messageReciveAckContent){
messageProducer.sendToUser(messageReciveAckContent.getToId(),
MessageCommand.MSG_RECIVE_ACK,
messageReciveAckContent,messageReciveAckContent.getAppId());
}
- The receiver is not online, and the server responds with ack
// 3. 发消息给对方在线端
List<ClientInfo> clientInfos = dispatchMessage(messageContent);
// 判断接收方是否有在线的
if (clientInfos.isEmpty()) {
revicerAck(messageContent);
}
3. Orderliness of messages
Because when ensuring the real-time nature of messages, a thread pool is used to handle the storage and distribution of messages, this may lead to multiple messages being sent out of order. For example, if two messages are sent, the one sent first comes last. Later, this is due to multi-threading processing of message distribution, so messages may be sent out of order.
The orderliness of messages here is solved by using Redis atomic increment. Every time a message arrives at the server, a message sequence number generated by Redis atomic increment is generated for it, and then handed over to the front end, allowing the front end to sort the messages according to the sequence number. .
// 获取消息序列号 Seq，准备返回给前端，让前端处理消息的有序性
long seq = redisSeq.doGetSeq(
messageContent.getAppId() + ":"
+ Constants.SeqConstants.Message + ":" + ConversationIdGenerate.generateP2PId(
messageContent.getFromId(), messageContent.getToId()
)
);
messageContent.setMessageSequence(seq);
@Service
public class RedisSeq {
@Autowired
StringRedisTemplate stringRedisTemplate;
public long doGetSeq(String key){
return stringRedisTemplate.opsForValue().increment(key);
}
}
4. Idempotence of messages
In order to ensure the reliability of the message, the front end will resend the message if it does not receive two acks within a period of time, which may lead to the idempotence of the message. For example, a message is sent and stored but times out or is not distributed successfully for some reason. Then the front-end sends the same message again (that is, the messageId is the same), which results in it being stored twice and possibly distributed twice, so the message No longer idempotent.
Solution: Because the message ID is unique, we can deduplicate it based on the message ID, that is, use Redis cache. When a message is stored, it is also cached in Redis based on the message ID. Then when sending the same message for processing, first determine whether it is in the cache. If there is this message, just distribute the operation directly.
public void process(MessageContent messageContent) {
MessageContent cache = messageStoreService.getMessageFromMessageIdCache(messageContent.getAppId(),
messageContent.getMessageId(),
MessageContent.class);
// 如果消息已经被存储过了，那直接进行分发即可
// 即不需要再次进行存储
if (!ObjectUtils.isEmpty(cache)) {
threadPoolExecutor.execute(() -> {
// 1. 回 ack 给自己
ack(messageContent, ResponseVO.successResponse());
// 2. 发消息给同步在线端
syncToSender(cache, new ClientInfo(cache.getAppId(), cache.getClientType(), cache.getImei()));
// 3. 发消息给对方在线端
List<ClientInfo> clientInfos = dispatchMessage(cache);
if (clientInfos.isEmpty()) {
revicerAck(cache);
}
});
return;
}
// 获取消息序列号 Seq，准备返回给前端，让前端处理消息的有序性
long seq = redisSeq.doGetSeq(
messageContent.getAppId() + ":"
+ Constants.SeqConstants.Message + ":" + ConversationIdGenerate.generateP2PId(
messageContent.getFromId(), messageContent.getToId()
)
);
messageContent.setMessageSequence(seq);
// 前置校验
// 这个用户是否被禁言、是否被禁用
// 发送方和接收方是否是好友
// ResponseVO responseVO = imServerPermissionCheck(fromId, toId, messageContent.getAppId());
// if (responseVO.isOk()) {
// 使用线程池处理校验后的处理
threadPoolExecutor.execute(() -> {
messageStoreService.storeP2PMessage(messageContent);
// 1. 回 ack 给自己
ack(messageContent, ResponseVO.successResponse());
// 2. 发消息给同步在线端
syncToSender(messageContent, new ClientInfo(messageContent.getAppId(), messageContent.getClientType(), messageContent.getImei()));
// 3. 发消息给对方在线端
List<ClientInfo> clientInfos = dispatchMessage(messageContent);
messageStoreService.setMessageFromMessageIdCache(messageContent.getAppId(),
messageContent.getMessageId(),
messageContent);
if (clientInfos.isEmpty()) {
revicerAck(messageContent);
}
});
}
public void setMessageFromMessageIdCache(Integer appId, String messageId,
Object messageContent) {
String key = appId + ":" + Constants.RedisConstants.cacheMessage + ":"
+ messageId;
stringRedisTemplate.opsForValue().set(key, JSONObject.toJSONString(messageContent),
300, TimeUnit.SECONDS);
}
public <T> T getMessageFromMessageIdCache(Integer appId, String messageId,
Class<T> clazz) {
String key = appId + ":" + Constants.RedisConstants.cacheMessage + ":"
+ messageId;
String obj = stringRedisTemplate.opsForValue().get(key);
return JSONObject.parseObject(obj, clazz);
}
5. Summary
