The solution to duplicate messages The
root cause of duplicate messages is that the network is unreachable. As long as data is exchanged over the network, this problem cannot be avoided. So the solution to this problem is to bypass this problem. The question then becomes: If the consumer receives two identical messages, what should be done?
- The business logic for processing messages on the consumer side remains idempotent
- Ensure that each message has a unique number and that the message processing is successful and the log of the deduplication table appears at the same time
Article 1 is easy to understand. As long as the idempotence is maintained, no matter how many repeated messages come, the final processing result will be the same. For example, if the database Where status=1, it will become 0 after the update. After one update is successful, the others will not be executed.
The second principle is to use a log table to record the ID of the successfully processed message. If the ID of the newly arrived message is already in the log table, then the message is no longer processed. For example: store the key of the message and the processing status in redis. Every time data comes, go to redis to check whether the message has been processed.
The first solution is obviously to be implemented on the consumer side and is not a function to be implemented by the messaging system.
Clause 2 can be implemented in the message system or on the business side. Under normal circumstances, the probability of duplicate messages is actually very small. If it is implemented by the message system, it will definitely affect the throughput and high availability of the message system. Therefore, it is best to let the business end handle the problem of message duplication by itself, which is also RocketMQ does not solve the cause of the message duplication problem.
RocketMQ does not guarantee the non-repetition of messages. If your business needs to ensure strict non-repetition of messages, you need to de-duplicate them on the business side.