RabbitMQ implements a detailed design and test plan for 100% message delivery

aims

• Realize a message queue that guarantees 100% delivery or consumption of messages under relatively harsh conditions.

Challenges and simple solutions

• 1. You say 100% delivery, if your mq hangs, how to deliver?
  使用镜像队列
• 2. What if the network is unstable and the message is not sent to the exchange or queue?
  conform模式 和 return模式 配合消息重发的线程池重发消息
  • 2.1. The message is sent to a certain mq, but this mq suddenly loses power and it is too late to synchronize to other mirror queues.
    消息持久化至硬盘，消息重启恢复
  • 2.2. What should I do if a message has just arrived in the memory and has not been solidified to the hard disk in time, and a fault occurs at this time, causing the message to be lost?
    这次消息是真的丢失了，但是有补偿机制
• 3. What if the message execution (consumption) fails, or the message is not sent to the queue after consumption?
  （1）发送消息时，除了发给消费端，还发给了补偿端做消息登记。 （2）消费端消费后，会把消息发送到补偿端进行同步。 （3）补偿机制会在间隔时间内重新发消息到补偿端，获取消息状态，要么重发，要么关闭。
• 4. What if the production end sends too many messages or the compensation mechanism sends too many messages, how to ensure idempotent consumption?
  redis分布式锁：Redisson实现消息幂等
• 5. What should I do if it is too late to consume when large traffic enters mq?
  mq有削峰填谷的算法
• 6. Compared with a single node, the above 100% delivery guarantee has much impact on performance?
  测试方案: （1）镜像队列在不同宕机条件下的消息可达性测试 （2）单节点和百分百投递队列的性能对比测试。
• 7. The message queue class is a singleton managed by spring, how to ensure that it will not cause thread safety problems when it is referenced in multiple places
  将scope改成request

Implementation architecture diagram

Image source: RabbitMQ guarantees 100% message delivery success plan

Implementation steps

• After the message is delivered to the client, the message will be dropped into the database (database a) as a credential for future inspection (message compensation)
• Messages are delivered to two mq queues at the same time, one is used for buffering information, and subsequent consumption (peak shaving and valley filling under heavy traffic), and the other is used for compensation mechanism to obtain messages, query and retransmit messages, and drop off to ( Database b)
• The message receipt after consumption by the server is sent to the queue, and the compensation mechanism updates the data (database b)
• The delay thread on the consumer side queries the message just now, and the compensation mechanism determines whether to resend the message or wait for the message according to the number of queries.

Test architecture diagram

Test case design

• Write a client program, use the encapsulated 100% delivery jar package to transmit messages, and expose an interface to the outside.
• Use jmeter to generate pressure and load through Nginx to the interfaces of two client programs to test the completion time and delivery success rate of 1,000 data, 10,000 data, and 100,000 data respectively.
• The consumer uses an interface to receive data, and uses thread sleep for 5 milliseconds to simulate thread processing data.
• After the consumer is not processing the data, check the message drop status and message consumption status of the database
• Consumption time formula :
  • Consumption time = the time when the last piece of data on the consumer side was printed-the time to start pressure,
  • Here you can test the delivery efficiency of mirror queue, single node, message solidification and non-solidification.
• Message delivery guarantee rate :
  • Query database storage data/pressure data,
  • Query the database to complete the consumption data / pressure data,
  • The minimum of the two is the message delivery guarantee rate.

To be continued