1. Introduction
1. The concept of MQ
The full name of MQ is Message Queue (message queue), which is a container for storing messages during message transmission. It is mostly used for communication between distributed systems. It is a "first in, first out" data structure.
2. MQ model
Producers send messages to MQ. MQ pushes messages to designated consumers, or consumers go to MQ to pull specific messages. Producers and consumers can also become clients, corresponding to MQ is the server
3. The difference between MQ and redis, traditional database
MQ message queue service: for data change scenarios, its function is weaker than the other two (sending data, fetching data), it is a temporary transfer place for data, mainly used for adding, deleting, or a small amount of query functions; processing speed is higher than The traditional database is fast (the operation is simpler and there are not so many constraints. The most important reason for high performance: MQ operations are mainly sequential read and write, and the database is a random disk read)
Redis cache technology: fast processing speed, using memory mechanism, mainly for query scenarios
Traditional database: transaction support, persistent storage, data management scenarios
4. Common MQ
Common MQ mainly include activemq, rabbitmq, kafka, rocketmq
Rocketmq is mainly used here, which is led by Ali. Official website documents: https://rocketmq.apache.org/zh/docs/
Two, the role of MQ
1. Decoupling
When system A produces key data and sends the data to multiple other systems for consumption, system A and other systems are seriously coupled. If the data generated by system A is placed in MQ, other systems go to MQ to obtain consumption data. When each system runs independently and only interacts with MQ, adding a new system to consume the data of system A does not need to modify the code of system A to achieve the effect of decoupling.
After using MQ, calls between different applications are decoupled to improve system fault tolerance. As shown in the figure, the order system directly sends the order message to MQ and it is over. Other systems obtain order information from MQ for processing
2. Asynchronous speed up
Asynchronous communication between different dependent services can improve the throughput of the entire system. As shown in the figure below, when MQ is not applicable, the total time for a single request is 20ms+3*300ms, after using MQ, 20+5+300ms
3. Shaving peaks and filling valleys
After using MQ, limit the speed of consuming messages to 1000. In this way, the data generated during the peak period will inevitably be backlogged in MQ, and the peak will be "cut" off, but because of the backlog of messages, a period of time after the peak period Within a certain period of time, the speed of consuming news will still be maintained at 1000 until the backlog of news is consumed, which is called "filling the valley".
3. Monitoring system
Download the jar package of rocketmq-console: https://download.csdn.net/download/qq_38571773/87388867
After downloading the jar package, start the command:
java -jar -Dserver.port=9981 -Drocketmq.namesrv.addr=ip:端口号 rocketmq-console-ng-1.0.1.jar # ip和端口号，为rocketmq部署时， 启动NameServer的服务器ip和端口号，默认端口是9876After startup, it can be accessed through the port number:
Mainly check the topic top, which is a unified mark of a certain type of storage, similar to a table in a database.
Under the consumer tab, the main focus is on TPS and delay. TPS is how many MQ messages are processed per second in the consumer program;
Delay is the accumulation of MQ messages, waiting for consumers to consume.
4. Precautions for MQ architecture testing
1. Delay caused by asynchrony
Different businesses have different concerns about response time. Therefore, it is necessary to consider whether to accept the delay time caused by asynchrony according to the specific business situation
For example, after ordering takeaway, give points for evaluation ---> 1), submit the evaluation first; 2), modify the user's points
Since the time requirement for increasing the integral is not so high --> program design: Design the related operations of the integral as asynchronous operations
2. Message loss
2.1 Message mode type:
sync synchronous mode: send messages in synchronous mode, this method returns the result only after the message is completely sent, this method has the time cost of waiting for the result to be sent synchronously, and has an internal retry mechanism
async synchronous mode: Sending messages adopts asynchronous sending mode, and returns immediately after the message is sent. When the message is completely sent, the callback function sendCallback will be called to inform the sender whether the sending is successful or failed. Asynchronous mode is usually used in response to time-sensitive business scenarios, that is, it cannot bear the time-consuming cost of waiting for the return when sending messages synchronously; asynchronous mode also implements a retry mechanism internally, and the default number of times is 2 --> the call fails for network problems
测试的时候要注意， 如果发送完毕之后，程序就挂掉了， 对于数据处理是否存在问题
异步发送消息代码之后，MQ不一定收到，而继续处理后面的代码 （理论上来说，上一个操作成功， 才能进行下一步操作）,因为性能的考虑， 进行异步的MQ消息发送，可能会出现: MQ发送失败，而后面的操作成功了
往往程序设计， 要在接受 MQ 确认没问题的消息之后，进行一系列具体数据的处理；收到 MQ 处理失败的callback回调消息之后，要进行相对应重试设计
one-way生产者单向发送:采用one-way发送模式发送消息的时候，发送端发送完消息后会立即返回，不会等待来自broker的ack来告知本次消息发送是否完全完成发送。这种方式吞吐量很大，但是存在消息丢失的风险，所以其适用于不重要的消息发送，比如日志收集
2.2 MQ服务器的刷盘类型
收到数据 ，最终要持久化到 磁盘，存在两种方式落盘：
SYNC_FLUSH（同步刷新）：MQ收到消息之后，写入磁盘
ASYNC_FLUSH（异步处理）：MQ收到信息，保存在内存，定时批量写入磁盘
3.幂等性
幂等性包含两次语义：1）一个消息最少被消费一次； 2）一个消息只会被成功消息一次
因为消息发送的，有重试机制，可能导致 同一条消息 发送多遍。
所以消费者收到消息之后，要检查是否重复处理，使用业务上的唯一ID，如支付ID、交易号、订单号、消息编码等来进行检查
4.通过MQ做系统解耦
如果通过MQ进行系统拆分，本质来讲就是分布式架构，需要监控多个系统而不是只看 接口后台
五、瓶颈分析
1.资源问题
1.1. MQ -- 高并发 数据 存取， 网络要求不低公司的消息内容很大
1.2. CPU、内存、磁盘不够--可考虑集群架构
2.Rocket MQ 本身资源
MQ是用java 语言开发的， 堆内存优化查看官方手册：https://rocketmq.apache.org/zh/docs/bestPractice/19JVMOS
3.消息积压
当 生产者 发送大量数据到MQ， 消费者处理能力跟不上，就会导致积压，可以通过检查消费者程序的 处理逻辑 或者 消费者程序所在的服务器 资源是否够用
六、调优建议
所有MQ的调优 都包含 使用的调优 以及 MQ本身的配置调优
主要调优手段来自官方文档：https://rocketmq.apache.org/zh/docs/bestPractice/15bestpractice