- Scan QR codes - to join the group assembled architecture
Students interested in technology can join the group (Note: Java)
MQ (message queue) is often used in software architecture, especially in distributed systems, it is also a very frequently used component.
The following explains in detail the usage scenarios, concepts, common problems and solutions of message queues.
One, message queue usage scenarios
1.1 Common usage scenarios
System decoupling
In a distributed environment, the interdependence between systems will eventually lead to chaos in the entire dependency relationship. Especially in the microservice environment, there will be interdependence, even circular dependencies, and the split and optimization of the later system Come a great burden. Then we can use MQ for processing. The upstream system delivers data to MQ, and the downstream system consumes MQ data. Delivery and consumption can be processed in a synchronous manner, because the performance of MQ receiving data is very high and will not affect the performance of the upstream system.
Asynchronous processing
If the synchronization method is adopted, the system performance (concurrency, throughput, response time) will have a bottleneck. How to solve this problem? Introduce a message queue to process unnecessary business logic asynchronously.
Asynchronous processing can also lead to the use of parallel processing . In our work, we developed a simple distributed task processing component based on messages. The component is simply divided into three parts: segmentation, loading, and execution.
Each stage is to act as a consumer, and then send messages as a producer after processing. Message consumption is stateless, and capacity can be expanded indefinitely on demand.
Traffic peaking
Due to the use of messages, our link becomes a process in which the producer sends the message, the message middleware stores the message, and finally the consumer pulls the message from the message middleware. The storage capacity of message middleware can effectively help consumers to buffer. Imagine that consumers can consume happily under normal traffic. When the instantaneous peak traffic comes, the consumer's spending power can't keep up, and it just happens to be blocked in the message middleware. After the peak, the consumer can quickly block it. The message is consumed.
Traffic cutting is also a common scenario in message queues, which is generally used widely in spike or group grab activities!
Data distribution
大部分开源的MQ中间件基本都支持一对多或者广播的模式，而且都可以根据规则选择分发的对象。这样上游的一份数据，众多下游系统中，可以根据规则选择是否接收这些数据，这样扩展性就很强了。
1.2 消息使用的先决条件
以上四种是MQ中间件最常见的场景，但是我们细想，MQ中间件的引入会带来什么问题呢？那就是实时性。所以MQ中间件使用的先决条件是：能容忍延迟，只要求最终一致性较为合适。
二、消息相关的概念
MQ特点
先进先出
不能先进先出，都不能说是队列了。消息队列的顺序在入队的时候就基本已经确定了，一般是不需人工干预的。而且，最重要的是，数据是只有一条数据在使用中。 这也是MQ在诸多场景被使用的原因。发布订阅
发布订阅是一种很高效的处理方式，如果不发生阻塞，基本可以当做是同步操作。这种处理方式能非常有效的提升服务器利用率，这样的应用场景非常广泛。持久化
持久化确保MQ的使用不只是一个部分场景的辅助工具，而是让MQ能像数据库一样存储核心的数据。分布式
在现在大流量、大数据的使用场景下，只支持单体应用的服务器软件基本是无法使用的，支持分布式的部署，才能被广泛使用。而且，MQ的定位就是一个高性能的中间件。
在JMS标准中，有两种消息模型P2P（Point toPoint）和Publish/Sub(Pub/Sub)。
P2P
点对点，一个发，一个消费。涉及到的角色 发布者（Publisher）、消费者（Consumer）、消息队列（Queue）
特点
一个消息只能被一个消费者消费，消费后会从队列里移除
发布者和消费者无关系，发布者发送消息的行为不会随消费者而改变
消费者消费完成消息，需要向队列Ack，消息队列发现消息消费成功即做消息移除
Pub/Sub
Publish and subscribe mode, one publish, multi-party subscription. The roles involved are publisher (Publisher), topic (Topic), and subscriber (Subscriber).
Features
Each message can have multiple consumers
For subscribers of a topic (Topic), a subscriber must be created before the publisher’s messages can be consumed
In order to consume messages, subscribers must keep running
3. Common problems and solutions
Message blocking
1. Message blocking is generally caused by a surge in traffic, which exceeds the consumption capacity of consumers;
2. Or the consumer has a logic problem, which leads to constant retry or long wait.
The first can be solved by expansion
The second type can only fix the problem urgently and release it online, which will cause a large backlog of messages during the blocking process. In this case, you can also consider temporary expansion
Repeat consumption
Repeated consumption generally occurs on the consumer side. For example, when the consumer is processed, there is a problem when preparing to ack. After the application is restarted, the message middleware thinks that the message has not been processed and pushed it to the consumer, or the consumer pulled it. Time to repeat.
The general approach is to be idempotent on the consumer side.
Message is lost
Message loss is generally divided into producer failure to send, message middleware loss, and consumption loss.
Producer loss: It may be thought that the network problem or the failure of the intermediate processing of the message is caused, and the message is missed.
Loss in the middle of the message: Generally, middleware can set a discarding strategy, and most MQ middleware products can ensure that data is not lost. This situation basically does not need to be considered.
Loss of consumption: Some message middleware supports automatic ack. When a consumer consumes a message, the message middleware will automatically ack regardless of whether the consumption is successful. At this time, it is generally appropriate to choose consumers to actively ack.
Message sequence
Message order is generally guaranteed by MQ middleware. Most MQ middleware can only achieve partial order. For example, Kafka can only guarantee the order of a single partition queue. Some will also achieve overall order, but the cost is relatively high. The company I currently serve now supports the overall order.
MQ components include activeMQ, rabbitMQ, rocketMQ, zeroMQ, Kafka; interested students can learn more about it.
Welcome everyone to leave a message to discuss~
