Message Queuing (MQ) is a method of communication (cross-process) between different applications. Applications communicate by writing and retrieving data (messages) in and out of the queue without linking them through a dedicated connection. Messaging refers to the communication between programs by sending data in messages, rather than by calling each other directly, which is typically used for techniques such as Remote Procedure Call (RPC). Queuing refers to applications communicating through queues. The use of queues removes the requirement that the receiving and sending applications execute simultaneously. This naturally achieves the goal of asynchrony. So what are the functional scenarios of MQ? The following are introduced one by one.
decoupling
The most direct use scenario of MQ is to decouple the two systems. For example, in our payment deduction business scenario, the user generates an order and sends it back immediately after sending MQ, and the settlement system consumes the MQ to deduct the amount of the user account. In this way, the order system only needs to focus on creating the order successfully, increasing the order volume as much as possible, and returning the order to the user immediately after the order is generated. The settlement system focuses on the deduction of the account amount to ensure that the account amount is ultimately consistent. This scenario will also involve retry idempotency issues, which will be introduced later.
shaving peaks and filling valleys
Taking the order system and settlement system as an example, if the order system calls the settlement system through the RPC framework, the amount of orders generated will be very large in the case of peak promotions, and because the speed of generating orders is also very fast, this will inevitably lead to The system pressure is caused to the settlement system, and the server utilization rate will be high. However, when the order volume is relatively small at the time point that is not the peak, the server utilization rate of the settlement system will be low. For the settlement system, the following peak and trough phenomenon will appear.
Then, if the order is stored in the MQ queue through MQ, and the consumer uses the pull method, and the pull speed is controlled by the consumer, the flow can be controlled to stabilize. In this way, for the settlement system, the purpose of cutting peaks and filling valleys has been achieved. Or play the goal of flow control. Next, we introduce the pull method.
The pull mode means that the user actively calls the pull method in the code, and there is no need to configure <mq:listener /> in the configuration file. The speed of the pull is controlled by the user, and one call to pull a message for consumption. Here, we should pay attention to consumption. If the consumption performance decreases, it will definitely cause a backlog, so the user enables multi-threading to control the degree of parallelism to improve the consumption speed.
Code sample:
| 1 2 3 4 5 |
|
method: pull(String topic,MessageListener listener)
topic: refers to the topic name of the consumption
listener: is a callback object. When the pull pulls the message, it will actively call listener.onMessage(),
The difference from the listening mode is that in the listening mode, the MQ client daemon thread continuously pulls messages for consumption. In the pull mode, the user controls the frequency of pulling, and messages will not be consumed if they are not actively called. But neither need to actively confirm the message. This method is more suitable for writing scenarios, as long as the final result is guaranteed, because reading needs to return immediately so as not to make users wait for a long time and affect the user experience.
eventual consistency
Consistency problems are divided into strong consistency, weak consistency, and eventual consistency. Most Internet businesses require eventual consistency. Taking the business scenario of the order system and the settlement system as an example, after the order system successfully creates an order, the result returned to the user is success, and it clearly tells the user that the corresponding amount will be deducted from the account. Then the settlement system needs to maintain the same state as the order system, that is, the same amount is actually deducted from the user account. The order system will involve two actions, one is to create a successful order, the other is to send a success notification to MQ, we can put these two actions into a local transaction, either success or failure. When sending MQ fails once, it can be compensated in combination with timed tasks, which can ensure that the result of generating the order can be placed in the storage of MQ. Similarly, the consumer end of the settlement system relies on the MQ retry mechanism to send messages until the consumer end finally confirms that the deduction business has been successfully processed. In this way, we add compensation through message landing, and the consumer considers the guarantee of repeated consumption from the perspective of business, that is, idempotent operations are performed, and MQ is used to achieve final consistency.
broadcast consumption
MQ has two message modes, one is point-to-point mode, and the other is publish/subscribe mode (the most commonly used mode). At the same time, the publish/subscribe mode can be divided into cluster consumption and broadcast consumption according to the consumption type. Most of the time we use cluster consumption.
Cluster consumption: MQ sends any message, and only one server in the cluster can randomly consume this message. As shown below:
Broadcast consumption: MQ sends each message, and each server in the cluster consumes it at least once. As shown below:
Example of broadcast consumption: message push system. First, a client establishes a persistent connection with a server in the message center application cluster and saves the connection session information in the memory of the current server. When the cluster consumes business messages, it does not know where the persistent connection established by the client is. on a server. At this time, through broadcast consumption, each server in the cluster can consume business messages. Before deciding to push the notification to the user, it will judge whether there is the client's connection session information in the current server memory. If so, it will be pushed, and then the client will pull the user's message entity through the http protocol. If the session information is not on the current server, it will be discarded. As shown below:
Notes for broadcast consumption:
1. The consumption progress is managed on the consumer side. For example, the offset folder is created in the main directory by default, and the offset file is stored in the offset directory. The probability of duplication is greater than cluster consumption.
2. MQ can ensure that each message is consumed by each consumer server at least once, but if the consumer fails to consume, it will not enter the retry, so the business side needs to pay attention to the consumption failure.
3. Since the broadcast consumption message will not be confirmed, the backlog displayed on the management terminal will remain unchanged, and the logarithm shall prevail.
I specially sorted out the above technologies. There are many technologies that can’t be explained clearly in a few words, so I just asked a friend to record some videos. The answers to many questions are actually very simple, but the thinking and logic behind them are not simple. If you know it, you also need to know why. If you want to learn Java engineering, high performance and distributed, explain the profound things in simple language. Friends of microservices, Spring, MyBatis, and Netty source code analysis can add my Java advanced group: 433540541. In the group, there are Ali Daniel live-broadcasting technology and Java large-scale Internet technology videos for free to share with you.
Consume analog broadcasts with a cluster
In the publish/subscribe mode, if it is consumed by the cluster, then a message can only be consumed by a random server in the cluster. If we need each server in the cluster to consume such as the above example of message push, we use broadcast consumption to achieve. However, broadcast consumption has some disadvantages, such as it does not support sequential messages, and the probability of repeated consumption progress in client maintenance is greater than that in cluster mode, and consumption progress cannot be maintained in broadcast mode, so the backlog number on the management side remains unchanged, we must The number of queues shall prevail, that is, queries that cannot support message accumulation. If we want to avoid these drawbacks, then we can use cluster consumption to simulate broadcasting. In cluster consumption, the consumption APPID on each of our servers is the same. If we want to achieve the effect of broadcasting, then the consumption APPID on each server remains Just be different.
retry pit
The retry function of MQ can ensure that the data results are finally processed, but at the same time, because of the retry, special attention needs to be paid to the idempotency problem during business processing. For example, in the payment deduction business, after the order system generates an order, it calls the settlement platform to deduct the user's account amount. The settlement platform should calculate according to the serial number. If the order system has a network abnormality when calling the settlement platform, the settlement platform has actually received the request and processed it. The order system side believes that an exception occurs and needs to be retried, and subsequent orders sent to the settlement platform will cause repeated deductions. Therefore, special attention should be paid to the serial number to ensure that the serial number sent each time during the retry process is consistent. The settlement platform will check the business according to the serial number. If it has been processed, it will be discarded to ensure idempotency.
Summarize
We introduced the common usage scenarios of MQ and the precautions for using each scenario. Especially in the retry function, retry is originally a method provided by MQ to keep data that can be finally confirmed, but if the business use does not pay attention to idempotency, it will bring inconsistencies in business data even like repeated deductions. This has more serious consequences. We also present an example of the use of broadcast consumption in the publish/subscribe model, as well as its disadvantages and the ability to use cluster consumption to simulate broadcast. In view of the fact that each of the above scenarios provides us with a good description, so that everyone can better play the powerful role of MQ in the process of using MQ in the future.