Message queue MQ (Message Queue)
Two commonly used message queue components:
RabbitMQ: High-availability and high-reliability messaging application scenarios, such as accounting failure retry, notification service, and messages are not allowed to be lost
Kafka: High-performance messaging application scenarios, such as logging, monitoring, and messages are allowed to be lost.
1. The basic concept of message queue
MQ has the characteristics of First Input First Output (FIFO), which can put the data (message) to be transmitted in the queue, and use the queue mechanism to realize message transmission-the producer generates the message and puts the message into the queue, It is then handled by the consumer. Consumers can go to the designated queue to pull messages, or subscribe to the corresponding queues, and the MQ server will push messages to them.
1.1. Broker
The concept of Broker comes from Apache ActiveMQ, which is the MQ server in layman's terms.
1.2. Producers and consumers of messages
Message Producer: Send messages to the message queue.
Message Consumer: Receive messages from the message queue.
2. Point-to-point mode
It is one-to-one, consumers take the initiative to pull data, and the message is cleared after the message is received
The point-to-point message queue model is usually a message delivery model based on pull or polling. This model requests information from the queue instead of pushing the message to the client. The characteristic of this model is that the message sent to the queue is received and processed by one and only one receiver, even if there are multiple message listeners.
3. Publish/Subscribe Model (Topic)
One-to-many, after data production, push to all subscribers
The publish-subscribe message model is a push-based message delivery model. The publish-subscribe model can have many different subscribers. Temporary subscribers only receive messages when they actively monitor the topic, while durable subscribers monitor all messages of the topic, even if the current subscriber is unavailable and in an offline state.
In the publish-subscribe message model, it is supported to publish messages to a specific topic Topic, and zero or more subscribers receive messages from this message topic. In this model, publishers and subscribers do not know each other. In the actual operation process, you must first subscribe, then send the message, and then receive the subscribed message. This order must be guaranteed.
4. The main role of the message queue
The most important role:
- asynchronous
- Decoupling
- Peak clipping
4.1. Decoupling
Allows you to independently extend or modify the processing on both sides, as long as you ensure that they comply with the same interface constraints.
4.2. Redundancy
Message Queue data for persistence until they have been fully processed, is persistent messages , through this way to avoid the risk of losing data. Many message queue used by the "Insert - get - to delete" paradigm, before the message removed from the queue, you need treatment system clearly indicates that the message has been processed, that is, ACK confirmation message- to ensure that Your data is stored safely until you finish using it.
Informed endurance
It is very important for some key core businesses. After enabling message persistence, after the message queue is down and restarted, the message can be restored from the persistent storage, the message is not lost, and consumption processing can continue.
Message ACK confirmation mechanism
In order to ensure that the message is not lost, the message queue provides the message Acknowledge mechanism, that is, the ACK mechanism. When the Consumer confirms that the message has been consumed and processed, and sends an ACK to the message queue, the message queue can delete the message. If the Consumer is down/shutdown and no ACK is sent, the message queue will think that the message has not been processed and will resend the message to other Consumers for consumption processing. .
4.3. Extensibility
Because the message queue decouples your processing process, it is easy to increase the frequency of message enqueuing and processing, as long as you increase the processing process.
4.4. Flexibility & peak processing capacity
In the case of a surge in traffic, applications still need to continue to function, but such bursts of traffic are not common. It is undoubtedly a huge waste to invest resources at any time in order to be able to handle such peak visits. The use of message queues enables key components to withstand sudden access pressure, and will not completely collapse due to sudden overloaded requests.
4.5. Recoverability
When some components of the system fail, the entire system will not be affected. The message queue reduces the coupling between processes, so even if a message processing process hangs up, the messages added to the queue can still be processed after the system is restored.
4.6. Order Guarantee
Based on the Queue message model, using the first-in-first-out feature of FIFO, the order of messages can be guaranteed.
In most usage scenarios, the order of data processing is important. Most message queues are sorted by nature, and it is guaranteed that the data will be processed in a specific order. (Kafka guarantees the orderliness of messages within a Partition)
4.7. Buffer
It helps to control and optimize the speed of data flow through the system, and solve the inconsistency of the processing speed of production messages and consumption messages.
4.8. Asynchronous communication
Synchronous and asynchronous sending and receiving of messages
- Synchronize:
The sending and receiving of messages supports synchronous sending and receiving.
At the same time, there is another synchronization method: in the synchronous sending and receiving scenario, the message producer and the consumer respond in a two-way manner. For example, Dilraba writes a letter to the post office, and then Yang Mi obtains the letter from the post office and writes a copy The receipt letter should be sent to the post office, and then Di Lieba will pick it up. Of course, Di Lieba has to specify the return address when writing a letter.
If the message is received synchronously (Pull), if the queue is empty, the reception will be in a synchronous blocking state at this time, and it will wait until the message arrives.
- asynchronous:
The sending and receiving of messages also supports asynchronous methods.
Send a message asynchronously without waiting for the confirmation of the message queue; receive a message asynchronously, and trigger the message consumer to receive the message in a Push mode.
In many cases, the user does not want and does not need to process the message immediately. The message queue provides an asynchronous processing mechanism that allows users to put a message into the queue, but does not process it immediately. Put as many messages as you want in the queue, and then process them when needed.
4.9. Transaction Support
The sending and receiving of messages supports transactions, such as: in the task center scenario, one processing may involve the receiving and processing of multiple messages, which are within the scope of the same transaction. If the processing of a message fails, the transaction is rolled back and the message is returned to the queue. in.
5. Application scenarios of message queues
- Asynchronous processing: such as SMS notification, terminal status push, App push, user registration, etc.
- Data synchronization: business data push synchronization
- Retry Compensation: Retry after accounting failure
- System decoupling: communication upstream and downstream, terminal abnormal monitoring, distributed event center
- Traffic peak reduction: the order status in the spike scenario
- Publish and subscribe: HSF service status change notification, distributed event center
- High concurrency buffer: log service, monitoring and reporting