Message queue middleware is an important component in distributed systems, mainly solving problems such as application coupling, asynchronous messages, and traffic cutting
Achieve high performance, high availability, scalability and eventual consistency architecture. The most comprehensive Java interview site
The most used message queues are RocketMQ, RabbitMQ, Kafka, ZeroMQ, MetaMQ
The following describes the common usage scenarios of message queues in practical applications.
Five scenarios: asynchronous processing, application decoupling, traffic shaving, log processing, and message communication.
Scenario 1: Asynchronous processing
Scenario description: After the user registers, he needs to send a registration email and a registration SMS. There are two traditional methods: 1. Serial way; 2. Parallel way
This article has been included in the Github warehouse, which includes computer foundation, Java foundation, multithreading, JVM, database, Redis, Spring, Mybatis, SpringMVC, SpringBoot, distributed, microservices, design patterns, architecture, school recruitment and social recruitment sharing, etc. Core knowledge points, welcome to star~
If you can't access Github, you can access the gitee address.
(1) Serial mode: After successfully writing the registration information into the database, send the registration email, and then send the registration SMS. After the above three tasks are all completed, return to the client
[External link picture transfer failed, the source site may have an anti-theft link mechanism, it is recommended to save the picture and upload it directly (img-i992Okcj-1681632859093) (http://img.topjavaer.cn/img/mq usage scenario 1.png) ]
(2) Parallel mode: After the registration information is successfully written into the database, the registration email and the registration SMS are sent at the same time. After the above three tasks are completed, return to the client. The difference from serial is that the parallel method can improve the processing time
[External link picture transfer failed, the source site may have an anti-theft link mechanism, it is recommended to save the picture and upload it directly (img-gp6kvSRM-1681632859098) (http://img.topjavaer.cn/img/mq usage scenario 2.png) ]
Assuming that each of the three service nodes uses 50 milliseconds, without considering other overheads such as the network, the serial time is 150 milliseconds, and the parallel time may be 100 milliseconds.
Because the number of requests processed by the CPU per unit time is certain, assume that the throughput of the CPU within 1 second is 100 times. Then the CPU can process 7 requests (1000/150) within 1 second in the serial mode. The number of requests processed in parallel is 10 times (1000/100)
Summary: As described in the above case, the performance (concurrency, throughput, response time) of the system in the traditional way will have a bottleneck. How to solve this problem?
The introduction of message queues will not require business logic and will be processed asynchronously. The modified structure is as follows:
[External link picture transfer failed, the source site may have an anti-theft link mechanism, it is recommended to save the picture and upload it directly (img-9HtSUIMS-1681632859099) (http://img.topjavaer.cn/img/mq usage scenario 3.png) ]
According to the above agreement, the user's response time is equivalent to the time for the registration information to be written into the database, which is 50 milliseconds. After registering emails and sending text messages and writing them into the message queue, they return directly, so the speed of writing to the message queue is very fast and can be ignored, so the user's response time may be 50 milliseconds. Therefore, after the architecture change, the throughput of the system is increased to 20 QPS per second. 3x faster than serial and 2x faster than parallel
Scenario 2: Application Decoupling
Scenario description: After the user places an order, the order system needs to notify the inventory system. Traditionally, the order system calls the interface of the inventory system. As shown below
[External link picture transfer failed, the source site may have an anti-theft link mechanism, it is recommended to save the picture and upload it directly (img-huhDH75D-1681632859101) (http://img.topjavaer.cn/img/mq usage scenario 4.png) ]
Disadvantages of the traditional model:
- If the inventory system is inaccessible, the order will fail to reduce the inventory, causing the order to fail
- Order system coupled with inventory system
How to solve the above problems? The solution after introducing the application message queue is shown in the following figure:
[External link picture transfer failed, the source site may have an anti-theft link mechanism, it is recommended to save the picture and upload it directly (img-RfeLiXlb-1681632859102) (http://img.topjavaer.cn/img/mq usage scenario 5.png) ]
- Order system: After the user places an order, the order system completes the persistence process, writes the message to the message queue, and returns the user's order successfully
- Inventory system: Subscribe to the news of the order, and use the pull/push method to obtain the order information, and the inventory system performs inventory operations according to the order information
- If: The inventory system is not working properly when placing an order. It does not affect the normal order placement, because after the order is placed, the order system writes to the message queue and no longer cares about other follow-up operations. Realize the application decoupling of order system and inventory system
Scenario 3: Traffic cutting
Traffic shaving is also a common scenario in message queues, and is generally used in seckill or group grabbing activities
Application scenario: In seckill activities, generally due to excessive traffic, the traffic will increase sharply and the application will hang up. To solve this problem, it is generally necessary to add a message queue at the front end of the application.
- The number of people who can control the activity
- It can alleviate the application of high traffic in a short period of time
[External link picture transfer failed, the source site may have an anti-theft link mechanism, it is recommended to save the picture and upload it directly (img-9RrFc6B3-1681632859102) (http://img.topjavaer.cn/img/mq usage scenario 6.png) ]
- After the user's request is received by the server, it is first written into the message queue. If the length of the message queue exceeds the maximum number, the user request will be discarded directly or the error page will be redirected
- According to the request information in the message queue, the seckill business will do follow-up processing
Scenario 4: Log Processing
Log processing refers to the use of message queues in log processing, such as the application of Kafka, to solve the problem of massive log transmission. The architecture is simplified as follows
[External link picture transfer failed, the source site may have an anti-theft link mechanism, it is recommended to save the picture and upload it directly (img-0UBVp3dC-1681632859103) (http://img.topjavaer.cn/img/mq usage scenario 7.png) ]
- The log collection client is responsible for collecting log data and writing it to the Kafka queue regularly
- Kafka message queue, responsible for receiving, storing and forwarding log data
- Log processing application: subscribe and consume log data in kafka queue
The following is an application case of Sina kafka log processing
[External link picture transfer failed, the source site may have an anti-theft link mechanism, it is recommended to save the picture and upload it directly (img-iLi0a03O-1681632859104) (http://img.topjavaer.cn/img/mq usage scenario 8.png) ]
(1), Kafka: message queue for receiving user logs
(2), Logstash: Do log analysis, unify into JSON output to Elasticsearch
(3), Elasticsearch: the core technology of real-time log analysis service, a schemaless, real-time data storage service, organizes data through index, and has powerful search and statistical functions
(4), Kibana: Elasticsearch-based data visualization components, super data visualization capabilities are an important reason why many companies choose ELK stack
Scenario 5: Message communication
Message communication means that message queues generally have built-in efficient communication mechanisms, so they can also be used in pure message communication. For example, implement point-to-point message queues, or chat rooms, etc.
Point-to-point communication:
[External link picture transfer failed, the source site may have an anti-theft link mechanism, it is recommended to save the picture and upload it directly (img-zkwlz0Ti-1681632859104) (http://img.topjavaer.cn/img/mq usage scenario 9.png) ]
Client A and Client B use the same queue for message communication.
Chat room communication:
[External link picture transfer failed, the source site may have an anti-theft link mechanism, it is recommended to save the picture and upload it directly (img-eVJggmMI-1681632859107) (http://img.topjavaer.cn/img/mq usage scenario 10.png) ]
Client A, client B, and client N subscribe to the same topic to publish and receive messages. Achieve a chat room-like effect.
The above are actually two message modes of message queues, point-to-point or publish-subscribe mode. The model is a schematic diagram for reference.
Finally, I would like to share with you a Github warehouse, which has more than 300 classic computer book PDFs compiled by Dabin, including C language, C++, Java, Python, front-end, database, operating system, computer network, data structure and algorithm, machine learning , programming life , etc., you can star it, next time you look for a book directly search on it, the warehouse is continuously updated~
