Message queue middleware is an important component in distributed systems, mainly solving problems such as application coupling, asynchronous messages, and traffic cutting
Achieve high performance, high availability, scalability and eventual consistency architecture
The most used message queues are ActiveMQ, RabbitMQ, ZeroMQ, Kafka, MetaMQ, RocketMQ
2. Message queue application scenarios
The following describes the common usage scenarios of message queues in practical applications. Four scenarios of asynchronous processing, application decoupling, traffic cutting and message communication
2.1 Asynchronous processing
Scenario description: After the user registers, he needs to send a registration email and a registration SMS. There are two traditional methods: 1. Serial way; 2. Parallel way
(1) Serial mode: After successfully writing the registration information into the database , send the registration email, and then send the registration SMS. After the above three tasks are all completed, return to the client
(2) Parallel mode: After the registration information is successfully written into the database, the registration email and the registration SMS are sent at the same time. After the above three tasks are completed, return to the client. The difference from serial is that the parallel method can improve the processing time
Assuming that each of the three service nodes uses 50 milliseconds, without considering other overheads such as the network, the serial time is 150 milliseconds, and the parallel time may be 100 milliseconds.
Because the number of requests processed by the CPU per unit time is certain, assume that the throughput of the CPU within 1 second is 100 times. Then the CPU can handle 7 requests (1000/150) within 1 second in the serial mode. The amount of requests processed in parallel is 10 times (1000/100)
Summary: As described in the above case, the performance (concurrency, throughput, response time) of the system in the traditional way will have a bottleneck. How to solve this problem?
The introduction of message queues will not require business logic and will be processed asynchronously. The modified structure is as follows:
According to the above agreement, the user's response time is equivalent to the time for the registration information to be written into the database, which is 50 milliseconds. After registering an email, sending a short message and writing it into the message queue, it returns directly, so the speed of writing to the message queue is very fast, which can basically be ignored, so the user's response time may be 50 milliseconds. Therefore, after the architecture change, the throughput of the system is increased to 20 QPS per second. 3 times better than serial and 2 times better than parallel
2.2 Application decoupling
Scenario description: After the user places an order, the order system needs to notify the inventory system. Traditionally, the order system calls the interface of the inventory system. As shown below
Disadvantages of the traditional model:
-
If the inventory system is inaccessible, the order will fail to reduce the inventory, causing the order to fail
-
Order system coupled with inventory system
How to solve the above problems? The solution after introducing the application message queue is shown in the following figure:
-
Order system: After the user places an order, the order system completes the persistence process, writes the message to the message queue, and returns the user's order successfully
-
Inventory system: Subscribe to the news of the order, and use the pull/push method to obtain the order information, and the inventory system performs inventory operations according to the order information
-
If: The inventory system is not working properly when placing an order. It does not affect the normal order placement, because after the order is placed, the order system writes to the message queue and no longer cares about other follow-up operations. Realize the application decoupling of order system and inventory system
2.3 Flow cutting
Traffic shaving is also a common scenario in message queues, and is generally used in seckill or group grabbing activities
Application scenario: In seckill activities, generally due to excessive traffic, the traffic will increase sharply and the application will hang up. To solve this problem, it is generally necessary to add a message queue at the front end of the application.
-
The number of people who can control the activity
-
It can alleviate the application of high traffic in a short period of time
-
After the user's request is received by the server, it is first written into the message queue. If the length of the message queue exceeds the maximum number, the user request will be discarded directly or the error page will be redirected
-
According to the request information in the message queue, the seckill business will do follow-up processing
2.4 Log Processing
Log processing refers to the use of message queues in log processing, such as the application of Kafka, to solve the problem of massive log transmission. The architecture is simplified as follows
-
Log collection client, responsible for log data collection, regularly writes and receives writes to the Kafka queue
-
Kafka message queue, responsible for receiving, storing and forwarding log data
-
Log processing application: subscribe and consume log data in kafka queue
The following is an application case of Sina kafka log processing: Transfer from (http://cloud.51cto.com/art/201507/484338.htm)
(1) Kafka: message queue for receiving user logs
(2) Logstash: Do log analysis, unify into JSON output to Elasticsearch
(3) Elasticsearch: the core technology of real-time log analysis service, a schemaless, real-time data storage service, organizes data through index, and has powerful search and statistical functions
(4) Kibana: Elasticsearch-based data visualization components, super data visualization capabilities are an important reason why many companies choose ELK stack
2.5 Message communication
Message communication means that message queues generally have built-in efficient communication mechanisms, so they can also be used in pure message communication. For example, implement point-to-point message queues, or chat rooms, etc.
Point-to-point communication:
Client A and client B use the same queue for message communication.
Chat room communication:
Client A, client B, and client N subscribe to the same topic to publish and receive messages. Achieve a chat room-like effect.
The above are actually two message modes of message queues, point-to-point or publish-subscribe mode. The model is a schematic diagram for reference.
3. Example of message middleware
3.1 E-commerce system
The message queue adopts highly available and durable message middleware. Such as Active MQ, Rabbit MQ, Rocket Mq.
(1) After the application completes the main logic processing, write it into the message queue. Whether the message is sent successfully can enable the confirmation mode of the message. (After the message queue returns the status of message receiving success, the application returns again, so as to ensure the integrity of the message)
(2) Extend the process (sending text messages, delivery processing) to subscribe to queue messages. Use push or pull to get messages and process them.
(3) While the message decouples the application, it brings data consistency problems, which can be solved by using the final consistency method. For example, the main data is written into the database, and the extended application realizes the follow-up processing based on the message queue in combination with the database method according to the message queue.
3.2 Log collection system
It is divided into four parts: Zookeeper registration center, log collection client, Kafka cluster and Storm cluster (OtherApp).
-
Zookeeper registry, proposes load balancing and address lookup services
-
The log collection client is used to collect the logs of the application system and push the data to the kafka queue
-
Kafka cluster: receiving, routing, storing, forwarding and other message processing
Storm cluster: at the same level as OtherApp, using pull to consume data in the queue
Four, JMS message service
When talking about message queues, you have to mention JMS. JMS ( Java Message Service, Java Message Service) API is a message service standard/specification that allows application components to create, send, receive, and read messages based on the JavaEE platform. It makes distributed communication less coupled, and messaging services more reliable and asynchronous.
In the EJB architecture, message beans can be seamlessly integrated with JM message services. In the J2EE architecture mode, there is a message server mode, which is used to realize the direct decoupling of messages and applications.
4.1 Message Model
In the JMS standard, there are two message models P2P (Point to Point), Publish/Subscribe (Pub/Sub).
4.1.1 P2P mode
P2P mode includes three roles: message queue (Queue), sender (Sender), receiver (Receiver). Each message is sent to a specific queue, and the receiver gets the message from the queue. Queues hold messages until they are consumed or timed out.
Features of P2P
-
Each message has only one consumer (Consumer) (that is, once consumed, the message is no longer in the message queue)
-
There is no time dependency between the sender and the receiver, which means that when the sender sends a message, it does not affect the message being sent to the queue regardless of whether the receiver is running or not
-
The receiver needs to respond to the queue after successfully receiving the message
If you want every message sent to be successfully processed, then you need P2P mode. (Architecture KKQ: 466097527, welcome to join)
4.1.2 Pub/sub mode
Contains three roles: Topic, Publisher, and Subscriber. Multiple publishers send messages to Topic, and the system delivers these messages to multiple subscribers.
Features of Pub/Sub
-
Each message can have multiple consumers
-
There is a time dependency between publishers and subscribers. For a subscriber of a topic (Topic), it must create a subscriber before it can consume the publisher's message
-
In order to consume messages, the subscriber must remain running
To alleviate such strict timing dependencies, JMS allows subscribers to create a durable subscription. This way, even if the Subscriber is not activated (running), it can receive the Publisher's messages.
If the message you want to send can be processed without any processing, or can be processed by only one messager, or can be processed by multiple consumers, then the Pub/Sub model can be used.
4.2 Message consumption
In JMS, the production and consumption of messages are asynchronous. For consumption, JMS messagers can consume messages in two ways.
(1) synchronization
Subscribers or receivers receive messages through the receive method, and the receive method will block until the message is received (or before timeout);
(2) asynchronous
A subscriber or receiver can register as a message listener. When the message arrives, the system automatically calls the listener's onMessage method.
JNDI: Java Naming and Directory Interface, is a standard Java naming system interface. Services can be found and accessed on the web. By specifying a resource name, the name corresponds to a record in the database or naming service, and returns the information necessary for resource connection establishment.
JNDI plays a role in finding and accessing the sending target or message source in JMS.