Message queues are important components in distributed systems, in many production environments, such as buying goods and the like need to control the amount of concurrent scenes need to use.
This section describes two modes of message queues and message queue usage scenarios
1. Message Queuing (MQ) Overview
Message Queue (Message Queue), a distributed system is an important component of its common usage scenarios can be simply described as:
When the time does not need to immediately get results, but they require concurrency control, it is almost required when using the message queue.
The main message queue to solve the coupled application, asynchronous processing, traffic cut front and other issues.
Using the message queue currently has more RabbitMQ, RocketMQ, ActiveMQ, Kafka, ZeroMQ, MetaMq the like, and some databases such as Redis, Mysql and phxsql message function can also be implemented queue.
2. Message Queue usage scene
Comprising four message queues in the practical application scenarios:
Using Coupling l: between multiple application processes the same message through the message queue, the call interface to avoid failure results in failure of the entire process;
l asynchronous processing: multi-application message queue for the same message is processed, concurrent processing of messages between applications, compared to serial processing, reduce processing time;
l limiting clipping: widely used or rush spike activity, avoid excessive lead to hang application;
l message drive system: The system is divided into the message queue, the message producers, the message consumer, the producer is responsible for generating messages, consumers (there may be more) responsible for processing the message;
The following detailed description of the four scenes and how the message queue used in the above four scenarios:
2.1 asynchronous processing
Specific scenarios: Users To use an application, registration, the system needs to send registered e-mail and SMS verification. Treatment of these two operations in two ways: serial and parallel.
(1) serial mode: the new registration information generation, before sending the registration message, and then send a verification message;
In this way, we need to send a verification message and then eventually returned to the client.
(2) Parallel processing: the new registration information is written by mail, text messaging and parallel processing;
In this way, text messaging and e-mail to be processed and then returned to the client.
Time is assumed that the above three processing subsystems are 50ms, and does not consider network delay, the total processing time:
Serial: 50 + 50 + 50 = 150ms
Parallel: 50 + 50 = 100ms
When using message queues:
After writing the message queue and immediately returns success to the client, the response time depends on the total time to write a message queue, the message queue and the write time itself can be very fast, substantially negligible, so the total treatment time 2-fold increase compared to serial, parallel doubled compared;
2.2 application coupling
Specific scenarios: QQ users to upload a picture album, face recognition system will be the picture, the general practice is, after the server receives a picture, image upload system immediately calls the face recognition system, after the completion of call returns success, as shown below:
This method has the following disadvantages:
Face recognition system is adjusted fail, causing image upload failed;
High latency, face recognition system requires treatment, and then returns to the client, even if the user does not need to know immediately the result;
Call each other between the image upload system with face recognition systems, coupling needs to be done;
When using message queues:
After the client upload an image upload system information such as the image UIN, batch write message queue, directly returns success; Face Recognition System and data are taken periodically from the message queue, the complete identification of the new picture.
At this time, image upload system does not need to care about whether face recognition systems process information on these pictures, these pictures and when information is processed. In fact, since the user does not need to immediately know the results of face recognition, face recognition system can choose different scheduling strategies, according to leisure, busy, normal time, the picture information in the queue for processing.
2.3 limiting clipping
Specific scenarios: spike shopping site to carry out activities, generally due to the instantaneous volume of traffic, the server receives too large, it will cause traffic surge, related systems can not handle the request even collapse. After addition of the message queue, the system can fetch data from the message queue, the message queue corresponding to a buffer made.
This method has the following advantages:
The first request message into the queue, and not directly processed by the service processing system, a buffer made, greatly reducing the pressure of the service processing system;
Queue length may be limited, in fact, when the spike, the spike can not enter a queue of users to trade, these requests may be directly discarded, returned event has ended or commodities sold information;
2.4 message-driven system
Specific scenario: The user uploads a new batch of photos, face recognition systems require the user to re-photograph all user generated after clustering, clustering is done by reconciliation system Face Index (speed up queries). Between these three subsystems are connected together by a message queue, the processing results of the previous phase in a queue, the stage acquires a message from the queue, processing continues.
This method has the following advantages:
Avoid direct calls the current system leads to a system failure;
Each subsystem for handling messages may be more flexible, you may be selected to process a message is received, the timing process can be selected, processing may be divided into different time processing speed;
3. The two modes of message queue
Message Queuing includes two modes, point to point mode (point to point, queue) and publish / subscribe model (publish / subscribe, topic).
3.1 point to point mode
Ad-hoc mode includes three roles:
message queue
The sender (producer)
Recipient (consumer)
Message sender to the message queue of production, and then removed from the queue the message recipient and the message consumer. After the message is consumed, queue storage is no longer there, so the message recipient has been impossible to consumer consumption of news.
Point to Point mode features:
Each message has a recipient (Consumer) (i.e., once the consumer, the message is no longer in the message queue);
Between sender and receiver no dependencies, then the sender to send a message, whether or not the recipient is running, will not affect the next time to send a message to the sender;
Upon successful reception of the message recipient needs to reply to the success of the queue, the message queue in order to delete the currently received message;
3.2 publish / subscribe model
Publish / subscribe model includes three roles under:
Role topic (Topic)
Publisher (Publisher)
Subscribers (Subscriber)
Posted by sending a message to the Topic, the system will deliver these messages to multiple subscribers.
Publish / subscribe model features:
Each message may have a plurality of subscribers;
There are dependent on the time between publishers and subscribers. After for a theme (Topic) subscribers, it must create a subscriber, the message to the consumer publisher.
In order to consume messages, subscribers need to subscribe in advance the character theme, and keep the line running;
The main characteristics of four common message queue (RabbitMQ / ActiveMQ / RocketMQ / Kafka), advantages, disadvantages,
More technical information may concern: gzitcast