OK then the MQ message queue What routines do?
- And the benefits of using Message Queuing scene
- Message Queuing will bring what the problem, what solutions
- How to use MQ (for example a simple example with ActiveMQ)
A scenario of message queues and benefits:
-
Asynchronous - Traffic clipping
Let's look at the traditional server receives a processing request process
As shown above, when not in use the message queue of the server, the user requests are straight hate database, in the case of high concurrent database pressure surge, so that not only the response speed becomes slower, and therefore may also hang up the database, resulting in user pages directly error, the project manager came to the door, then * #! % @! # ** ...... (PS: Despite being linked to a service, but a treasure of the user, an alert message will be thrown pot to the network nowhere oh ~)
We look after the addition of the message queue server receives the request processing flow of what happens
As shown above, the message queue after use, the user requests to return data even in the case of high concurrent immediately after transmitting to the message queue, and then obtain data from the message queue by a consumer process the message queue, asynchronously written to the database. Since the message queue server processes the message a lot faster than the database, therefore ** response rate (user experience flu) ** be greatly improved.
So we can conclude that the message queue has a good flow clipping effect functions - that is, through asynchronous processing, the messages are stored for a short time high of concurrent transactions generated in the message queue, thereby slashing the concurrent transactions peak . As in some of the spike activity of certain electronic business platform, the rational use of message queues can withstand a large number of activities beginning of the influx of requests shock to the system. Because the user requested data immediately after writing the message queue back to the user, but the data in a subsequent request may fail verify operations, write operations database . So after using message queues for asynchronous processing, require appropriate changes to business processes with, for example, the user after submitting an order, the order data is written to the message queue, can not be returned to the user immediately orders submitted successfully , you need to really deal with the order in the message queue of the consumer process after completion of the order, even after the library, and then notify the user via email or SMS orders successfully, in order to avoid trade disputes. This is similar to mobile phone we usually train tickets and so on.
-
Asynchronous - decoupled system
I do look at first while the traditional system data transfer mode
As shown above, the main system and the other coupling systems too, are a direct call, or a little bit changes add modules, both have to change the code, too cumbersome
Then, after we look at adding message queue, the system structure of what happens
As shown above, we know that if the module between the direct call does not exist , then add or modify the module on the module to other modules less influence, such a system scalability undoubtedly better.
The use of message queues publish - subscribe model work, the message sender (producer) announced that one or more message recipients (consumers) a subscription message. From the figure can be seen that there is no direct coupling between the message sender (producer) and the message receiver (consumer), the message sender to send a message to a message queue that is distributed ends the processing of the message, the recipient message from the distribution after subsequent processing message queue and obtain the message, it does not need to know the message comes from. For new business, as long as interested in that type of news, you can subscribe to this message, no impact on existing systems and services, enabling the scalable business website design.
In addition, to avoid the message queue server downtime caused by the loss of a message , the message will be successfully sent to the queue of messages stored on the message server producer, and other consumer after the message is processed real server to delete the message. After the message queue server downtime, server manufacturers will choose other servers distributed message queue server cluster post messages.
In addition to publishing subscription model outside, message queues, there are other modes of transport
Point to point model
The base model, only a sender, a receiver and a distributed queue.
Producer consumer model
If the sender and receiver can have multiple deployment instances, even of different types; but share the same queue, it becomes a standard producer consumer model. In this model, the role of three commonly referred to production (Producer), distributed queue (Queue), consumers (Consumer).
Summary way: a message queue for concurrent processing capability and scalability of the system has improved
Second, the use of message queues will bring what the problem:
- ** reduced availability: ** Before joining MQ, you do not consider the situation MQ server hang, after the introduction of MQ you will need to consider the availability decreases.
- Increased complexity: After adding MQ, you need to ensure that the message is not repeated consumption, handling of message loss, assure an orderly messaging and other issues. So what needs to be considered more and system complexity increases.
- ** Data Consistency: ** brought asynchronous message queues can really improve system response speed, however, if the message is not really correct consumer news consumer how to do? This will lead to inconsistent data situation.
2.1 What solutions
- For usability issues
After the introduction of the message queue, availability of the system decreases. The actual project sending MQ messages, if not a cluster, where mq accident to the machinery down, then mq messages can not be sent, the system collapsed, so we need to cluster MQ, MQ when one of them broke down, the rest of MQ machine can then continue to operate in production, no one stand-alone use of message queues. If so, it is certainly in order and with the use (technical complexity seems a bit, well Fudge overcharged money), for that matter, we need to have a deeper understanding of MQ cluster technology, a variety of different messaging middleware cluster approach, with the following ActiveMq the cluster as an example (Zookeeper + ActiveMq), look at the map
This scenario is called the master / slave (master / slave mode). In this scenario, I have three servers (primary and standby), in any case, only the "master" at work, "prepared" is the main failure when, replacing "master" to provide services. With the support of zookeeper, this process is accomplished, Zookeeper and node provides directory service, when my three servers starts, it creates a temporary node corresponds to its own in a specified directory zookeeper's (a process called " registration "), so-called temporary nodes is by heartbeats (regularly send data packets) to maintain the zookeeper server when the primary server fails (can not send data packets to the zookeeper server, zookeeper removes the change temporary node server to register with the zookeeper, zookeeper will assign serial numbers, we believe that the small serial number, is the "master", the serial number of the big one, is "prepared."
When the address that sets the "master" server of our client (usually a web server) needed to access the service, you need to connect zookeeper, obtain a temporary list of nodes in the specified directory, which is registered server information, access to small serial number , subsequent access operation. In order to achieve "always access the primary server." . When the "master" server fails, zookeeper delete the temporary node corresponding from the specified directory, and can notify all clients concerned about this change, efficient and rapid dissemination of this information. The time when the next request, or ZooKeeper connection, but in this case in fact access the alternate MQ.
For how to configure a cluster here is not to demonstrate, on their own Internet search tutorial, a lot!
- For the complexity of the problem
- How to ensure that messages are not repeated consumption of it?
To answer this question, we must first know why the message will be repeated consumption, mostly resulting in unreasonable because the network, information is no longer transferred to the message queue, the message queue does not result in the consumer know that they have had the message, and the message distribution again to other consumers. So we have the following three ideas to solve the problem the way
①. If the message is do insert operation of the database, this message is to make a unique primary key, even if the situation repeated consumption occurs, it will lead to a primary key conflict, to avoid the database appear dirty data. ②. If you do get the news operation set the redis, do not solve, because you set several times whether the results are the same, even if the power had set the operation and other operations. ③. If the above two conditions is not enough, prepare a third party to do the service consumer records. In redis example, a global id assigned to the message, as long as the message consumer through the <id, message> redis written in the form of KV. Before consumers start spending it, go redis there is no consumption record query can be.
2. How to ensure the reliability of transmission of the message it?
Actually, this is an extension of the first question, in other words, we want to ensure the reliability of transmission, in fact, it is to ensure that prevent producers lost data, message queues, lost data, lost consumer data only
In fact, these issues early in the middleware developers have taken into account, but also provides a number of documents can be configured to set the parameters of our own, message queues generally persisted to disk Do not worry about data loss if the producer then MQ transactions will be rolled back, can try to resend, consumers lost words are generally used automatic confirmation message mode leads to consumer information is deleted, as long as the modifications to manual confirmation on the line, that is to say after the completion of consumer spending, a call to the MQ confirmation method on the line
3. How to get assurance from the message queue data execution order?
By algorithm, we will need to maintain the order of messages into the same message queue, and then only with a consumer to spend the queue.
(. 1) RabbitMQ : a plurality of split queue, each queue a consumer, it is a little more queue, is troublesome indeed point; or it corresponds to a queue, but a consumer, then the consumer do queuing with internal memory queue, then distributed worker to handle different underlying
(2) Kafka used to live : a topic, a partition, a consumer, a single-threaded internal consumption, and write memory queue of N, N threads are then consumption to a memory queue
4. How to solve the delay and message queues expire problem? There are millions of messages backlog continued for several hours, how to solve?
The problem is, examine how you solve the problem quickly after production environment ,, accident delayed the message queue and self-protection mechanism is expired message queue, the purpose is to prevent itself being tumbled, of course, can be closed to protect, for example, when a message is the consumer fails five times, put the news dropped, etc., try not to turn off the protection mechanism, so the question is, who the message is discarded, do not do it? Not really, we can for this business, check out the batch of data will be lost, and write a temporary program, check out a little bit, and then re-poured mq inside, the lost data make it up to him.
The data is given to the end consumer by push or pull mode, each of which has any drawbacks?
-
Push model real-time, but because of the state of maintenance and other issues, it is difficult to put into practice the message middleware, as
-
In the end the need to maintain state Broker Consumer, and not suitable for Broker to support a large number of scenes Consumer Consumer Consumer speed is inconsistent, Broker be pushing hard to handle different situations Consumer Consumer's Broker difficult to deal with the situation can not consume messages, because I do not know Consumer downtime is temporary or permanent) another push message (the amount may be large) will increase the load or crushed Consumer Consumer.
If the corresponding only a Consumer, better than with push pull.
-
Pull mode will be relatively simple to implement, but the real-time depending on the frequency in rotation in demanding real-time high of scenes not suitable for use.
III. How to use MQ (for example to ActiveQM)
Attach official website: activemq.apache.org/
Attach to start the service access address: http://127.0.0.1:8161/admin/ username / password admin / admin
Attach the code, jar wrapped himself under pan.baidu.com/s/1SUBoypW-...
Published subscription model
Producer - Post
public class JMSProducer {
private static final String USERNAME=ActiveMQConnection.DEFAULT_USER; // 默认的连接用户名
private static final String PASSWORD=ActiveMQConnection.DEFAULT_PASSWORD; // 默认的连接密码
private static final String BROKEURL=ActiveMQConnection.DEFAULT_BROKER_URL; // 默认的连接地址
private static final int SENDNUM=10; // 发送的消息数量
public static void main(String[] args) {
ConnectionFactory connectionFactory; // 连接工厂
Connection connection = null; // 连接
Session session; // 会话 接受或者发送消息的线程
Destination destination; // 消息的目的地
MessageProducer messageProducer; // 消息生产者
// 实例化连接工厂
connectionFactory=new ActiveMQConnectionFactory(JMSProducer.USERNAME, JMSProducer.PASSWORD, JMSProducer.BROKEURL);
try {
connection=connectionFactory.createConnection(); // 通过连接工厂获取连接
connection.start(); // 启动连接
session=connection.createSession(Boolean.TRUE, Session.AUTO_ACKNOWLEDGE); // 创建Session
// destination=session.createQueue("FirstQueue1"); // 创建消息队列
destination=session.createTopic("FirstTopic1");
messageProducer=session.createProducer(destination); // 创建消息生产者
sendMessage(session, messageProducer); // 发送消息
session.commit();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally{
if(connection!=null){
try {
connection.close();
} catch (JMSException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
}
/**
* 发送消息
* @param session
* @param messageProducer
* @throws Exception
*/
public static void sendMessage(Session session,MessageProducer messageProducer)throws Exception{
for(int i=0;i<JMSProducer.SENDNUM;i++){
TextMessage message=session.createTextMessage("ActiveMQ 发送的消息"+i);
System.out.println("发送消息："+"ActiveMQ 发布的消息"+i);
messageProducer.send(message);
}
}
}
复制代码Consumers - Subscribe
/**
* 消息监听-订阅者一
* @author Administrator
*
*/
public class Listener implements MessageListener{
@Override
public void onMessage(Message message) {
// TODO Auto-generated method stub
try {
System.out.println("订阅者一收到的消息："+((TextMessage)message).getText());
} catch (JMSException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
复制代码public class JMSConsumer {
private static final String USERNAME=ActiveMQConnection.DEFAULT_USER; // 默认的连接用户名
private static final String PASSWORD=ActiveMQConnection.DEFAULT_PASSWORD; // 默认的连接密码
private static final String BROKEURL=ActiveMQConnection.DEFAULT_BROKER_URL; // 默认的连接地址
public static void main(String[] args) {
ConnectionFactory connectionFactory; // 连接工厂
Connection connection = null; // 连接
Session session; // 会话 接受或者发送消息的线程
Destination destination; // 消息的目的地
MessageConsumer messageConsumer; // 消息的消费者
// 实例化连接工厂
connectionFactory=new ActiveMQConnectionFactory(JMSConsumer.USERNAME, JMSConsumer.PASSWORD, JMSConsumer.BROKEURL);
try {
connection=connectionFactory.createConnection(); // 通过连接工厂获取连接
connection.start(); // 启动连接
session=connection.createSession(Boolean.FALSE, Session.AUTO_ACKNOWLEDGE); // 创建Session
// destination=session.createQueue("FirstQueue1"); // 创建连接的消息队列
destination=session.createTopic("FirstTopic1");
messageConsumer=session.createConsumer(destination); // 创建消息消费者
messageConsumer.setMessageListener(new Listener()); // 注册消息监听
} catch (JMSException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
复制代码I think a good distributed message queue, you should have the following capabilities: high throughput, low latency (due to different scenes), transmission transparent, strong scalability, the ability to have redundant disaster, the consistency of the order of delivery, asynchronous, synchronous + send, improve the operation and maintenance and monitoring tools and open source