1. Why should I use message queues
A: This question, we answered only three main scenarios (there are undeniable pinch, but the only answer three main), that the following six words: decoupling, asynchronous, clipping
(1) decoupling
The traditional model:
The traditional model of disadvantages:
Coupling inter-system too, as shown above, the system calling code system A and system B directly in C code, if the future access system D, the system also need to modify the code A, too cumbersome!
Intermediate mode the advantages of:
Writes messages to the message queue, the system requires its own subscription message from the message queue, so that the system A without any modification.
(2) Asynchronous
traditional model:
The traditional model of disadvantages:
Some non-essential business logic running in a synchronized manner, too time-consuming.
Middleware mode:
Intermediate mode the advantages of:
(3) clipping
The traditional model:
The traditional model of disadvantages:
Concurrent amount of time, all requests to the database directly dislike, resulting in connection failure database
Middleware mode:
Intermediate mode the advantages of:
A system in accordance with the amount of slowly concurrent database can handle, and slowly pull message from the message queue. In production, this short peak of the backlog is allowed.
2. What are the disadvantages will be the message queue
answer:
1. reduce system availability: You want it, as long as the other systems have been run properly, then your system is normal. Now you have to join the queue into the message, that message queue hung up, your system is not the Oh. Therefore, the system will reduce the availability of
2. System complexity increases: adding the message queue, to give more consideration many issues, such as: consistency issues, how to ensure that messages are not repeated consumption, how to ensure the reliability of information transmission. Therefore, the need to consider more things, tingling complexity increases.
3. How message queue selection?
A: First, we ActiveMQ first on the community to see the update frequency of the MQ:
-
Apache ActiveMQ 5.15.3 Release
-
Christopher L. Shannon posted on Feb 12, 2018
-
Apache ActiveMQ 5.15.2 Released
-
Christopher L. Shannon posted on Oct 23, 2017
-
Apache ActiveMQ 5.15.0 Released
-
Christopher L. Shannon posted on Jul 06, 2017
-
Omitted the following records
We can see, ActiveMQ months to send a version, it is said research focus in their next-generation products Apollo.
Next, we go take a look RabbitMQ community, update frequency of RabbitMQ
-
RabbitMQ 3.7.3 release 30 January 2018
-
RabbitMQ 3.6.15 release 17 January 2018
-
RabbitMQ 3.7.2 release23 December 2017
-
RabbitMQ 3.7.1 release21 December 2017
-
Omitted the following records
-
...
We can see, RabbitMQ release a lot more frequently than ActiveMq. As RocketMQ and kafka not take everyone looked, in short, it is also active in more than ActiveMQ. Details, look yourself.
| characteristic | ActiveMQ | RabbitMQ | RocketMQ | kafka |
|---|---|---|---|---|
| Development language | java | erlang | java | scala |
| Stand-alone Throughput | Ten thousand | Ten thousand | 100 000 | 100 000 |
| Timeliness | ms level | Class us | ms level | ms level within |
| Availability | High (master-slave architecture) | High (master-slave architecture) | Very high (distributed architecture) | Very high (distributed architecture) |
| Features | Mature product, many companies applied; there are more documents; better support a variety of protocols | Based on erlang development, so the concurrent competence, performance is extremely good, low latency; rich management interface | MQ function relatively complete, scalable good | MQ supports only the main features, like some message queries, and so did not provide a message back, after all the data is ready for the big, wide application in the field of big data. |
Integrated the above materials come to the following two points:
(1) small and medium sized software companies, the proposed election RabbitMQ on the one hand, erlang language born with characteristics of high concurrency, and he's very convenient to use management interface. As the saying goes, as also Xiao, also lost! He drawbacks here, too, though RabbitMQ is open source, but there are a few domestic energy customized development erlang programmers do? Fortunately, RabbitMQ community is very active, can solve the bug encountered in the development process, this point is very important for small and medium sized companies. The reason does not consider rocketmq and kafka on the one hand small and medium sized software companies as good as the Internet company, the amount of data is not so large, the election messaging middleware, more complete functionality should be preferred, as to kafka excluded. The reason is not considered rocketmq, rocketmq is produced Ali, Ali give up if maintenance rocketmq, small and medium sized companies are generally not spare people to customized development rocketmq therefore not recommended.
(2) large software companies, depending on the particular use and between rocketMq kafka second election. On the one hand, large software companies, have enough money to build a distributed environment, but also have a sufficiently large amount of data. For rocketMQ, large software companies can extract manpower to rocketMQ be customized development, after all, we have the capacity to change the JAVA source person, or quite a few. As kafka, according to the business scene selection, if there is a log collection function, is definitely the preferred kafka. The specific what to do, to see usage scenarios.
4. How to ensure that messages are not repeated consumption
A: The first is why will cause duplication of spending?
In fact, no matter what kind of message queues, resulting in repeated consumption reasons they are in fact similar. Under normal circumstances, when consumers consume messages, post-consumer completed, a confirmation message will be sent to the message queue, the message queue will know the message is consumed, it will delete the message from the message queue. Just different forms of confirmation messages sent a different message queue, such as RabbitMQ is to send a confirmation message ACK, RocketMQ CONSUME_SUCCESS return a success flag, kafka fact, the concept of a offet, briefly about, is that every message has an offset after, kafka over-consumption message, you need to submit offset, let the consumer know that they have a message queue before.
That the cause of repeated consumption? Is because the network transmission failure and so on, information is no longer transferred to the message queue, the message queue does not result in the consumer know that they have had the message, and the message will again be distributed to other consumers.
How to solve? The problem for the business scenarios to answer, divided into the following three conditions:
(1) For example, do you get this message insert operation of the database, it would be easy to give this message to make a unique primary key, even if the situation repeated consumption occurs, it will lead to a primary key conflict, to avoid the database appear dirty data.
(2) As another example, do you get the news operation set the redis, it would be easy, not solve, because you set several times whether the results are the same, even if the power had set the operation and other operations.
(3) If the above two cases is not enough, the big move. Prepare a third-party media, do consumer records. To redis example, a global id assigned to the message, as long as the consumer through the message, <id, message> to write redis KV form before consumers start spending it, go redis query can have no consumption record.
5. How to ensure the reliability of transmission of consumption?
A: In fact, this transmission reliability, each MQ must be analyzed from three perspectives:
- Producers lost data
- Lost data message queue
- Consumers lost data
(1) lose the data producer
from the producer data lost this perspective, RabbitMQ provide transaction model to ensure and confirm message is not lost producers.
transaction mechanism that is to say, before sending the message, open transactions (channel.txSelect ()), then sends a message that if anything unusual occurs during sending, the transaction will be rolled back (channel.txRollback ()), if the transaction is committed successfully sent (channel.txCommit ()).
However, this approach has a drawback: throughput. Because, in accordance, with the majority of production experience confirm mode. Once entering confirm channel mode, all messages are posted in the channel will be assigned a unique ID (starting at 1), once the message is delivered to all matching queue, RabbitMQ sends an ACK to the producer ( the message contains a unique ID), which the manufacturers know the message has correctly reached a destination queue. If rabbitMQ could not process the message, it will send a Nack message to you, you can perform a retry operation. Ack and Nack processing codes are as follows:
channel.addConfirmListener(new ConfirmListener() { @Override public void handleNack(long deliveryTag, boolean multiple) throws IOException { System.out.println("nack: deliveryTag = "+deliveryTag+" multiple: "+multiple); } @Override public void handleAck(long deliveryTag, boolean multiple) throws IOException { System.out.println("ack: deliveryTag = "+deliveryTag+" multiple: "+multiple); } });
(2) lost data message queue
Where lost data message queue processing, typically it is on persistent disks. This configuration can confirm persistence mechanism used in conjunction with, you can give the producer sends a message Ack signal after persistent disk. In this way, if the message before persistent disk, rabbitMQ killed, so producers can not receive Ack signal, the producers will be automatically re-issued.
So how persistent it, by the way here, in fact, very easy, just two steps below
- The persistent queue identifier durable set to true, represents a durable queue
- When sending a message to deliveryMode = 2
After this set, even rabbitMQ hung up, restart can recover data
(3) Consumer lost data
Consumer data is typically lost due acknowledgment message using the automatic mode. In this mode, consumers will automatically receive a confirmation message. Then rabbitMQ will immediately delete the message, in this case, an exception if the consumer fails to process the message, the message will be lost.
As solutions to manual acknowledgment message.
6 .. how to ensure the order of the message
A: To address this issue, through an algorithm, you will need to maintain the order of messages into the same message queue (kafka in that partition, rabbitMq in that queue). And then only with the team to spend a consumer column.
Some people may ask: What if for throughput, consumers have more to spend how to do?
This problem, there is no fixed answer routine. For example, we have operating a microblogging, microblogging, write a review, delete it, and the three asynchronous operation. If this is a business scenario, and that as long as retry on the line. For example, a consumer you do an operation to write a review, but this time, micro-blog is not made, write a review must be a failure, and so for some time. And other consumer after another, the first to write a review of the implementation of the operation, and then execute, you can to succeed.
In short, for this question, my view is to ensure an orderly into the team on the line, after the order of the team to the consumers themselves to ensure there is no fixed routine.