MQ current mainstream products
1.ZeroMQ
Known as the fastest message queuing systems, in particular the demand for high throughput scenarios.
Scalable, flexible development using C language, is really just re-packaged a socket library, if used as a message queue, the need to develop a lot of code. ZeroMQ provide only non-persistent queues, that if down machine, data will be lost. Wherein, Twitter using the Storm ZeroMQ as transmission data stream.
2.RabbitMQ
Erlang Concurrent combine the advantages of language itself, we support many protocols: AMQP, XMPP, SMTP, STOMP, it is also true, so it becomes very heavyweight, is more suitable for the development of enterprise-class.
Better performance, but not conducive to do secondary development and maintenance.
3.ActiveMQ
The historic open source project is a subproject of Apache. In many products have been applied to achieve the JMS1.1 specification, can be spring-jms easy integration to achieve a variety of protocols, not lightweight (source code than RocketMQ and more), support persisted to the database, the number of queues for more the situation is not good support.
4.Redis
KV as a memory-based database, which provides news subscription service that can be used as MQ, there is less application cases, and inconvenient extension. For the team and the operations team RabbitMQ and Redis, and implementing a million times, once every 100,000 recorded execution time.
Test data is divided into data 128Bytes, 512Bytes, 1K 10K and four different sizes.
Experiments show that: when the team, when performance data is relatively small Redis is higher than RabbitMQ, and if the data size exceeds 10K, Redis is slow can not stand; when the team, regardless of the data size, Redis have shown a very good performance , but the team performance RabbitMQ is much lower than Redis.
5.Kafka / Jafka
Kafka is a subproject of the Apache, is a high-performance cross-language distributed publish / subscribe message queuing systems, and Jafka is on Kafka hatch comes, that is an upgraded version of Kafka.
It has the following characteristics:
- Rapid persistence, the message may be persistent in O (1) of the overhead;
- High throughput, on a common server throughput rate can be achieved either 10W / s; a fully distributed system, Broker, Producer, Consumer have native support for distributed automatically, automatic load balancing;
- Supports Hadoop data loaded in parallel, the same as for off-line analysis of log data and Hadoop system, the restriction but requires real-time processing, this is a viable solution.
- Kafka through parallel loading mechanism of Hadoop unified message processing online and offline. Apache Kafka relative to ActiveMQ is a very lightweight messaging system, in addition to very good performance outside, or a good work distributed systems.
When do you need a message queue
When you need to use the message queue, the need for it to be considered first.
You can use mq scene there are many, the most common types:
- Decoupling of doing business
- The final consistency
- broadcast
- Peak load shifting flow control
On the other hand, if you need strong consistency, focus on business logic processing results, the RPC is more appropriate.
Message Queue usage scenarios
1. Decoupling
Decoupling is the message queue to solve the most essential issues. The so-called decoupling, simple point of speaking is a transaction, only interested in the core of the process. The need to rely on other systems, but not so important things have to notify, without waiting for the results. In other words, the model message-based care is the "notice", rather than "treatment."
As an example, on the order system, order the final payment may need to send a text message to the user after integrating what success, but in fact this is not the core processes of our system up.
If the external system lack pace (such as SMS gateway speed is not good), then the time will lengthen the main flow of a lot of users would not want to pay for a click to have a good few minutes to see the results. Then we only need to notify the SMS system, "we paid a success," do not have to wait for it to complete the process immediately.
2. The final consistency
Eventual consistency refers to the state of the two systems consistent, either succeed or fail.
Of course there is a time limit, in theory, the sooner the better, but in fact in the case of various exceptions, there may be some delay to reach a final status agreement, but the status of the last two systems are the same.
Some industry as "eventual consistency" born of the message queue, such as:
- Notify (Ali)
- QMQ (where to go), etc.
That was designed, highly reliable trading is to inform the system.
Transfer to a bank to understand the process of eventual consistency, transfer the demand is very simple, if A system of deducting money is successful, the system B plus money must succeed. Otherwise roll back together, like nothing had happened.
However, this process, there are many possible accident:
- A successful deducting money, call money plus B interfaces fail.
- A successful deducting money, call money plus B interfaces, although successful, but getting the final results of the network caused by abnormal timeout.
- A successful deducting money, B plus the money failed, A deduction would like to roll back the money, but the A machine down machine.
Visible, trying this seemingly simple thing really made of, really not so easy.
Consistency across all the VM, from a technical point of view common solution is:
- Strong consistency, distributed transactions, but landed too hard and too costly, later will be specifically mentioned.
- Eventual consistency, mainly with the "record" and "compensation" approach. Before doing all of the uncertainty of things, first thing recorded, and then do something uncertain, the result may be: success, failure, or uncertain, "uncertain" (such as timeouts, etc.) may be equivalent to failure . Success can record things to clean up, and for the failure and uncertainty, can rely on timed tasks, etc. to all things fail to re-engage again, until it succeeds.
- Back to the earlier example, deducting money A system in case of success, to give B "notice" it is recorded in the library (in order to guarantee the highest reliability of the system can notify the B deducting money plus money and success of these two thing maintained in a local transaction in), notice is successfully delete this record, notification fails or uncertain task is to rely on the timing of compensatory notify us until we put a status update to the correct date.
- The whole model is still based on RPC can do, but can be abstracted into a unified model, based on the message queue to do a "corporate bus."
- In particular, local transactions and maintain business change notification message, floor together (with the rollback fails), then RPC arrive broker, broker after a successful landing, RPC returns successfully, you can delete the local news. Otherwise, local news has been relying on timed tasks constantly polling retransmission, thus ensuring reliable message floor broker.
- broker to the consumer to send a message similar to the process, it has been sending the message until the consumer sends the consumer confirmation of success.
- Let's ignore the problem of duplicate messages, messages through two floor plus compensation is downstream will be able to receive messages. The state machine then rely on the version number, etc. to do the heavy sentence, update their business, to achieve the ultimate consistency.
Eventual consistency is not essential characteristics of the message queue, but it does rely on the consistency of the message queue to do the final thing.
In addition, all does not guarantee 100% of the messages are not lost message queue, in theory, can not achieve eventual consistency. Well, in theory, should be said that 100% troubleshoot system failures and severe bug.
Kafka like may design a class, have lost at the design level of the message (such as the timing of the brush plate, will be lost if power message). Even if only one-thousandth of lost messages, the business must also use other means to ensure the correct results.
2. Broadcast
One of the basic functions of the message queue is broadcast.
If there is no message queue, whenever a new business party access, we have a new FBI interface. With the message queue, we only need to be concerned about whether the message delivery queue, as to who would like to subscribe, downstream of things, no doubt greatly reduce the workload of the development and the FBI.
For example, this article mentioned at the beginning of Products announced product changes, as well as many attractions library to re-update messages may be "concerned" parties have a number, but the product center and attractions libraries only need to contribute to change the message, who cares who access.
3. The peak shifting and flow control
Imagine what downstream for processing capabilities are different.
For example, to withstand front-end Web tens of millions of requests per second, what amazing things not only need to get a bit machine, then build some of the LVS and Nginx, and load balancing equipment can be.
But the processing power of the database is very limited, even with SSD plus library sub-table, stand-alone processing power is still ten thousand . Due to cost considerations, we can not afford to dream of the number of machines to catch up with the front-end database.
This problem also exists between the system and the systems, such as may be due to the short messaging system board effect, on the gateway card speed (hundreds of times per request), with the tip of concurrency is not an order of magnitude.
However, users receive a text message about a half a minute at night, there is generally not much problem. If there is no message queue, nor that between the two systems can not be achieved by the Consultative complex sliding windows.
However, system complexity exponentially in the upstream or downstream bound do storage, a series of questions to be processed and timing congestion. And whenever there is processing capacity gap, we need to develop a separate set of logic to maintain this logic. Therefore, using the communication system dump the contents of an intermediate two systems, and when the downstream system has the ability to process these messages, and then process these messages, it is a relatively common method.
Message Queuing uses summary
1. The message queue is not a panacea for the required strength and to ensure delay sensitive transactions, is superior to the RPC message queue.
2. For some innocuous, or for someone else but for very important things that they are not so concerned, you can do it with a message queue.
3. The final consistency of the message queue support, can be used to process delay less sensitive to "distributed transaction" scene, and with respect to the bulky distributed transaction may be better handled.
4. When there is a gap on a downstream system capacity, using a generic message queue do "funnel", there is a time in downstream processing capacity, and then distributed.
5. If you have a lot of downstream system to inform you of the care system issued when the message queue to use it decisively.
This article from the blog article multiple platforms OpenWrite release!