Why use Message Queue
- Decoupling
It is extremely difficult to predict what requirements the project will encounter in the future at the beginning of the project. The message queue inserts an implicit, data-based interface layer in the middle of the process, and both processes must implement this interface. This allows you to extend or modify both processes independently, as long as you make sure they adhere to the same interface constraints - Redundancy
Sometimes processing fails when processing data. Unless the data is persisted, it will be lost forever. Message queues avoid the risk of data loss by persisting data until they have been fully processed. In the "insert-get-delete" paradigm used by many message queues, before removing a message from the queue, your processing needs to explicitly indicate that the message has been processed, ensuring that your data is safe Save it until you are done using it. - Scalability
Because message queues decouple your processing, it is easy to increase the frequency of message enqueuing and processing; just add additional processing. No need to change the code, no need to adjust the parameters. Expansion is as easy as turning up the power button. - Flexibility & Peak Processing Capability
In the case of a surge in traffic, the application still needs to continue to function, but such burst traffic is not common; Huge waste. Using message queues enables critical components to withstand increased access pressure rather than crashing completely due to overloaded requests. - Recoverability
When a part of the system fails, it does not affect the entire system. Message queues reduce the coupling between processes, so even if a process processing a message hangs, messages added to the queue can still be processed after the system is restored. And this ability to allow retries or defer processing of requests is often the difference between a slightly inconvenient user and a downright frustrated user. - Delivery Guarantees
The redundancy mechanism provided by message queues ensures that messages are actually processed as long as a process reads the queue. On top of this, IronMQ provides a "only once" guarantee. Each message can only be processed once, no matter how many processes are fetching data from the queue. This is possible because fetching a message simply "books" the message, temporarily removing it from the queue. Unless the client explicitly indicates that it has finished processing the message, the message is put back on the queue and can be processed again after a configurable amount of time.
- Order Guarantees
In many cases, the order in which data is processed is important. Message queues are inherently ordered, and can guarantee that data will be processed in a specific order. IronMO guarantees that message slurries are processed in FIFO (first-in-first-out) order, so the position of messages in the queue is where they are retrieved from the queue. - Buffering
In any critical system, there will be elements that require different processing times. For example, loading an image takes less time than applying filters. Message queues help tasks perform most efficiently through a buffer layer—processing writes to the queue will be as fast as possible, not constrained by the preparatory processing of reads from the queue. This buffering helps control and optimize the speed at which data flows through the system. - Understanding the flow of data
in a distributed system can be a huge challenge to get an overall impression of how long user actions take and why. The message series conveniently aids in identifying underperforming processes or areas by how often messages are processed, where the data flow is not optimized enough. - Asynchronous communication
Many times, you don't want or need to process the message immediately. Message queues provide asynchronous processing mechanisms that allow you to put a message on the queue, but not process it immediately. You can put as many messages into the queue as you want, and then process them when you want.
Common Message Queue Comparison
- RabbitMQ
RabbitMQ是使用Erlang编写的一个开源的消息队列，本身支持很多的协议：AMQP，XMPP, SMTP, STOMP，也正因如此，它非常重量级，更适合于企业级的开发。同时实现了Broker构架，这意味着消息在发送给客户端时先在中心队列排队。对路由，负载均衡或者数据持久化都有很好的支持。 - Redis
Redis是一个基于Key-Value对的NoSQL数据库，开发维护很活跃。虽然它是一个Key-Value数据库存储系统，但它本身支持MQ功能，所以完全可以当做一个轻量级的队列服务来使用。对于RabbitMQ和Redis的入队和出队操作，各执行100万次，每10万次记录一次执行时间。测试数据分为128Bytes、512Bytes、1K和10K四个不同大小的数据。实验表明：入队时，当数据比较小时Redis的性能要高于RabbitMQ，而如果数据大小超过了10K，Redis则慢的无法忍受；出队时，无论数据大小，Redis都表现出非常好的性能，而RabbitMQ的出队性能则远低于Redis。 - ZeroMQ
ZeroMQ号称最快的消息队列系统，尤其针对大吞吐量的需求场景。ZMQ能够实现RabbitMQ不擅长的高级/复杂的队列，但是开发人员需要自己组合多种技术框架，技术上的复杂度是对这MQ能够应用成功的挑战。ZeroMQ具有一个独特的非中间件的模式，你不需要安装和运行一个消息服务器或中间件，因为你的应用程序将扮演了这个服务角色。你只需要简单的引用ZeroMQ程序库，可以使用NuGet安装，然后你就可以愉快的在应用程序之间发送消息了。但是ZeroMQ仅提供非持久性的队列，也就是说如果down机，数据将会丢失。其中，Twitter的Storm中默认使用ZeroMQ作为数据流的传输。 - ActiveMQ
ActiveMQ是Apache下的一个子项目。 类似于ZeroMQ，它能够以代理人和点对点的技术实现队列。同时类似于RabbitMQ，它少量代码就可以高效地实现高级应用场景。 - Kafka/Jafka
Kafka是Apache下的一个子项目，是一个高性能跨语言分布式Publish/Subscribe消息队列系统，而Jafka是在Kafka之上孵化而来的，即Kafka的一个升级版。具有以下特性：快速持久化，可以在O(1)的系统开销下进行消息持久化；高吞吐，在一台普通的服务器上既可以达到10W/s的吞吐速率；完全的分布式系统，Broker、Producer、Consumer都原生自动支持分布式，自动实现复杂均衡；支持Hadoop数据并行加载，对于像Hadoop的一样的日志数据和离线分析系统，但又要求实时处理的限制，这是一个可行的解决方案。Kafka通过Hadoop的并行加载机制来统一了在线和离线的消息处理，这一点也是本课题所研究系统所看重的。Apache Kafka相对于ActiveMQ是一个非常轻量级的消息系统，除了性能非常好之外，还是一个工作良好的分布式系统。
Kafka简介
Kafka是一种分布式的，基于发布/订阅的消息系统。主要设计目标如下：
- 以时间复杂度为O(1)的方式提供消息持久化能力，并保证即使对TB级以上数据也能保证常数时间的访问性能
- 高吞吐率。即使在非常廉价的商用机器上也能做到单机支持每秒100K条消息的传输
- 支持Kafka Server间的消息分区，及分布式消息消费，同时保证每个partition内的消息顺序传输
- 同时支持离线数据处理和实时数据处理
引自：http://blog.csdn.net/allthesametome/article/details/47362451