1. Introduction
ActiveMQ：
ActiveMQ is an open source message broker developed by the Apache Software Foundation based on the Java language and implements JMS.
RabbitMQ：
RabbitMQ is an open source message broker software (also known as message-oriented middleware) that implements the Advanced Message Queuing Protocol (AMQP). The RabbitMQ server is written in the Erlang language, while clustering and failover are built on the Open Telecommunications Platform framework. All major programming languages have client libraries that communicate with proxy interfaces. RabbitMQ supports multiple messaging protocols, delivery confirmations and other features.
Kafka：
Apache Kafka is an open source messaging system project developed by the Apache Software Foundation and written in Scala.
Kafka was originally developed by LinkedIn and open sourced in early 2011. Graduated from Apache Incubator in October 2012.
The goal of this project is to provide a unified, high-throughput, low-latency platform for processing real-time data. Kafka is a distributed, partitioned, multi-replica log submission service. It provides the functionality of a messaging system through a unique design.
RocketMQ：
Apache RocketMQ is a distributed messaging and streaming platform with low latency, strong consistency, high performance and reliability, terascale capacity and flexible scalability. It borrows design ideas from Kafka, but it is not a copy of Kafka. Implemented based on JMS.
Pulsar：
Apache Pulsar is the top project of the Apache Software Foundation. It is a next-generation cloud-native distributed message flow platform that integrates messaging, storage, and lightweight functional computing. It is designed with a separation of computing and storage architecture. It supports multi-tenancy, persistent storage, and multi-machine room cross-region data replication. It has streaming data storage characteristics such as strong consistency, high throughput, low latency, and high scalability. It is regarded as the real-time message streaming transmission and storage in the cloud native era. and calculate the optimal solution.
For more detailed introduction, please read the document: " Comparison of RabbitMQ, RocketMQ, Kafka, and Pulsar message queues "
2. Function and performance comparison
- consumption push-pull model
The way client consumers obtain messages, Kafka and RocketMQ pull messages through long polling Pull, RabbitMQ, Pulsar, and NSQ all use Push.
The pull type message queue is more suitable for high-throughput scenarios, allowing consumers to control their own flow and obtain messages based on their actual consumption capabilities. The push type message queue has better real-time performance, but it needs a good flow control strategy (backpressure). When the consumer's consumption capacity is insufficient, the number of push consumption can be reduced to avoid overwhelming the consumer.
- delay queue
Delayed delivery of messages. When a message is generated and delivered to the message queue, some business scenarios do not want consumers to receive the message immediately, but wait for a specific period of time before the consumer can get the message for consumption. Delay queues are generally divided into two types, message-based delay and queue-based delay. Message-based delay refers to setting a different delay time for each message. When new messages enter the queue, they are sorted according to the delay time. Of course, this will have a greater impact on performance. Another kind of queue-based delay refers to setting up queues with different delay levels. The delay time of each message in the queue is the same, which avoids the loss of performance caused by sorting based on delay time. Through a certain scanning strategy The timed-out message can be delivered.
Usage scenarios for delayed messages include anomaly detection retries, order timeout cancellation, etc., for example:
- If the service request is abnormal, the abnormal request needs to be placed in a separate queue and retried after 5 minutes;
- The user purchases goods but has not paid yet. The user needs to be reminded to pay regularly and the order will be closed when the timeout expires;
- For an interview or meeting appointment, send a notification to remind you again half an hour before the interview or meeting starts.
Kafka does not support delayed messages. Pulsar supports second-level delayed messages. All delayed delivery messages will be recorded by the Delayed Message Tracker with the corresponding index. When the consumer consumes, it will first go to the Delayed Message Tracker to check whether there are any expired messages that need to be delivered. If there are any expired messages, message, take out the corresponding index from the Tracker, find the corresponding message and consume it. If there is no expired message, consume the normal message directly. For long-term delay messages, they will be stored on disk and loaded into memory when the delay interval is approaching.
RocketMQ open source version delayed messages are temporarily stored in an internal topic, do not support arbitrary time precision, and support specific levels, such as timing 5s, 10s, 1m, etc.
RabbitMQ needs to install a rabbitmq_delayed_message_exchange plug-in.
NSQ saves delayed messages through an in-memory priority queue, supporting second-level precision and up to 2 hours of delay.
- dead letter queue
Due to some reasons, the message cannot be delivered correctly. In order to ensure that the message will not be discarded for no reason, it is generally placed in a queue with a special role. This queue is generally called a dead letter queue. Corresponding to this, there is also the concept of "rollback queue". Imagine that if an exception occurs when the consumer consumes, then the consumption will not be confirmed (Ack), and the message will always be unavailable after the operation of rolling back the message. will be placed at the top of the queue and then continuously processed and rolled back, causing the queue to fall into an infinite loop. In order to solve this problem, you can set up a rollback queue for each queue. Both it and the dead letter queue provide a mechanism guarantee for exception handling. In actual situations, the role of the rollback queue can be played by the dead letter queue and the retry queue.
Kafka does not have a dead letter queue and records the offset of the current consumption through Offset.
Pulsar has a retry mechanism. When some messages are consumed by consumers for the first time and do not receive a normal response, they will enter the retry topic. When the retries reach a certain number of times, they will stop retrying and deliver to the dead letter topic. middle.
RocketMQ uses DLQ to record all consumption failure messages.
RabbitMQ implements a dead letter queue in a form similar to a delay queue.
NSQ does not have a dead letter queue.
- priority queue
The priority queue is different from the first-in-first-out queue. Messages with high priority have the privilege of being consumed first, which can provide downstream guarantees of different message levels. However, this priority also needs a premise: if the consumer's consumption speed is greater than the producer's speed, and there is no message accumulation in the message middleware server (generally simply called the Broker), then set the priority for the sent messages There is no substantive meaning, because a message is consumed by the consumer as soon as the producer sends it, which means that there is at most one message in the Broker, and the priority is meaningless for a single message.
Kafka, RocketMQ, Pulsar, and NSQ do not support priority queues, and message priority can be achieved through different queues.
RabbitMQ supports priority messages.
- Message traceback
Generally, messages are processed after consumption is completed, and the message can no longer be consumed. Message backtracking is just the opposite. It means that after the message is consumed, the previously consumed message can still be consumed. For messages, a common problem is "message loss". It is generally difficult to trace whether it is really lost due to defects in the message middleware or due to misuse by the user. If the message middleware itself has a message traceback function, You can reproduce the "lost" messages through backtracing consumption to find out the source of the problem. The role of message backtracking goes far beyond this, such as index recovery and local cache reconstruction. Some business compensation solutions can also be implemented through backtracking.
Kafka supports message backtracking and can reset the Consumer's Offset based on the timestamp or specified Offset so that it can be consumed repeatedly.
Pulsar supports backtracking messages by time.
RocketMQ supports backtracking by time, and the implementation principle is consistent with Kafka.
RabbitMQ does not support backtracking, messages will be marked for deletion once marked for confirmation.
NSQ general messages are not traceable, but you can use the nsq_to_file tool to write messages to a file and then replay the messages from the file.
- Message persistence
Traffic peak shaving is a very important function of message middleware, and this function actually benefits from its message accumulation capability. In a sense, if a message middleware does not have the ability to accumulate messages, then it cannot be regarded as a qualified message middleware. Message accumulation is divided into memory accumulation and disk accumulation. Generally speaking, the capacity of the disk will be much larger than the capacity of the memory. For disk-type stacking, the stacking capacity is the size of the entire disk. From another perspective, message accumulation also provides redundant storage functions for message middleware.
Kafka and RocketMQ directly flush messages into disk files for persistence, and all messages are stored on disk. As long as the disk capacity is sufficient, unlimited message accumulation can be achieved.
RabbitMQ is a typical in-memory stack, but this is not absolute. After certain conditions are triggered, there will be a paging action to page the messages in the memory to the disk (the paging action will affect throughput), or directly use a lazy queue to page the messages in the memory to the disk. Messages are persisted directly to disk.
Pulsar messages are stored on the BookKeeper storage cluster and are also disk files.
NSQ writes messages to files through the nsq_to_file tool.
- Message confirmation mechanism
The message queue needs to manage the consumption progress and confirm whether the consumer successfully processes the message. The message queue component using the push method often confirms a single message. For unconfirmed messages, delayed re-delivery or entry into the dead letter queue is performed.
Kafka confirms the message through Offset.
RocketMQ, like Kafka, also submits Offset. The difference is that consumers can mark messages that fail to consume as message consumption failures. The Broker will retry delivery. If multiple consumption failures accumulate, they will be delivered to the dead letter queue.
RabbitMQ is similar to NSQ. The consumer confirms a single message, otherwise it will be put back in the queue to wait for the next delivery.
Pulsar uses specialized Cursor management. Cumulative confirmation has the same effect as Kafka; single or selective confirmation is provided.
- Message TTL
Message TTL indicates the survival time of a message. If no consumer consumes the message within the TTL time after the message is sent, the message queue will delete the message or put it into the dead letter queue.
Kafka deletes messages based on the set retention period. It is possible that the message was not consumed and was deleted after expiration. TTL is not supported.
Pulsar supports TTL, if the message is not consumed by any consumer within the configured TTL period, the message will be automatically marked as acknowledged. The difference between message retention period and message TTL is that message retention period applies to messages marked as acknowledged and set to deleted, while TTL applies to messages that are not acked. TTL in Pulsar is illustrated in the legend above. For example, if subscription B has no active consumers, message M10 will be automatically marked as acknowledged after the configured TTL period has elapsed, even though no consumer actually read the message.
RocketMQ has relatively little information on message TTL, but the interface seems to support it.
RabbitMQ has two methods. One is to set it in the queue properties when declaring the queue. The messages in the entire queue have the same validity period. You can also set attributes for the message when sending the message, and set a different TTL for each message.
NSQ does not seem to be supported yet, and there is an Issue of Feature Request in the Open state.
- Multi-tenant isolation
Multi-tenancy refers to the ability to serve multiple tenants through a single software instance. A tenant is a group of users who have the same "view" of the system. In systems that do not support multi-tenancy, it is often necessary to create multiple message queue instances for different users or different clusters to achieve physical isolation, which will bring high operation and maintenance costs. As an enterprise-level messaging system, Pulsar's multi-tenant capabilities are designed to meet the following requirements:
- Ensure that stringent SLAs can be met smoothly.
- Ensure isolation between different tenants.
- Enforce quotas on resource utilization.
- Provides per-tenant and system-level security.
- Ensure low-cost operation and maintenance and simple management as possible.
Pulsar meets the above needs in the following ways:
- Get the security you need with authentication, authorization, and ACLs (Access Control Lists) for each tenant.
- Enforce storage quotas per tenant.
All isolation mechanisms are defined in the form of policies, which can be changed during operation, thereby reducing operation and maintenance costs and simplifying management.
- message sequence
Message sequentiality refers to ensuring that messages are in order. The order of message consumption is consistent with the order of production.
Kafka ensures that messages within the partition are ordered.
Pulsar supports two consumption modes. The exclusive subscription streaming mode only guarantees the order of messages, and the shared subscription queue model does not guarantee the order.
RocketMQ needs to use locks to ensure that a queue has only one consumer thread consuming at the same time to ensure the orderliness of messages.
RabbitMQ has strict sequential conditions, requiring single-threaded sending and single-threaded consumption, and does not use advanced functions such as delay queues and priority queues.
NSQ uses golang's own case/select to implement message distribution. It does not provide ordering guarantees, cannot match characteristic messages to consumers, and cannot achieve message ordering.
- Message query
In actual development, it is often necessary to check the content of messages in MQ, such as querying the specific messages of MQ through a certain MessageKey/ID. Or you can track the link of the message to know where the message comes from and where it is sent, so you can quickly troubleshoot and locate the problem.
The Kafka storage layer is implemented in the form of a distributed commit log, and each write operation is sequentially appended to the end of the log. Reading is also sequential reading. Search function is not supported.
Pulsar can query the message content, message parameters and message trajectory of a specific message through the message ID.
RocketMQ supports querying messages by Message Key, Unique Key, and Message Id.
RabbitMQ uses an index-based storage system. These save the data in a tree structure to provide the quick access needed to confirm individual messages. Since RabbitMQ messages will be deleted after confirmation, only unconfirmed messages can be queried
NSQ itself does not support message persistence and message retrieval, but you can use tools such as nsq_to_http to write messages to a storage that supports indexing.
- consumption pattern
Kafka has two consumption modes, which will ultimately ensure that only one consumer consumes a partition:
- Subscribe method: When the number of topic partitions or the number of consumers changes, rebalance will be performed; by registering a rebalance listener, you can manually manage the offset without registering a listener, and Kafka will automatically manage it.
- Assign method: Manually map the consumer to the partition, and Kafka will not perform rebanlance.
Pulsar has the following four consumption modes. The exclusive mode and disaster recovery mode are similar to Kafka. They are stream models. Each partition has only one consumer consumption, which can ensure the orderliness of messages. Sharing mode and Key sharing mode are queue models. Multiple consumers can increase consumption speed, but cannot guarantee orderliness.
- Exclusive exclusive mode (default mode): A Subscription can only be associated with one Consumer. Only this Consumer can receive all messages from the Topic. If the Consumer fails, consumption will stop.
- Disaster recovery mode (Failover): When there are multiple consumers, they will be sorted in dictionary order, and the first consumer is initialized as the only consumer to receive messages. When the first consumer disconnects, all messages (unacknowledged and subsequently incoming) will be distributed to the next consumer in the queue.
- Shared mode (Shared): Messages are distributed to different consumers through a round robin polling mechanism (which can also be customized), and each message will only be distributed to one consumer. When a consumer disconnects, all unacknowledged messages sent to it will be rescheduled and distributed to other surviving consumers.
- KEY sharing mode (Key_Shared): When there are multiple consumers, the messages will be distributed according to the key. Messages with the same key will only be distributed to the same consumer.
RocketMQ has two consumption modes, BROADCASTING broadcast mode and CLUSTERING cluster mode.
Broadcast consumption refers to: a message is consumed by multiple consumers. Even if these consumers belong to the same ConsumerGroup, the message will be consumed once by each Consumer in the ConsumerGroup. The concept of ConsumerGroup in broadcast consumption can be considered meaningless in terms of message division.
Cluster consumption mode: Consumer instances in a ConsumerGroup evenly share consumption messages. For example, a Topic has 9 messages, and one of the ConsumerGroup has 3 instances (maybe 3 processes, or 3 machines), then each instance only consumes part of them, and the consumed messages cannot be consumed by other instances.
The consumption of RabbitMQ and NSQ is similar. They are similar to the Pulsar sharing mode. In the form of a queue, increasing the number of consumers in a consumer group can increase the consumption speed.
- message reliability
Message loss is a common problem that you have to face when using message middleware. Behind it, message reliability is also a key factor in measuring the quality of message middleware. Especially in the field of financial payments, message reliability is particularly important. For example, when a service fails, will some messages that have been successfully produced by the producer be lost during high availability switching? Synchronous flushing is an effective way to enhance the reliability of a component, and message middleware is no exception. Both Kafka and RabbitMQ can support synchronous flushing, but in most cases, the reliability of a component should not be determined by synchronous flushing. Instead of using extremely performance-consuming operations to ensure it, a multi-copy mechanism is used to ensure it.
Kafka can set the reliability level by configuring the request.required.acks parameter, which indicates how many copies of a message have to confirm successful reception before it is successfully sent by the task.
- request.required.acks=-1 (Full synchronous confirmation, strong reliability guarantee)
- request.required.acks=1 (leader confirms receipt, default)
- request.required.acks=0 (no confirmation, but high throughput)
Pulsar has a concept similar to Kafka, called Ack Quorum Size (Qa). Qa is the number of Bookies that need to be confirmed by a reply after each write request is sent. The larger the value, the longer it takes to confirm that the write is successful. Its value The upper limit is the number of copies Qw. For consistency, Qa should be (Qw+1)/2 or more, that is, to ensure data security, the lower limit of Qa is (Qw+1)/2.
RocketMQ is similar to Kafka.
RabbitMQ is a master-slave architecture that implements multiple copies and strong consistency semantics through mirrored ring queues. Multiple copies can ensure that after the master node goes down abnormally, the slave can be promoted as the new master and continue to provide services to ensure availability.
NSQ will store messages into local files through the go-diskqueue component, and control the queue size in memory through the mem-queue-size parameter. If mem-queue-size=0, each message will be stored on the disk, and there is no need to worry about node restarts. resulting in message loss. However, since it is stored in the local disk, if the node goes offline, the messages accumulated in the node disk will be lost.