Kafka or RocketMQ

Data reliability

• RocketMQ supports asynchronous real-time flashing, synchronous flashing, synchronous replication, and asynchronous replication. • Kafka uses asynchronous flashing, asynchronous replication / synchronous replication.
Summary: RocketMQ's synchronous flashing is more reliable than Kafka in stand-alone. It will cause data loss due to operating system Crash. Kafka Synchronous Replication theoretically has lower performance than RocketMQ's Synchronous Replication. The reason is that Kafka's data is organized in units of partitions, meaning that there will be hundreds of data partitions on a Kafka instance, there is only one data partition on a RocketMQ instance, RocketMQ You can make full use of the IO Group Commit mechanism to transfer data in batches. Compared with asynchronous Replication, the performance loss is about 20% to 30%. Kafka has not tested it personally, but personally thinks it will be lower than RocketMQ in theory.

Performance comparison

• Kafka stand-alone write TPS is about 1 million / second, message size is 10 bytes •
RocketMQ stand-alone write TPS single instance is about 70,000 / second, stand-alone deployment of 3 Broker can run up to 120,000 / Second, the message size is 10 bytes.
Summary: Kafka's TPS ran to a single machine million, mainly because the Producer side merged multiple small messages and sent them to Broker in batches.
Why didn't RocketMQ do this?

1. Producers usually use the Java language, cache too many messages, GC is a very serious problem
2. Producer calls the send message interface, the message is not sent to the broker, and returns success to the business. At this time, the producer is down, which will cause the loss of the message and the business error
3. Producer is usually a distributed system, and each machine is multi-threaded. We believe that the online system produces a single Producer with a limited amount of data per second, which cannot be tens of thousands.
4. The function of caching can be completed by upper-layer services. Number of queues supported by a single machine • If Kafka has more than 64 queues / partitions on a single machine, the load will obviously soar. The more queues and the higher the load, the longer the response time for sending messages. The number of Kafka partitions cannot be too large
   . RocketMQ single machine supports up to 50,000 queues, and the load will not change significantly. What are the benefits of more queues?
5. A single machine can create more topics, because each topic is composed of a batch of queues
6. The size of the consumer ’s cluster is proportional to the number of queues. The more queues, the larger the consumer cluster.

Real-time message delivery

• Kafka uses short polling, and its real-time performance depends on the polling interval. Longer polls are supported in versions after 0.8. •
RocketMQ uses long polling, which is consistent with the Push method in real time, and the delivery delay of messages is usually a few milliseconds. Retry of failed consumption • Failure of Kafka consumption does not support retry. •
RocketMQ consumer fails to support regular retry, retry each interval extended summary: top-class applications such as the current time calling gateway operators, prepaid fails, pressure may be other
excessive force, and then later call will be successful, such as Alipay to bank deductions are similar needs. The retry here requires a reliable retry, that is, the message of failed retry is not lost due to Consumer downtime.

Strict message order

• Kafka supports message order, but after an agent goes down, messages will be out of order. •
RocketMQ supports strict message order. In a sequential message scenario, after a Broker goes down, sending messages will fail, but it will not. will be out of order
MySQL binary log distribution requires strict message ordering

Timed message

• Kafka does not support timed messages. • RocketMQ supports two types of timed messages. ○ The open source version of RocketMQ only supports timed levels, which can be customized by timed users. ○ Alibaba
Cloud MQ specifies millisecond-level delay time distributed transaction messages. • Kafka does not Support distributed transaction messages •
Alibaba Cloud MQ supports distributed transaction messages. In the future, the open source version of RocketMQ also has plans to support distributed transaction message query. • Kafka does not support message query. •
RocketMQ supports querying messages based on message identifiers, and also supports according to messages. Content query message (specify a message key when sending a message, any string, such as an order number)
Summary: Message query is very helpful to locate the problem of message loss, such as an order processing failure, whether the message was not received or received There was an error in processing. • messages back
Kafka could theoretically be offset in accordance with backtracking messages • RocketMQ support back in time to the message, the accuracy of milliseconds, for example, a minute of a second re-start when a consumer from one day before the news
summary: typical business scenarios such as consumer do Order analysis, but due to program logic or dependent system failures and other reasons, all the messages consumed today are invalid and need to be re-started from zero yesterday, then the message replay function starting from time is very helpful to the business.

Consumption parallelism

•
Kafka's consumption parallelism depends on the number of partitions configured by the topic. For example, if the number of partitions is 10, up to 10 machines can be used for parallel consumption (each machine can only open one thread), or one machine (10 threads for parallel consumption) . That is, the consumption parallelism and the number of partitions are the same.
• RocketMQ consumption parallelism is divided into two cases. ○ The parallelism of the sequential consumption mode is exactly the same as that of Kafka. ○ The
parallelism of the out-of-order mode depends on the number of threads of the Consumer. Threads, then the parallelism is 1000.

Message track

• Kafka does not support message trajectory •
Alibaba Cloud MQ supports message trajectory development language friendliness • Kafka is written in Scala • RocketMQ uses Java language to write broker-side message filtering • Kafka does not support agent-side message filtering • RocketMQ Supports two agent-side message filtering methods
○ Filtering based on message variables, equivalent to the concept of subtopics ○
Uploading a piece of Java code to the server, you can do any form of filtering on the message, and even filter and split the Message body. Message stacking capacity
Theoretically Kafka is stronger than RocketMQ's stacking capacity, but RocketMQ stand-alone can also support billion-level message stacking capacity. We believe that this stacking capacity can fully meet business needs.
Open source community activity • The Kafka community is slow to update •
RocketMQ ’s GitHub community has 250 individuals, company users have registered contact information, and the QQ group has more than 1,000 people. MQ ### Commercial Support •
Kafka ’s original development team established a new company, and there are currently no related products to see. •
RocketMQ has been commercialized in Alibaba Cloud. It is currently available for commercial use in the form of cloud services and promises 99.99% reliability to users At the same time, it completely solves the problem of operation and maintenance complexity for users to build MQ products.
Maturity • Kafka is relatively mature in the field of logging •
RocketMQ has a large number of applications in the Alibaba Group, which generates a lot of messages every day and is smooth Supported multiple Tmall double eleven massive messages test, is a weapon for data peaking and valley filling

From http://jm.taobao.org/2016/03/24/rmq-vs-kafka/