quote
Taobao's internal transaction system uses the Notify message middleware independently developed by Taobao, and uses MySQL as the message storage medium, which can be fully expanded horizontally. In order to further reduce costs, we believe that the storage part can be further optimized. At the beginning of 2011, Linkin open sourced Kafka, which is excellent Kafka's message middleware, after the Taobao middleware team has done a full review of Kafka, Kafka's infinite message accumulation and efficient persistence speed attracted us, but at the same time, we found that this message system is mainly located in log transmission, for use in Taobao transactions, There are still many features that are not satisfied in scenarios such as order and recharge. For this reason, we have re-written RocketMQ in Java language, aiming at reliable message transmission without logs (log scenarios are also OK). At present, RocketMQ is widely used in orders in Alibaba Group. Transaction, recharge, stream computing, message push, log stream processing, binglog distribution and other scenarios.
In order to facilitate everyone's selection, we have compiled a comparison document between RocketMQ and Kafka. If there is any error in the text, please write to me to correct it. [email protected]
Data Reliability
RocketMQ supports asynchronous real-time flushing, synchronous flushing, synchronous replication, and asynchronous replication
Kafka uses asynchronous flushing, asynchronous replication
Summary: RocketMQ's synchronous flushing is more reliable than Kafka on a single machine High, data will not be lost due to operating system crash. At the same time, synchronous replication is also more reliable than Kafka asynchronous replication, and the data has no single point at all. In addition, Kafka's Replication is based on topic, which supports host downtime and automatic switching of standby machines. However, there is a problem here. Since it is asynchronous replication, data will be lost after switching. At the same time, if the leader restarts, it will be different from the existing leader. A data conflict occurs. The open source version of RocketMQ does not support the downtime of the Master, and the Slave automatically switches to the Master. The Alibaba Cloud version of RocketMQ supports the automatic switching feature.
Performance comparison
Kafka single-machine writing to TPS is about one million pieces/second, and the message size is 10 bytes
. RocketMQ single-machine writing to TPS single-instance is about 70,000 pieces/second. With 3 brokers deployed on a single machine, it can run up to 120,000 pieces/second. , the message size is 10 bytes.
Summary : Kafka's TPS runs to one million on a single machine, mainly because the Producer side merges multiple small messages and sends them to the Broker in batches.
Why didn't RocketMQ do this?
Producer usually uses Java language, cache too many messages, GC is a very serious problem
The Producer calls the send message interface, but the message is not sent to the Broker, and returns success to the business. At this time, the Producer is down, which will lead to message loss and business errors. The
Producer is usually a distributed system, and each machine is sent by multiple threads. We think that the line On the system above, the amount of data generated by a single Producer per second is limited, and it cannot be tens of thousands.
The function of caching can be completely completed by the upper-layer business.
The number of queues supported by a
single machine Kafka single machine exceeds 64 queues/partitions, and the load will increase significantly. The more queues, the higher the load, and the longer the response time for sending messages. RocketMQ single machine supports up to 50,000 queues, and the load will not occur. What is the benefit of significantly changing the queue?
A single machine can create more topics, because each topic is composed of a batch of queues. The cluster size of the Consumer is proportional to the number of queues. The more queues, the larger the Consumer cluster can be. Real-time
message
delivery Kafka uses the short polling method, which is real-time Depending on the polling interval
, RocketMQ uses long polling, which is consistent with the real-time performance of the Push method. The message delivery delay is usually several milliseconds.
Consumption failure Retry
Kafka consumption failure Does not support retry
RocketMQ consumption failure Supports scheduled retry, and the interval between each retry is extended.
Summary : For example, a recharge application calls the operator gateway at the current moment, and the recharge fails, which may be due to excessive pressure on the other party. The call will be successful later, such as Alipay debiting the bank is also a similar requirement.
The retry here requires reliable retry, that is, the message of the failed retry is not lost because the Consumer is down.
Strict message order
Kafka supports message order, but when a Broker goes down, messages will be out of order
RocketMQ supports strict message order. In the sequential message scenario, after a Broker goes down, sending messages will fail, but it will not be out of order. Mysql Binlog distribution requires strict message order.
Timing
messages Kafka does not support timing messages
. RocketMQ supports two types of timing. The open-source version of RocketMQ only supports timing levels. Alibaba Cloud ONS supports timing levels and the specified millisecond-level delay time.
Distributed
transaction messages Kafka does not support distributed transaction
messages . Alibaba Cloud ONS supports distributed timing messages, and the future open source version of RocketMQ also has The plan supports distributed transaction messages
Message
query Kafka does not support message query
RocketMQ supports querying messages based on Message Id, and also supports querying messages based on message content (specify a Message Key when sending a message, an arbitrary string, such as an order Id)
Summary: message Querying is very helpful for locating message loss issues, such as an order that failed to process, whether the message was not received or was received and processed incorrectly.
Message backtracking
Kafka can theoretically backtrack messages according to Offset.
RocketMQ supports backtracking messages according to time, with a precision of milliseconds. For example, re-consume messages from a certain time, minute and second a day ago.
Summary : A typical business scenario is such as a consumer doing order analysis, but due to The program logic or the failure of the dependent system causes all the messages consumed today to be invalid, and needs to be consumed again from 0:00 yesterday. The time-based message replay function is very helpful for business.
Consumption Parallelism
The consumption parallelism of Kafka depends on the number of partitions configured by the topic. If the number of partitions is 10, then a maximum of 10 machines can be used for parallel consumption (each machine can only start one thread), or one machine consumes (10 threads consume in parallel). That is, the consumption parallelism is consistent with the number of partitions.
The parallelism degree of RocketMQ consumption is divided into two cases. The parallelism degree of
sequential consumption mode is exactly the same as that of Kafka
. The parallelism degree of out-of-order mode depends on the number of threads of the Consumer. The degree is 1000.
Message track
Kafka does not support message
track Alibaba Cloud ONS supports message track
Development language friendliness
Kafka uses Scala to write
RocketMQ uses Java language to write
broker-side message filtering
Kafka does not support broker-side message filtering
RocketMQ supports two broker-side message filtering methods
According to Message Tag Filtering is equivalent to the concept of sub-topic. Uploading a piece of Java code
to the server can filter messages in any form, and even filter and split Message Body.
In theory, Kafka's message stacking capability
is stronger than RocketMQ's stacking capability. However, RocketMQ can also support 100 million-level message stacking capabilities. We believe that this stacking capability can fully meet business needs.
Open source community activity
Kafka community is slow to update
RocketMQ's github community has 250 individuals and company users who have registered their contact information, and more than 1,000 Q and Q groups.
business support
The original development team of Kafka established a new company. At present, there are no related products.
RocketMQ has been open for public testing on Alibaba Cloud for nearly half a year. Currently, it is free for everyone to use as a cloud service for commercial use, and promises users 99.99% reliability. At the same time, it completely solves the problem. The complexity of operation and maintenance of MQ products built by users themselves
Maturity
Kafka is relatively mature in the field of logs
RocketMQ is used in a large number of applications within the Alibaba Group, generating massive messages every day, and has successfully supported multiple Tmall Double Eleven massive amounts The news test is a powerful tool to cut peaks and fill valleys in data.