Reprinted from: https://github.com/alibaba/RocketMQ/wiki/rmq_vs_kafka
Taobao's internal transaction system uses the Notify message middleware independently developed by Taobao, and uses MySQL as the message storage medium, which can be completely expanded horizontally. In order to further reduce costs, we believe that the storage part can be further optimized. At the beginning of 2011, Linkin open sourced Kafka, which is excellent Kafka's message middleware, after the Taobao middleware team has done a full review of Kafka, Kafka's infinite message accumulation and efficient persistence speed attracted us, but at the same time, we found that this message system is mainly located in log transmission, for use in Taobao transactions, There are still many features that are not satisfied in scenarios such as order and recharge. For this reason, we have re- written RocketMQ in Java language, aiming at reliable message transmission without logs (log scenarios are also OK). At present, RocketMQ is widely used in orders in Alibaba Group. Transaction, recharge, stream computing, message push, log stream processing, binglog distribution and other scenarios.
In order to facilitate everyone's selection, we have compiled a comparison document between RocketMQ and Kafka. If there is any error in the text, please write to me to correct it. [email protected]
data reliability
- RocketMQ supports asynchronous real-time flushing, synchronous flushing, synchronous replication, and asynchronous replication
- Kafka uses asynchronous brushing, asynchronous Replication
Summary: RocketMQ's synchronous flushing is more reliable than Kafka on a single machine, and will not cause data loss due to operating system crashes. At the same time, synchronous replication is also more reliable than Kafka asynchronous replication, and the data has no single point at all. In addition, Kafka's Replication is based on topic, which supports host downtime and automatic switching of standby machines. However, there is a problem here. Since it is asynchronous replication, data will be lost after switching. At the same time, if the leader restarts, it will be different from the existing leader. A data conflict occurs. The open source version of RocketMQ does not support the downtime of the Master, and the Slave automatically switches to the Master. The Alibaba Cloud version of RocketMQ supports the automatic switching feature.
Performance comparison
- Kafka's single-machine write to TPS is about one million per second, and the message size is 10 bytes
- RocketMQ single machine writes about 70,000 messages/second to TPS single instance, and deploys 3 brokers on a single machine, which can run up to 120,000 messages/second, and the message size is 10 bytes.
Summary: Kafka's TPS runs to one million on a single machine, mainly because the Producer side merges multiple small messages and sends them to the Broker in batches.
Why didn't RocketMQ do this?
- Producer usually uses Java language, cache too many messages, GC is a very serious problem
- The Producer calls the send message interface, but the message is not sent to the Broker, and returns success to the business. At this time, the Producer is down, which will result in message loss and business errors.
- Producer is usually a distributed system, and each machine is sent by multiple threads. We believe that the amount of data generated by a single Producer per second in an online system is limited, and it is impossible to tens of thousands.
- The function of caching can be completely completed by the upper-layer business.
Number of queues supported by a single machine
- If there are more than 64 queues/partitions in a single Kafka machine, the load will increase significantly. The more queues, the higher the load, and the longer the response time for sending messages.
- RocketMQ single machine supports up to 50,000 queues, and Load will not change significantly
What are the benefits of having more queues?
- A single machine can create more topics, because each topic is composed of a batch of queues
- The cluster size of the Consumer is proportional to the number of queues. The more queues, the larger the Consumer cluster can be.
消息投递实时性
- Kafka使用短轮询方式，实时性取决于轮询间隔时间
- RocketMQ使用长轮询，同Push方式实时性一致，消息的投递延时通常在几个毫秒。
消费失败重试
- Kafka消费失败不支持重试
- RocketMQ消费失败支持定时重试，每次重试间隔时间顺延
总结：例如充值类应用，当前时刻调用运营商网关，充值失败，可能是对方压力过多，稍后在调用就会成功，如支付宝到银行扣款也是类似需求。
这里的重试需要可靠的重试，即失败重试的消息不因为Consumer宕机导致丢失。
严格的消息顺序
- Kafka支持消息顺序，但是一台Broker宕机后，就会产生消息乱序
- RocketMQ支持严格的消息顺序，在顺序消息场景下，一台Broker宕机后，发送消息会失败，但是不会乱序
Mysql Binlog分发需要严格的消息顺序
定时消息
- Kafka不支持定时消息
- RocketMQ支持两类定时消息
- 开源版本RocketMQ仅支持定时Level
- 阿里云ONS支持定时Level，以及指定的毫秒级别的延时时间
分布式事务消息
- Kafka不支持分布式事务消息
- 阿里云ONS支持分布式定时消息，未来开源版本的RocketMQ也有计划支持分布式事务消息
消息查询
- Kafka不支持消息查询
- RocketMQ支持根据Message Id查询消息，也支持根据消息内容查询消息（发送消息时指定一个Message Key，任意字符串，例如指定为订单Id）
总结：消息查询对于定位消息丢失问题非常有帮助，例如某个订单处理失败，是消息没收到还是收到处理出错了。
消息回溯
- Kafka理论上可以按照Offset来回溯消息
- RocketMQ支持按照时间来回溯消息，精度毫秒，例如从一天之前的某时某分某秒开始重新消费消息
总结：典型业务场景如consumer做订单分析，但是由于程序逻辑或者依赖的系统发生故障等原因，导致今天消费的消息全部无效，需要重新从昨天零点开始消费，那么以时间为起点的消息重放功能对于业务非常有帮助。
消费并行度
-
The consumption parallelism of Kafka depends on the number of partitions configured by the topic. If the number of partitions is 10, then a maximum of 10 machines can be used for parallel consumption (each machine can only start one thread), or one machine consumes (10 threads consume in parallel). That is, the consumption parallelism is consistent with the number of partitions.
-
RocketMQ consumption parallelism is divided into two cases
- The parallelism of sequential consumption is exactly the same as that of Kafka
- The degree of parallelism in out-of-order mode depends on the number of threads of the Consumer. For example, if a topic is configured with 10 queues, 10 machines consume, and each machine has 100 threads, then the parallelism is 1000.
message track
- Kafka does not support message trails
- Alibaba Cloud ONS Support Message Track
Development language friendliness
- Kafka is written in Scala
- RocketMQ is written in Java language
Broker-side message filtering
- Kafka does not support message filtering on the broker side
- RocketMQ supports two broker-side message filtering methods
- Filter according to Message Tag, which is equivalent to the concept of sub-topic
- Uploading a piece of Java code to the server can filter messages in any form, and even filter and split the Message Body.
message stacking capability
In theory, Kafka has a stronger stacking capability than RocketMQ, but RocketMQ can also support a billion-level message stacking capability. We believe that this stacking capability can fully meet business needs.
Open source community activity
- Kafka community updates are slow
- RocketMQ's github community has 250 individual and company users who have registered contact information, more than 1,000 people.
business support
- Kafka's original development team established a new company, and there are currently no related products to see
- RocketMQ has been open for public beta testing on Alibaba Cloud for nearly half a year. It is currently available for commercial use in the form of cloud services for free. It promises users 99.99% reliability, and completely solves the operation and maintenance complexity of building MQ products for users.
maturity
- Kafka is relatively mature in the field of logging
- RocketMQ is used by a large number of applications within the Alibaba Group, generating massive messages every day, and has successfully supported many Tmall Double Eleven mass message tests, which is a powerful tool for data peaks and valleys.