introduction
In distributed systems, we widely use message middleware to exchange data between systems to facilitate asynchronous decoupling. There are a lot of open source message middleware now. Some time ago, our own product RocketMQ (the core of MetaQ) was also open sourced smoothly and got everyone's attention.
So, which one has the strongest performance of message middleware?
With this question in mind, our middleware test group compared the performance of three common message products (Kafka, RabbitMQ, RocketMQ).
Kafka is LinkedIn's open source distributed publish-subscribe messaging system, which is currently part of the Apache rating project. The main feature of Kafka is that it processes message consumption based on the Pull mode and pursues high throughput. The initial purpose is to use for log collection and transmission. Version 0.8 starts to support replication, but does not support transactions, and there are no strict requirements for message duplication, loss, or error. It is suitable for data collection services of Internet services that generate large amounts of data.
RabbitMQ is an open source message queue system developed using Erlang language and implemented based on AMQP protocol. The main features of AMQP are message-oriented, queuing, routing (including point-to-point and publish/subscribe), reliability, and security. The AMQP protocol is more used in enterprise systems. In scenarios that require high data consistency, stability, and reliability, performance and throughput requirements are second.
RocketMQ is Alibaba's open source message middleware. It is developed in pure Java and features high throughput, high availability, and suitable for large-scale distributed system applications. The idea of RocketMQ originated from Kafka, but it is not a copy of Kafka. It optimizes the reliable transmission and transactionality of messages. At present, it is widely used in transactions, recharge, stream computing, message push, log stream processing, etc. binglog distribution and other scenarios.
Testing purposes
Compare the performance of Kafka, RabbitMQ, and RocketMQ to send small messages (124 bytes). In this stress test, we only focus on the performance indicators of the server, so the standards of the stress test are:
Continue to increase the pressure on the sender until the system throughput no longer rises and the response time lengthens. At this time, the performance bottleneck of the server has occurred, and the corresponding optimal throughput of the system can be obtained.
testing scenarios
In the synchronous sending scenario, the performance of the three message middleware is clearly distinguished:
The throughput of Kafka is as high as 17.3w/s, which is worthy of being the industry leader of high-throughput message middleware. This mainly depends on its queue mode to ensure that the process of writing to disk is linear IO. At this point, the broker disk IO has reached the bottleneck.
RocketMQ also performed well, with a throughput of 11.6w/s and a disk IO %util close to 100%. After the RocketMQ message is written into the memory, it returns to ack, and a separate thread is dedicated to the operation of flushing the disk. All messages are written to files sequentially.
The throughput of RabbitMQ is 5.95w/s, and the CPU resource consumption is high. It supports the AMQP protocol, which is very heavyweight. In order to ensure the reliability of the message, a trade-off is made in throughput. We also did a performance test of RabbitMQ in the message persistence scenario, and the throughput was around 2.6w/s.
Test conclusion
In terms of the performance of the server to process synchronous transmission, Kafka>RocketMQ>RabbitMQ.
appendix:
test environment
The server is deployed on a single machine, and the machine configuration is as follows:
App version:
test script
To be continued
Earlier we compared the simplest small message sending scenario, and Kafka temporarily won. However, as RocketMQ, which has undergone previous double eleven baptisms, it has more advantages in Internet application scenarios.
Next, we will compare the three types of message middleware based on different influencing factors such as the number of partitions, message size, and consumption form. Stay tuned for follow-up reports!
Reprinted in: http://jm.taobao.org/2016/04/01/kafka-vs-rabbitmq-vs-rocketmq-message-send-performance/