Apache RocketMQ 5.0 Tencent Cloud implementation practice

12.23 Yuan Chuanghui·Shanghai Station, let’s talk about LLM infrastructure

Apache RocketMQ Development History Review

RocketMQ was first born in Taobao's online e-commerce transaction scene. After years of double eleven promotion traffic peaks, it was donated to the Apache community in 2016 and became the top project of the Apache community. It has been used in domestic and foreign e-commerce, finance, Internet and other industries. It has been verified by customers from all walks of life and has been widely recognized.

The Apache RocketMQ community officially released the new 5.0 version in October 2022. The Tencent Cloud Message Queue team also worked closely with the community to support the commercial version of 5.0. Now we will summarize the lessons learned from the entire implementation process and give back to the community. .

What is RocketMQ 5.0?

A new version number?

A newly designed API?

A series of new features implemented?

A new architecture that separates storage and calculation?

A new commercial product form?

RocketMQ’s new thinking for cloud native?

The Apache RocketMQ community has shared and introduced the new 5.0 architecture from different angles over the past year, resulting in many users having inconsistent understandings of the new 5.0 architecture. In fact, the understandings from the above different angles are correct. This article attempts to give a more comprehensive explanation from multiple dimensions. and review to help users fully understand the thinking logic behind the evolution of RocketMQ 5.0 architecture.

Evolution goals of RocketMQ 5.0

Infrastructure adapts to cloud native

The environment that RocketMQ relies on to run has undergone tremendous changes in the past ten years. From the earliest physical machine deployment, to now cloud computing has become deeply rooted in the hearts of the people. Resources are becoming more and more "elastic scaling, pay-as-you-go, high SLA", and computing resources are containerized. , storage resources have evolved into standard distributed storage, such as block storage, file storage and object storage, which are becoming more mature and standardized. Especially object storage represented by S3 has a great cost advantage compared to other storage, so new The evolution of architecture is about taking full advantage of new computing and storage resources.

Lightweight API and complete multi-language SDK support

The protocols before RocketMQ 4.x were based on the private Remoting protocol designed more than ten years ago, which resulted in very high costs for developing non-Java language SDKs. Therefore, the new API and Proxy mode designed based on gRPC in 5.0 can greatly facilitate multi-language SDKs. Develop and enrich the multi-language ecosystem.

Integration of messages, events, flow scenarios and ecological expansion

The community has also open sourced peripheral projects such as EventBridge, Connector, Stream, MQTT, etc., which will help improve and enhance the surrounding ecology and expand more business scenarios.

Key new features of RocketMQ 5.0

In order to support the above three main evolution goals, RocketMQ 5.0 version introduces a large number of new technologies and features. Here is a brief introduction to some key features one by one. Many of them have been put into actual use in Tencent Cloud and have exerted business value.

POP consumption model

Versions before RocketMQ 5.0 only provided the Pull consumption mode (even PushConsumer simulates the Push effect through Pull and long rotation training). The Pull consumption mode is similar to the Kafka consumption mode. It also requires load balancing on the client and calculation of client instances and queues. The mapping relationship, and then consuming messages and maintaining the location information of the queue, through the new POP consumption model, brings the following obvious benefits:

No need to calculate allocation logic on the client, simplifying client logic
Reduces the complexity of client SDK development and facilitates rapid support of multi-language clients
The consumption site is completely maintained on the Broker side to avoid consumption delays caused by slow single consumption nodes.
The Proxy mode adapts to other protocols more smoothly, making it easier to perform push-pull conversions when supporting multiple consumption models.

In the product form of RocketMQ 5.0, Tencent Cloud adopts the POP consumption mode in the process of supporting the new 5.0 protocol and being compatible with other message protocols, which conveniently supports the complete statelessness and load balancing of Proxy.

For more detailed POP solution design extended reading reference:

New API design based on gRPC

In the past few years, the RocketMQ community has become increasingly aware of the shortcomings of the Remoting protocol in the process of supporting RocketMQ 4. Proxy solution, this new API is equivalent to the official unified scalable Proxy solution, which facilitates companies to merge and be compatible with some other protocols based on this Proxy, unify and simplify the architecture, and finally form a storage core with RocketMQ Broker , a unified architecture of stateless Proxy separated storage and calculation that is compatible with various message protocols.

In the process of implementing the new architecture of Tencent Cloud, because the Proxy has to handle CPU-intensive calculations such as protocol serialization and conversion, attention must be paid to optimizing the CPU usage. We have also submitted multiple optimization codes to the community. The following are our specifications for 4C8G Reference pressure test data:

Typical flame diagram occupancy analysis during the Proxy pressure test process is as follows:

For more detailed information on new solution design and extended reading, please see:

Second-level scheduled messages

Timed messages are a type of message often used in online messaging scenarios. After the sender sends the message, it does not want the subscriber to consume the message immediately. Instead, it waits for a period of time before the message becomes visible to the subscriber. A typical business scenario is placing an order. Check the order status after five minutes, or generate points or coupons at a fixed time the next day after the transaction is successful.

Versions before RocketMQ 5.0 could only use the fixed interval mechanism of retrying messages to implement fixed-level timing messages. In the new version 5.0, timing messages have been re-implemented and can support ultra-large-scale and long-time timing messages with any second-level granularity.

For more detailed information about the second-level timing message scheme design and further reading reference:

[RIP43] Support timing messages with arbitrary time delay

Tiered storage

RocketMQ 4.x only supports block devices such as local disks or cloud disks as persistent storage media. Although block device storage can bring low latency and reliability, its storage cost is 5 to 10 times that of object storage, and message queues The data is typical hot and cold distributed data. According to the author's statistics in the actual system, about 85% of the hot data is read through the memory cache within 10 minutes, and the second 10% of the warm data may be read within 1 hour, which is about 5%. % of cold data will only be read in scenarios where it is accumulated for a long time or consumed retrospectively.

RocketMQ 5.0 introduces tiered storage technology, which can move cold data to cheaper storage, such as object storage, which can greatly reduce comprehensive storage costs without reducing user experience. The implementation idea is as shown in the figure below. The purpose of hierarchical storage is achieved by asynchronously copying messages to hierarchical storage when writing, prioritizing reading from local storage when reading, and then reading remote storage if there is no hit.

In the process of implementing tiered storage, Tencent Cloud chose Tencent Cloud Cloud Disk CBS as the primary storage and Tencent Cloud Object Storage COS as the secondary storage. The following table is a performance test report we performed on Tencent Cloud when tiered storage was enabled. It will have almost no impact on online business, which is consistent with the expected behavior of the code (dispatch asynchronous write object storage, hot data read local cache), secondary storage single partition consumption can support 7500msg/s, and expanded partitions can expand the consumption speed in equal proportions , which can meet online demand standards.

For more detailed implementation solutions, see the RIP documentation:

KV based million queue index

In the RocketMQ 4.x version, each Topic actually stores messages in multiple queues. The queue data is stored in a unified Commitlog. The message queue index is stored through the file queue. When the queue is less than 10,000 When the number of queues exceeds 100,000, the read and write services can be provided stably and efficiently. When the number of queues exceeds 100,000, the queue index will degrade into serious random writes, resulting in serious performance degradation.

RocketMQ 5.0 introduces RocksDB storage queue index and uses the LSM index structure characteristics of RocksDB to convert random writes of a large number of file queue indexes into sequential writes of SST files. Even if there are millions of message queues, from the perspective of the entire architecture, the underlying cloud disk is still Only a small number of sequentially written files can still provide stable message reading and writing services.

The following is the data measured in our test environment. When the number of queues is small, the TPS and time-consuming indicators of the two solutions are not much different. However, after the number of queues exceeds 200,000, the index solution based on RocksDB has advantages in performance and stability. obvious.

For a more detailed implementation plan introduction, please refer to:

Native batch messaging support

The batch message in the RocketMQ 4.x version is a "pseudo-batch" message implementation. The sender needs to send a message array organized at the business level. After RocketMQ Broker receives the message array, it will split it into multiple messages one by one. For processing message writing, although the old solution has good compatibility and is simple to implement, it only optimizes the network overhead, and the optimization of compression and storage performance is not obvious.

RocketMQ 5.0 introduces the new AutoBatch feature and optimizes the entire link of batch messages, from the automatic batch programming interface on the sending end to the new storage format and index structure.

The following is the stress test data of several scenarios provided by RocketMQ community salty fish (guyinyou · GitHub), comparing the normal message throughput Double the improvement, compared with Kafka, it can achieve almost the same throughput under the same node specifications and business scenarios. In the scenario of increased partitions, it provides better sending delay jitter:

Using the same test nodes and Kafka for performance comparison, the deployment architecture adopts a test scenario of 3 nodes and 2 replicas. Under the same 16 partitions, almost the same throughput but lower sending delay can be achieved, and as the number of partitions increases, RocketMQ It has obvious advantages in stability and sending delay.

For more detailed design documents and test results, please see:

jRaft Controller implementation

RocketMQ 4.x master-slave replication provides a simple and efficient message high-reliability solution, but there is always a missing function that cannot automatically switch masters and slaves. Although the DLedger mode solves the problem of automatic master selection through three replicas based on Raft, the performance is relatively poor. Poor, and the machine cost is high.

RocketMQ 5.0 adds a new Controller component (this component can be deployed together with Namesrv) to solve the problem of automatic switching in master-slave replication deployment mode. However, the community's default Controller component is based on DLedger, and Raft does not implement it. Not perfect, Tencent Cloud Messaging team members have provided the community with a new more mature Raft implementation solution based on SOFAJRaft , which can achieve a more stable and reliable master-slave switch.

The performance comparison of different replicas and synchronization mechanisms is shown in the figure below:

The chaos test results in Tencent Cloud test environment all passed:

For more detailed design document reference of JRaft Controller:

Other new feature index

The RocketMQ community will have detailed RIP documents and review processes for major feature changes. Due to space limitations, this article only selects a few key new features for a brief introduction. You can refer to more new features All community RIP list links.

The commercialization process of Tencent Cloud RocketMQ

The Tencent Cloud Message Queuing team has gone through a complete commercialization process based on the Apache RocketMQ community in the past few years, and officially launched the 5.x commercial version in September 2023, providing extremely flexible and lower-cost RocketMQ services to meet different scenarios. The differentiated demand for RocketMQ has been implemented in many industries such as finance, travel, education, and games.

New storage and computing separation architecture upgrade

Within Tencent Cloud, we have also adjusted the deployment architecture. According to the storage and calculation separation architecture recommended by RocketMQ 5.0, we can simplify the operation and maintenance of RocketMQ and provide users with a better upgrading and deployment flexibility experience.

The picture below shows the new storage and calculation separation architecture of RocketMQ 5.0:

Brand new 5.0 Serverless product form

Based on the above new storage and computing separation new architecture, we have launched a new TDMQ RocketMQ 5.0 Serverless product form. Through a new pay-as-you-go billing model, and the professional version and above provide elastic TPS capabilities for free, it can cope with emergencies at a lower cost. Send traffic.

Summary and prospects of implementation practice

Looking back on the development history of RocketMQ in the past ten years, we can see the vigorous development of the RocketMQ community and the rapid iterative evolution of functions, and there is also rapid implementation support from major domestic cloud vendors. Tencent Cloud will continue to invest heavily in R&D to provide RocketMQ users with Provides more and better choices.

Tencent Cloud has also actively given back to the RocketMQ community during the commercialization process of RocketMQ. In the past year, Tencent Cloud has contributed 30+ defect fixes and performance optimization code mergers to the RocketMQ community, and has contributed a RIP 67, and hope to work more closely with the RocketMQ community in the future, contribute to the prosperity and development of Apache RocketMQ, and provide users with better RocketMQ services.