Detailed explanation of the advantages of RocketMQ in business message scenarios

Author: Longji

01 Message scene

RocketMQ 5.0 is a real-time data processing platform that integrates message and event streams, and is the de facto standard in the field of business messaging. Many Internet companies use RocketMQ in business messaging scenarios.

The "message, business message" we mentioned repeatedly refers to the decoupling of distributed applications, which is the basic business disk of RocketMQ. Through this article, we will gain an in-depth understanding of the advantages and capabilities of RocketMQ 5.0 in business messaging scenarios, and understand why RocketMQ can become the de facto standard in the field of business messaging.

The classic scenario of RocketMQ in the field of business messaging is application decoupling. This is also the core scenario for solving the distributed Internet architecture of Ali e-commerce in the early days of RocketMQ. It mainly undertakes the asynchronous integration of distributed applications (microservices) to achieve the effect of application decoupling. Decoupling is the most important pursuit of all software architectures.

Distributed applications (microservices) using synchronous RPC versus asynchronous messaging. For example, in a business system, there are three upstream applications and four downstream applications. If synchronous RPC is used, there will be a dependency complexity of 3*4; while using asynchronous messages, it can be simplified to 3+4 Dependent complexity of , simplified from multiplication to addition.

By introducing a message queue to implement asynchronous integration of applications, four major decoupling advantages can be obtained.

code decoupling

Greatly improve business agility. If the method of synchronous invocation is used, every time the business logic is extended, the upstream application needs to explicitly call the downstream application interface, the code is directly coupled, the upstream application needs to change and publish, and business iterations constrain each other. By using message queues to expand new business logic, you only need to add downstream applications to subscribe to a topic, and the upstream and downstream applications are transparent to each other, and the business can remain flexible, independent and fast iterative.

delay decoupling

If you use the synchronous call method, as the business logic increases, the number of remote calls for user operations will increase, the business response will become slower and slower, the performance will decline, and the business development will be unsustainable. With the message queue, no matter how many services are added, the upstream application only needs to call the sending interface of the message queue once to respond to online users, and the delay is constant, basically within 5ms.

Availability Decoupling

If a synchronous call is used, any downstream service unavailability will cause the entire link to fail. This structure is similar to a series circuit, and even when some calls fail, there will be inconsistencies in the state. However, using RocketMQ for asynchronous integration, as long as the RocketMQ service is available, the user's business operations are available. The RocketMQ service is provided through a broker cluster composed of multiple pairs of active and standby. As long as one pair of active and standby is available, the overall service is available. As a basic software, the availability is much greater than that of ordinary business applications, and the business advancement of downstream applications can be delivered through MQ's reliable message delivery to achieve.

flow decoupling

That is to cut peaks and fill valleys. If a synchronous call is used, the upstream and downstream capacities must be aligned, otherwise cascade unavailability will occur. Full capacity alignment requires a lot of effort for full-link stress testing and more machine costs. With the introduction of RocketMQ, based on RocketMQ's accumulation capacity of billions of messages, downstream businesses that do not require high real-time performance can be consumed as much as possible, which not only ensures system stability, but also reduces machine costs and R&D and operation costs.

02 Basic Features

Ali's transaction application process is as follows: when a user places an order on Taobao, he will call the transaction application to create an order. The transaction application will drop the order into the database, then generate an order creation message to RocketMQ, and return the interface to the end user for the successful order creation. The progress of the completed transaction process relies on RocketMQ to deliver the order creation message to the downstream application. The member application receives the order message and needs to give the buyer points and gold coins to trigger the user to motivate related businesses. The shopping cart application is responsible for deleting the items in the shopping cart to prevent users from making repeated purchases. At the same time, the payment system and logistics system will also advance the payment link and contract fulfillment link based on the change of the order status.

Over the past decade or so, Ali's e-commerce business has continued to flourish, and the number of downstream applications for transactions has reached hundreds and is still increasing. The e-commerce architecture based on RocketMQ has greatly improved the agility of Ali's e-commerce business. The upstream core trading system does not need to care about which applications are subscribing to transaction messages. The delay and availability of transaction applications have also been kept at a high level, relying only on a small amount The core system and RocketMQ will not be affected by hundreds of downstream applications.

The downstream business types of transactions are different. There are a large number of business scenarios that do not require real-time consumption of transaction data. For example, logistics scenarios can tolerate a certain delay. Through RocketMQ's 100-million-level accumulation capability, the machine cost is greatly reduced. The shared-nothing architecture of RocketMQ is capable of unlimited horizontal expansion. It has supported the high-growth Double Eleven message peak for 10 consecutive years, and reached 100 million TPS a few years ago.

03 Enhancement

In the classic scenario, RocketMQ has many differentiated advantages and enhancements compared to other message queues.

First of all, in terms of stability, stable transactions are the most important requirement in financial scenarios. The stability of RocketMQ is not limited to the high-availability architecture, but to build a stable competitiveness through a full range of product capabilities. For example, in the retry queue, when the downstream consumer fails to consume a certain message because the business data is not ready or other reasons, RocketMQ will not block the consumption, but can add the message to the retry queue, and then retry according to time decay. If a message cannot be successfully consumed after more than a dozen retries due to some factors, RocketMQ will transfer it to the dead letter queue, and users can use other means to process the failed message, which is just needed by the financial industry.

At the same time, if the business does not meet expectations due to code bugs after successful consumption, the application can fix the business bugs and republish them, and then apply the message backtracking function to pull the messages back to the previous point in time, so that the business can be reprocessed according to the correct logic.

RocketMQ's consumption implementation mechanism adopts the adaptive pull mode of consumption, which can prevent consumers from being overwhelmed by large traffic in extreme scenarios. At the same time, in the consumer's SDK, the threshold protection of the number of cached local messages and the memory usage of messages is implemented to prevent the memory risk of consumer applications.

Secondly, RocketMQ also has excellent observability, which is an important auxiliary means for stability. RocketMQ is the industry's first message queue that provides message-level observability. Each message can carry a business primary key. For example, in a transaction scenario, users can use the order ID as the business primary key of the message. When the business of an order needs to be checked, the user can query the generation time and message content of the message based on the order ID. The observable data of the message can also be drilled down to see which producer machine sent the message, which consumer machine consumed it at what time, and whether the consumption status was successful or failed through the message track.

In addition, it supports dozens of core measurement data, including cluster producer traffic distribution, slow consumer ranking, average consumption delay, consumption accumulation quantity, consumption success rate, etc. Based on the rich indicators, users can build a more complete monitoring and alarm system to further strengthen the stability.

In order to support a more flexible application architecture, RocketMQ provides a variety of modes in key interfaces such as production and consumption.

Producer interface : RocketMQ provides both synchronous sending interface and asynchronous sending interface. Synchronous sending is the most commonly used mode. The orchestration of the business process is serial. After the application sends the message and the Broker completes the storage and returns success, the application executes the next logic. However, in some scenarios, multiple remote calls are involved in completing the business. In order to further reduce the delay and improve performance, the application will adopt a fully asynchronous method to send remote calls concurrently (it can be a combination of multiple messages or RPC), asynchronous Collect results and push them into business logic.

There are also two ways in terms of consumer interfaces :

Listener mode passive consumption

This is currently the most widely used method. Users do not need to care about when the client goes to the Broker to pull messages, when to send a confirmation of successful consumption to the Broker, and does not need to maintain details such as the consumption thread pool and local message cache. You only need to write a piece of business logic for the message listener, and return Success or Failure according to the business execution result. It belongs to the fully managed mode, users can focus on the writing of business logic, and completely entrust the implementation details to the RocketMQ client.

active consumption model

Give more autonomy to users, also known as Simple Consumer. In this mode, users can decide when to go to the Broker to read messages and when to initiate consumption confirmation messages. The execution thread of the business logic is also autonomously controllable. After reading the message, the consumption logic can be executed in a custom thread pool. In some scenarios, the processing time and priority of different messages will be different. Using the Simple Consumer mode, users can perform secondary distribution according to the attributes and sizes of messages, and isolate them to different business thread pools for processing. This mode also provides the ability to set the timeout period of message granular consumption. For some messages that consume a long time, users can call the change Invisible Duration interface to extend the consumption time and avoid retrying after timeout.

04 Summary

Classic message scenario: application decoupling;

RocketMQ basic features: publish subscription, reliable message, billion-level stacking, unlimited expansion;
Enhancement capabilities for business message scenarios: stability, observability, and diversified interfaces.

RocketMQ learning community experience address

The RocketMQ learning community is online! AI interaction, one second to understand the source code of RocketMQ functions. RocketMQ learning community is the first knowledge service community based on AIGC in China, aiming to become a "personal second" on the way of RocketMQ learning.

PS: The RocketMQ community uses RocketMQ 5.0 data as the main training content. During continuous optimization iterations, the answers are all generated by artificial intelligence models. The accuracy and completeness of the answers cannot be guaranteed, and they do not represent the attitude or opinion of the RocketMQ learning community.

Experience the RocketMQ learning community immediately (PC is recommended to experience the full functionality)

https://rocketmq-learning.com/

【Event】Experience the six major production environments of RocketMQ with one click

Free trial + 30-second one-click experience, low threshold, fast, efficient, and easy to operate, let you know about RocketMQ, a cloud message queue that "has gone through the test of trillions of data floods"!

Click here to join the event now!