In 2022, the official version of RocketMQ 5.0 will be released. Compared with version 4.0, the architecture will become cloud-native and cover more business scenarios.
How to evolve from the Internet era to the cloud era?
1. Evolution history of message queue
The operating system, database, and middleware are the troika of basic software, and message queue is one of the most classic middleware, with a history of more than 30 years.
- The first stage (1980~2000):The most representative one isIBM MQ, expensive and suitable for high-end enterprises, MQ’s own software architecture is a monolithic architecture.
- The second stage (2000~2007):After entering the 2000s, the first generation of open source message queues emerged, giving birth to JMS and AMQP. The big standard corresponds to the two major implementations of the first generation of open source message queues: ActiveMQ and RabbitMQ, which jointly led the early open source message queue technology. Open source has also greatly promoted the popularity of message queues, lowered the threshold for use, and made the technology universal, gradually becoming the standard configuration of enterprise-level architecture. Compared with today,this MQ-like is mainly oriented to traditional enterprise-level application scenarios. Their traffic is generally relatively small, and the demand for horizontal scalability is not that strong.
- The third stage (2007~2017):This is the stage when PC Internet and mobile Internet explode. Due totraditional message queue ( ActiveMQ, RabbitMQ), none of them could withstand the access traffic of hundreds of millions of users and the massive data transmission, so in this process the so-called Internet Message Queue was born. Its core capability is to fully adopt a distributed architecture. It has strong horizontal expansion capabilities. Typical open source representatives include: Kafka and RocketMQ. Closed sources include Taobao’s Notify. The birth of Kafka also extends the message middleware from the traditional messaging field to the streaming field, from the distributed The scenario of asynchronous decoupling of applications extends to the scenario of big data stream storage and stream computing.
- The fourth phase (2014~present):In recent years - cloud computing, Internet of Things (IoT), cloud native, big data, they have led a new wave .
RocketMQ in the Internet Era
The birth background of RocketMQ
Although there were many commercial or open source message queues in the industry at that time, such as IBM MQ, ActiveMQ, and RabbitMQ, without exception,These message queues were all born out of traditional In enterprise-level application scenarios, they cannot bear the stringent requirements of the Internet for high concurrency and unlimited horizontal expansion. Take RabbitMQ as an example. RabbitMQ's queue traffic and storage load are both single-machine and cannot meet the needs of business horizontal expansion. It is It is impossible to expand horizontally based on the explosive development of Internet business.
And At that time, another message queue that adopted a distributed architecture and had unlimited horizontal expansion capabilities was Kafka, but at that time it was mainly used in log transmission scenarios. In terms of stability, it has not been verified by large-scale core business, and it also prefers a simple log-type message queue, which cannot meet the demands of Internet e-commerce for complex messaging features , such as some message filtering or some delayed messages, etc.; on the other hand, traditional message queues cannot solve the distributed consistency requirements of e-commerce transactions. After asynchronous decoupling of applications is achieved through message queues, the e-commerce business also needs to ensure/guarantee different upstream and downstream applications, and the status of orders must be ultimately consistent, otherwise a large number of dirty data, leading to a large number of business errors.
Therefore, for a large-scale e-commerce transaction system, it requires both high performance and consistency, and traditional distributed transaction technology is unable to do anything. For example, although IBM MQ can use XA transactions to meet the functional requirements of distributed consistency, the delay and cost caused by XA are unbearable for massive Internet traffic.
Therefore, in order to solve the e-commerce business's needs for high performance, consistency, and unlimited expansion of message queues, self-developed message queues became Alibaba's only way out at that time. It was against this background that the Internet message queue RocketMQ came into being.
In order to support ultra-large-scale complex e-commerce businesses, RockteMQ has focused on four aspects and formed four major advantages:
Advantage 1: The ability to support very large-scale complex services and rich message features;
Advantage 2: Data consistency,RocketMQ has created a number of key features in terms of consistency, the most representative of which is distribution RocketMQ is the first message queue to implement this feature. It can ensure that the upstream and downstream businesses of the transaction reach final agreement on the order status, so this solution has become the current asynchronous The de facto standard for message consistency solutions has been adopted by many Internet companies. In addition to distributed consistency, RocketMQ also provides sequential message features to meet the needs of sequential consistency;
Advantage Three: Stability. Stability can be considered the most basic feature of e-commerce transactions and financial scenarios, and it is also the fundamental characteristic of RocketMQ.Stability is not limited to the high availability of data and services. RocketMQ has comprehensively built stability from the product level, such as the ability to trace messages, message traces, and dead letter queues (message dead letter mechanism) ;
Advantage 4: High performance, even under the extreme traffic of Double Eleven, RokcetMQ's message writing delay is very low; at the same time, RocketMQ uses Shared - Noting's distributed architecture has a minimalist architecture and zero external dependencies; in addition, it also has the ability to infinitely expand in terms of throughput, with trillion-level throughput guaranteed to meet both microservice and big data scenarios; it has supported dual-core systems for ten consecutive years. The trillion-level message flood peak on the 11th provides low-latency message services for millions of client instances.