Author: Jiaze
ApsaraMQ keeps pace with the times and forges ahead
It has been more than ten years since the birth of Alibaba Cloud Message Queue. This year, Alibaba Cloud’s messaging products have been fully branded to ApsaraMQ, keeping pace with the times and forging ahead.
In 2012, RocketMQ was born within the group and used for core business links such as Taobao transactions. After years of double 11 trillion-level tests and stability polishing by the group, it relies on cloud computing incubation and the collaboration of the open source community. In 2015, RocketMQ provided commercial message queue services for enterprises on Alibaba Cloud. Then based on RocketMQ, we expanded commercial products such as Kafka, RabbitMQ, MQTT, MNS, etc., and are committed to serving enterprises with more scenario needs;
In 2017, RocketMQ became a top-level project of the Apache Foundation;
In 2022, RocketMQ5.0 will be officially released, completely built on cloud native infrastructure, fully releasing the pooling capabilities of cloud infrastructure and improving the core technical indicators of messaging in all aspects. New improvements have been made in event and stream processing. In terms of message processing, a message model is built on top of the queue model . The business can only care about the message without caring about the queue. Based on the message consumption model, the client, connection and consumption are all stateless and can drift on any Proxy node. Really lightweight.
In 2023, the entire range of products will be serverless, providing enterprises with extremely fast, flexible, and more cost-effective cloud message queue products to help enterprises reduce costs in the use of message queues.
Reliable message channel for application and data integration
Looking back on the past ten years or so, messaging products have always adhered to the trinity of development ideas of open source, group, and business, and the core evolution and products have been iterated together. Over the past ten years, Alibaba Cloud has served a total of 100,000 companies, covering more than 20 industries such as the Internet, retail, finance, and automobiles. Large-scale production practices have continued to accumulate the core advantages of products. At the same time, in response to enterprise demands, the product matrix is enriched to cover the needs of diverse business scenarios. Provide more accurate product selection for enterprises to migrate to the cloud.
- RocketMQ is widely used in the fields of microservices and applications for its 4-9 availability and 9-9 data reliability. At the same time, RocketMQ provides timing, sequence, and transaction diversity message types to meet the complex business needs of applications;
- RabbitMQ is also a representative of the open source ecosystem for messaging. Commercial RabbitMQ solves the pain points of open source RabbitMQ such as the inability to flexibly expand and shrink, brain splits, and stacking, and supports the seamless migration of open source RabbitMQ to the cloud;
- Kafka is a representative of streaming storage and is widely used in big data/log processing scenarios. The commercial version is compatible with open source versions before 3.3, and is deeper and stronger in storage and calculation separation, high availability disaster recovery, and elasticity. Compared with Open source self-construction costs are lower and service availability is higher;
- In the terminal scenario of the Internet of Things/Mobile Internet, MQTT provides tens of millions of links and millions of concurrent services. Compared with the open source version, it strengthens the back-end ecological construction of MQTT services and makes data links sustainable.
- MNS is a messaging product benchmarked against AWS SQS/SNS. It uses a lightweight message queue to support simple business asynchronous communication scenarios, and provides lightweight event notification and SaaS integration capabilities.
Whether it is migrating open source to the cloud or building a new business, in the ApsaraMQ product series, you can always find a reliable messaging middleware suitable for business scenarios. In recent years, the Alibaba Cloud messaging team has invested heavily in storage and computing separation, multi-level storage and elasticity, and carried out architecture upgrades to move messaging products from cloud-native to serverless.
Cloud products move from cloud native to serverless
The essence of cloud native is to fully unleash the dividends of cloud computing and achieve cost reduction and efficiency increase of enterprise business. In the cloud native stage, cloud vendors provide fully managed middleware services. While being open source compatible, they supplement operation and maintenance and visual user interfaces to achieve out-of-the-box effects. At this time, most commercial forms are packages with reserved resources. The annual and monthly subscription fee is for enterprises.
As the scale of cloud resources continues to expand, the scale effect allows enterprises to pay according to usage, but there are still scale limitations for single instances, making it difficult to cope with sudden business traffic.
With the development of cloud native, the idea of serverless is constantly being strengthened, and the realization of serverless products is also constantly empowered. Serverless has reached the implementation stage. Enterprises only need to focus on their own business logic and are indifferent to the use of resources and business traffic.
Cloud messaging service designed for serverless
From the user's perspective, serverless messaging continues to evolve in terms of cost, ease of use, stability, and ecological integration:
low cost:
- Architectural decoupling separates network, computing and storage to improve its own computing capabilities while making full use of the elastic capabilities of cloud-native networks and storage;
- Improve product performance and reduce the cost of using the product's own resources. At the same time, build an SRE platform to achieve automated delivery, upgrades, expansion, and reduce delivery costs and operation and maintenance costs;
- Optimize the billing method and decouple billing items (message volume, storage, network) so that billing items are paid based on actual resource usage, reducing the waste of excessive estimates caused by fixed billing items in the past.
Ease of use: Mainly considering the convenience, reliability, losslessness and control plane migration after migration to the cloud, providing valuable business data, not only helping enterprises observe the use of cloud products, but also providing reference for business optimization.
stability:
Stability is the basic consideration for enterprises to use cloud products. Cloud products operate stably, the process can be monitored, and emergency plans are important guarantees for business continuity.
Ecological integration:
A rich ecosystem of messaging products can help companies quickly complete business development and amplify the value of messaging.
ApsaraMQ core product upgrade
The full range of ApsaraMQ products are released in Serverless version. RocketMQ, RabbitMQ, and MQTT Serverless versions have a threshold of 0 instances and are charged based on actual usage. Computing costs can be reduced by up to 75%, and concurrent traffic can reach 10,000 QPS per second with adaptive elasticity. After separation of storage and calculation and multi-level storage architecture, message storage costs can be reduced by up to 1/3.
Kafka released version 3.0 at the end of October. Computing nodes are stateless, and storage and data availability are moved to the cloud storage infrastructure. This solves the problems of limited single disk throughput, binding storage and computing ratios, and data migration that affects business and cross-border expansion and contraction. Availability zone high availability performance and other issues. Single instance traffic can reach GB adaptive elasticity in seconds. Compared with open source and self-built, the cost is reduced by up to 20%.
Separation of storage and calculation is the basis for serverless messaging services
Just like application decoupling, when computing and storage are technically separated, the computing layer can focus more on the definition and evolution of business logic. Only in this way can we better realize usage according to quantity and adaptive flexibility according to demand.
In the system architecture, metadata (user instance relationships, subscription relationships, parameter configurations, etc.) are managed uniformly; business message data is completely protected by professional storage facilities based on basic storage facilities, making the computing layer truly stateless.
The computing layer sinks complex data reliability mechanisms to the storage layer:
-
Reduces the complex logic of the computing layer, such as data replication costs, improving performance and stability;
-
Reduces the impact of cross-AZ network data transmission, and at the same time, reduces network throughput to avoid network congestion;
-
It reduces the resource overhead and business impact caused by data replication during expansion and contraction, and improves service availability.
The storage layer makes full use of Alibaba Cloud's basic capabilities.
Pangu DFS uses multiple copies and EC strategies to ensure extremely high data reliability. Scatter the data into different racks. When a cheap server fails, a copy of the data can still be quickly restored to ensure data security. This reliability guarantee provides assurance to the upper layer that the data will not be lost. It has laid a solid foundation, making message data more reliable and data more secure. Pangu DFS supports cross-data center disaster recovery strategies, with hundreds of microsecond-level average latency, millisecond-level long-tail latency, and a single storage node’s IOPS processing capacity of a 200Gbps network. At the same time, data reliability reaches 12 9s, and availability reaches 5 9s. .
Second-level elasticity of computing power reduces idle costs
The traffic graph of messages is basically a tidal type, or a tidal type accompanied by high glitches. In the past, self-built users and cloud products could basically be summarized as resource reservation types.
Users use a fixed batch of resources to support business traffic. When core business traffic peaks, wealthy users reserve or add machines in advance to alleviate the glitch traffic, while less wealthy users limit non-core business traffic. However, it brings problems of difficulty in evaluation and waste of cost to users.
In the Serverless architecture version, network, computing and storage are separated, and network and storage are handed over to basic products to provide adaptive elasticity, and pay-as-you-go.
In addition to the resource elasticity method, the computing layer adds an intelligent pre-elastic method based on the instance's past business traffic indicators and cluster water level indicators. At the same time, a method for scheduling instance traffic across clusters has also been added. Shorten the time during which business traffic is restricted as much as possible to achieve the goal of quickly supporting glitch traffic.
Multi-level storage enables on-demand storage, large capacity, and low cost
Messages are intermediate data. In theory, messages are produced and consumed quickly, and data flows in memory at the fastest speed. But in actual production, we may face a scenario where the message cannot be consumed in time, or the message needs to be consumed again.
In these scenarios, the data is often not in PageCache and needs to be read from disk and loaded into PageCache. When there are many cold reads, PageCache competes fiercely, and frequent swapping in and out causes cache pollution, seriously affecting write performance. In addition, the same thread pool that handles production and consumption requests may be affected by cold read blocking, causing all requests to fail in a timely manner. In order to alleviate this problem, through optimization strategies such as "hot and cold data isolation", separation of hot and cold threads (coroutines), and preloading, message production will not be affected and the service will be more stable.
The introduction of cheap multi-level storage reduces message storage costs; at the same time, it can extend the data storage time.
Reliable SLA guarantee is the basis for stable service
Based on a series of architectural evolutions, product availability is guaranteed in technical implementation. Single node or availability zone failures do not affect service availability. At the same time, single node failures can also use the characteristics of containers to quickly self-heal. Service availability can reach up to 4 9s, and data reliability can reach 9 9s;
In terms of runtime stability guarantee, theory and practice are combined. Through the SRE platform, black box detection and indicator monitoring are combined, covering more than 50 detection indicators. It can achieve 24-hour detection coverage and be alerted within minutes of abnormalities. , service SLO is observable. For each item of abnormal alarm, there is a corresponding actionable emergency plan.
MQ User Plane fully upgraded
In addition, the product experience has also been improved. The full range of ApsaraMQ products is unified for user viewing. Based on Metric indicators, message traces, key events and business logs, alarms, inspections and diagnosis are provided for business development and operation and maintenance personnel; based on expert experience , define event models and provide root cause analysis to help enterprises quickly troubleshoot and solve business problems.
For architects, it provides one-click experience of product capabilities and cost estimation capabilities for pre-access evaluation, allowing analysts to quickly understand the product through experience; it unifies the messaging series product ecology to quickly realize data integration with the ability to inflow and outflow messages. .
Smooth visualization migration to the cloud with low risk and low cost
During the migration to the cloud phase, a visual migration tool is provided to ensure smooth and lossless migration to the cloud.
A rich messaging ecosystem helps enterprises quickly build applications
After migrating to the cloud, Messaging provides a rich ecosystem to help enterprises quickly build applications.
With EventBridge/function computing/cloud workflow as the core, we build a cloud integration paradigm and enhance the connectivity between cloud products; in terms of cloud product event integration, EventBridge has currently integrated 200+ cloud product event sources and more than 3,000 event types.
In terms of SaaS application integration, more than 50 SaaS services including DingTalk, JushiTa and the cloud can be connected to EB through webhook.
In addition, EB has currently connected to all Alibaba Cloud's cloud product APIs, and any event can be accessed through the cloud product API.
Cloud workflow provides visual process orchestration, complete API docking, and EventBridge cloud services can be orchestrated without writing code; function computing improves the integration experience and can complete analysis scenarios from message data to big data storage based on EventBridge.
Confluent becomes back-to-back technology partner
On October 31, at the Computing Conference in Hangzhou, Ding Yu, head of Alibaba Cloud's cloud native application platform, announced that Confluent has become Alibaba Cloud's technology partner, and the cooperation has been upgraded to jointly expand and serve the Asia-Pacific market. With this partnership signing, Alibaba Cloud and Confluent will further deepen their cooperation. Confluent will be a product of Alibaba Cloud and will be fully released on Alibaba Cloud domestic sites and international Asia-Pacific sites. Confluent specifications will be launched under Alibaba Cloud Apsara for Kafka and fully support Confluent. Commercialization capabilities and solutions. Through technical support provided jointly with the Confluent team, domestic enterprises will use Confluent services on Alibaba Cloud.
