On May 25, 2022, Alibaba Cloud's open source PolarDB-X was upgraded and released a new version! PolarDB-X has been serving Alibaba's e-commerce core system since 2009, and began to provide commercial services in 2015, and was officially open sourced in October 2021. This release is the first major version upgrade after open source. It launched version 2.1, which has made great progress in stability, ecological integration and ease of use. This version aligns with the commercial version for the first time in terms of kernel capabilities, adding X-Paxos, There are many important features such as automatic partitioning, OSS hot and cold data separation, etc., and it continues to iterate in the direction of MySQL ecological integration and K8S ecological integration.
Click here to watch the live replay of the press conference
PolarDB-X is a cloud-native distributed database system designed for ultra-high concurrency, massive storage, and complex query scenarios. It adopts the shared-nothing and storage computing separation architecture, supports horizontal expansion, distributed transactions, mixed load and other capabilities, and has the characteristics of enterprise level, cloud native, high availability, and high compatibility with MySQL system and ecology. PolarDB-X was originally born to solve the database scalability bottleneck of Alibaba Tmall's "Double Eleven" core trading system, and has since grown along with Alibaba Cloud. It is a mature and stable database system that has been verified by a variety of core business scenarios.
new features:
This open source includes 5 core features to comprehensively improve the stability and ecological compatibility of PolarDB-X .
01. Highly available open source capabilities complement
Distributed consensus algorithm (Consensus Algorithm) is a basic problem in the field of distributed computing. Its most basic function is to reach a consensus (strong consensus) on a (some) value among multiple processes, and then solve the problem. The availability of distributed systems (high availability) is a problem. In recent years, the continuous rise of NewSQL and cloud-native databases has greatly promoted the combination of relational databases and consistency protocols. Common technologies include Paxos and Raft.
On April 1, 2022, PolarDB-X officially open source X-Paxos, based on native MySQL storage nodes, providing Paxos three-copy consensus protocol, which can achieve high availability and disaster tolerance of financial-grade databases, and achieve RPO=0 production Level availability, which can meet disaster recovery architectures such as three computer rooms in the same city and three centers in two places.
The Paxos protocol is very necessary for the cloud-oriented architecture. The essence of the cloud is virtualization and resource pooling. The change and elasticity of nodes is a routine operation. We need to solve the ability of user-oriented transparent operation and maintenance. In any case, data cannot be Lost, can not be wrong.
02. Distributed horizontal expansion capability upgrade
PolarDB-X, as a MySQL-based native distribution, in addition to providing financial-level disaster recovery capabilities based on Paxos RPO=0, the most important feature is the distributed horizontal expansion, and a new version of the data was officially launched in PolarDB-X version 2.1.0 Partition table, providing Auto partition mode.
The database in Auto mode supports automatic partitioning, that is, data can be automatically and evenly distributed in the cluster without specifying a partition key when creating a table; it also supports manual partitioning of tables using standard MySQL partition table syntax. Combined with the new version of the partition table capabilities, new capabilities such as support for hotspot splitting, TTL (Time To Live) partitioning, and Locality affinity scheduling are added, allowing you to easily enjoy the transparent distribution, elastic scaling, and partition management of distributed databases. dividend.
For details, please refer to the document: AUTO Mode Database .
Based on the new version of the partition table, it is scalable to provide distributed thermal analysis capabilities, sample renderings
Thermal analysis
03. MySQL ecological adaptation acceleration
There is a special CDC (Change Data Capture) component in the PolarDB-X architecture, which is mainly used to provide distributed incremental log acquisition. As MySQL native distribution, the corresponding distributed CDC is also fully compatible with MySQL Binlog in design. In PolarDB-X 2.1.0 version, we have further improved the adaptation and compatibility with MySQL's existing CDC ecosystem.
First of all, the binlog service of PolarDB-X CDC is certified to be compatible with open-source MySQL binlog parsing components such as canal, maxwell, debezium, and Flink CDC. Secondly, PolarDB-X CDC has added a replica service, which is fully compatible with MySQL Replication related protocols. Through the MySQL start slave command, PolarDB-X can be used as an open source MySQL standby database to synchronize data in real time.
PolarDB-X and Flink CDC
04. Perfect lightweight deployment function
PolarDB-X Operator is a Kubernetes-based PolarDB-X cluster management and control system. It hopes to provide complete lifecycle management capabilities on native Kubernetes to meet users' lightweight deployment. In PolarDB-X 2.1.0 version, we have further improved some of the operation and maintenance capabilities, such as providing the monitoring system of Prometheus + Grafana, improving the ability to upgrade distributed nodes, expand and shrink, and version upgrade.
05. OSS cold and hot data separation
TTL（time-to-live）
How to strip cold data from InnoDB row storage? This is a headache for many developers. If the form of delete from statement + where condition is used to delete cold data, it is very likely that the number of scanned rows is too large and the data is too scattered, which will cause the lock table and affect the access of the entire database instance; Drop the old time partitions one by one, and many tables that are not suitable for partitioning by time will be helpless.
In response to this practical problem reported by users, PolarDB-X introduces a new feature of TTL (time-to-live) to help users complete the separation of hot and cold data. Instead of manual maintenance, users can automatically expire data by specifying the start time, partition size, and expiration time in advance. We further transparently partition each physical table in the lower storage layer, and the data is aggregated according to the latest update time.
For example, for the order table t_orders, users are hash partitioned by order id. After the introduction of TTL, each partition is further divided transparently. The expiration of the old time partition (the 2022-01 partition in the figure) is like tearing off a post-it note, and the hot and cold data is stripped without locking the table and manual partitioning.
For the specific use of TTL, you can refer to the official website document: What is the TTL function ?
High performance query
When the cold data is stripped from the main database and archived to the OSS storage service, we get an archive table with OSS as the storage carrier. It is fully compatible with MySQL data types and various query methods. Under the premise of low cost and high availability, it can bring the same experience as the main table. In order to meet the query needs of different users for historical data, we have taken into account both point and complex analytical queries in the design. We conducted a corresponding evaluation for this. Since PolarDB-X on OSS uses column storage, it has natural advantages in report query. Therefore, compared with the row storage mode of PolarDB-X on MySQL, the TPC-H test results have been greatly improved; Sysbench with 100 million rows of data volume The point check test also shows that the archive table can meet the query requirements of historical data. In the process of realizing the above functions, the most critical design is the file system, multi-level cache, multi-level index and query pruning. In addition, it also includes column storage index selection, vectorized calculation, AGG acceleration, etc., which we will introduce in detail in subsequent articles.
TPC-H performance test
Specifications:
CPU: 6 * 16C
Memory: 6 * 128GB
SF = 100 (TPC-H 100GB)
takes about 89s (PolarDB-X on MySQL takes 150s)
Sysbench performance test
specifications:
● Stress test ECS: 1 * 8C32G
● CN: 6 * 16C128G
● Sysbench table rows: 100 million
● Concurrency: 100 The
sysbench performance test data is as follows:
One-click migration
After the separation of hot and cold data is completed, how to quickly archive the data to OSS? Based on the MySQL standard syntax, we provide a very simple and convenient way, only need to execute a table creation statement:
CREATE TABLE [oss_table_name] LIKE [innodb_table_name]
ENGINE = 'OSS' ARCHIVE_MODE = 'TTL'After execution, the OSS table will clone the table structure of the InnoDB table, eliminating the need for users to design the archive table structure; at the same time, the cold data archive table and the source table are bound, and the expired data of the source table will be automatically imported into the archive table. After that, users can complete various data access including point check, range query, and complex analytical query through SQL, just like accessing ordinary tables.
Manually force expiration
If you want more flexible expiration and archiving operations, the following statements allow you to manually expire data and import the expired data into OSS:
ALTER TABLE [innodb_table_name] EXPIRE LOCAL PARTITION [local_partition_name]More features please click here
More detailed Features
Added support for creating database and specified table building mode (new partition table mode and old sub-database sub-table mode), the default is sub-database sub-table mode Partitioning strategies include Hash/Range/List, etc.
Added support for dynamic pruning capabilities for partitioned tables, including constant folding, range merging, and prefix query pruning that support partition column conditions
Added support for JOIN calculation pushdown for partitioned tables
New Added partition management capabilities for partitioned tables, including functions such as adding, deleting, splitting, merging, and migrating partitions
Added table group and other capabilities (including table group creation, deletion, change, etc.), and supports JOIN during partition changes Calculation pushdown is not affected
Added support for global index tables to use MySQL partition table syntax and partitioned according to partitioning strategies such as Hash/Range/List Added
support for automatic splitting using partition table syntax
Table
Added new partition table GSI automatic split will carry primary key, which can handle GSI hotspot issues
Added support for instance scaling
etc.)
Optimized the Check Table command to support metadata consistency verification such as main table partition, index table partition and column definition
Added SQL Advisor to support recommended broadcast tables
Added support for Instant Add Column function
Added support for Explain Statistics pull Get all the information the optimizer needs for
optimization
Optimized the performance of data verification tasks in some DDL background operations to speed up GSI/DDL change operations
Added support for MySQL-compatible Replica-related commands
Added support for storage node PAXOS three-node clusters
Added Replica components, Supports using PolarDB-X as MySQL Slave to consume data through the syntax of change master …
Supports the recording of Rows_query_event type data in the global Binlog, precondition: the DN node binlog_rows_query_log_events parameter needs to be set to On
Added Flink CDC access
New Added CR PolarDBXMonitor to monitor PolarDBXCluster
Added Helm Chart polardbx-monitor, including customized kube-prometheus and predefined Dashboard to display PolarDB-X cluster monitoring information
PXD tool supports single-copy and triple-copy deployment modes
PolarDB-X source code open address
Compute layer: https://github.com/apsaradb/GalaxySQL
Storage layer: https://github.com/apsaradb/GalaxyEngine
Introduction to PolarDB Open Source Community
The PolarDB open source community is a technical exchange platform for PolarDB, an open source database product of Alibaba Cloud. As an open source database product, it is inseparable from the support of users and developers. You can ask questions, functional requirements, exchange experience, share best practices, submit issues, contribute code, etc. for PolarDB products in the community.
In order to allow community members to communicate more conveniently and promote the development of the database industry, the community will organize online and offline meetups, organize exchange activities for colleges and enterprises, and organize technical competitions. Welcome database enthusiasts, users, and developers to join the community.
PolarDB-X open source official website:
https://www.polardbx.com/home
PolarDB-X open source training camp is now registering:
https://developer.aliyun.com/trainingcamp/f1b1508330684d6b975b350c285936ca