Memory is a very important data storage medium for databases. It carries all query and calculation requests and plays an important role in improving performance, real-time analysis and other scenarios. Precisely because memory is so important, insufficient memory is often encountered, causing the query to be forcibly terminated, and even negatively affecting the performance of other online queries. For example:
In lightweight ETL or query acceleration scenarios (such as INSERT INTO AS SELECT tasks, or building materialized views of large wide tables for modeling), since the queries involved are usually complex calculations such as multiple large table associations or high-cardinality aggregation, etc. It is difficult to apply query optimization during actual task calculations, resulting in high memory usage and task failure.
In ad-hoc random probing, the analyst accidentally wrote a very complex query involving a very large data set, causing the query to fail. In fact, compared with the queries of key online businesses, such occasional queries and ETL tasks have a greater impact on the query. Performance requirements are usually not high. We just need them to be able to run smoothly with limited resources and produce results without having an impact on the online business, so as not to lose the big for the small.
Intermediate result placement
Memory traffic management now, please bypass disk
In order to make more reasonable use of limited memory resources, StarRocks has launched the resource isolation function since version 2.2, which allocates memory and CPU through resource groups. If memory is compared to a highway for data computing, resource isolation is like a bus lane. Although it isolates online queries from ETL and occasional large queries to a certain extent, allowing online queries to have independent lanes and be more stable, it does not solve how to make ETL and large queries in other lanes pass smoothly without causing traffic. jams or even accidents.
Since version 3.0, StarRocks has launched the intermediate result placing function (Spill). Simply put, intermediate results are written to disk by temporarily writing the intermediate results of large operators involved in the query to disk, thereby freeing up memory space. Intermediate result placement is like a traffic policeman. It can control traffic when the memory is blocked and allow some intermediate results to wait around the disk. This ensures that the query operation continues to execute without failing due to insufficient memory.
Currently, StarRocks supports the placement of intermediate results of aggregation operators, sorting operators, and HASH JOIN (LEFT JOIN, RIGHT JOIN, FULL JOIN, OUTER JOIN, SEMI JOIN, and INNER JOIN) operators. Through disk placement, the memory used by queries can be enlarged ten times, thus supporting more large queries and ETL application scenarios.
Counter placement
Intelligent memory management assistant
However, downloading intermediate results to disk is a way of exchanging disk space for memory space, which will definitely involve additional I/O overhead and performance loss. How to dynamically perform disk placement based on memory conditions and ensure low query performance loss is a relatively big challenge.
StarRocks supports two placement mechanisms: forced placement and automatic placement. In the automatic disk placement mode, the system will automatically determine whether the current query needs to start disk placement in real time based on the current memory usage. Specifically, before each operator actually processes data, StarRocks will estimate how much memory it needs to apply for to complete the execution, and try to reserve it. If the reservation ends and still does not exceed 80% of the query memory limit, the operator placement will not be triggered. Otherwise, the operator placement will be triggered to reserve memory to serve other queries.
Query is more stable and performance is still leading
Based on StarRocks version 3.1, the TPC-DS 10T test set is used to test the ability of intermediate results to be placed on disk.
Can the results be stably produced on a very small cluster?
Cluster configuration: 1FE (8C 16G) 3BE (16C 64G), two Alibaba Cloud PL1 level cloud disks are mounted under each BE
When the intermediate result placement was not enabled, 15 of the 99 queries failed. After enabling the placement of intermediate results, all queries were executed successfully. Operator placement significantly improves query stability.
How is the performance?
StarRocks and Spark were compared on the same machine configuration. When automatic operator placement is enabled and the query runs smoothly, the overall query performance of StarRocks is 4.35 times that of Spark.
Materialized view + intermediate result placement A powerful tool for flexible acceleration
In Redefining Materialized Views, the Speedy Lake Warehouse Artifact You Must Have, we introduced how to flexibly accelerate lake warehouses through materialized views. In the integrated lake-warehouse architecture, materialized views play a key role in data modeling, transparent acceleration, and incremental calculations, which can further reduce the complexity of data processing, improve query performance, and optimize data timeliness, allowing users to While the architecture is upgraded, you can also enjoy the upgrade of the user experience.
Imagine a very common query acceleration scenario: the ODS layer contains multiple fact tables and dimension tables. Queries often need to perform various aggregation calculations on the basis of multi-table associations. Frequent queries consume a lot of memory and also cause problems. Query performance has a certain impact. An intuitive idea is to build a materialized view and use the intelligent query rewriting function of the materialized view to achieve transparent acceleration.
In order to allow more queries to be transparently rewritten into materialized views, we recommend creating materialized views for the most common parts of the query. In this scenario, a large wide table materialized view without predicates is constructed, and all subsequent queries involving related tables can be automatically rewritten. For details, please see:
https://docs.starrocks.io/zh-cn/latest/using_starrocks/query_rewrite_with_materialized_views
However, since there are no predicates, no pushdown optimization can be applied to the construction and refreshing of materialized views. It usually requires the complete association of huge multiple tables, which is a big challenge for memory.
The combination of materialized views and intermediate result placement is exactly what we need to deal with this scenario. Let’s take the TPC-H 1TB test set as an example. This test set involves a large number of queries that aggregate order tables and dimension tables and then roll up and filter them. Test Results:
Cluster information: StarRocks 3.1, 1 FE 3 BE, 16Core 64GB, 600GB disk per BE
direct inquiry
In the case of direct queries, many queries fail due to memory constraints.
After enabling intermediate result placement, all queries can produce results stably.
Create materialized views of large wide tables to speed up queries
In the case of a direct build, the build failed due to memory overrun.
After enabling intermediate results placement, the materialized view can be successfully constructed and refreshed, and the query speed is significantly improved.
Materialized view accelerated SQL example:
It can be seen from the case:
In the case of direct query, the placement of intermediate results significantly improves the stability of the query.
Placing intermediate results on disk improves the success rate of building and refreshing materialized views, and significantly accelerates queries after creating materialized views.
future outlook
The placement of intermediate results is a very important part of the StarRocks lake warehouse integrated structure. It plays an important role in scenarios such as complex analysis, data processing, and materialized view construction under large amounts of data. It releases the memory shackles and solves the analysis scenarios and needs of more users. Complex calculation queries or data processing and modeling can all run smoothly with controllable resources. In the future, the placement of intermediate results will be combined with resource isolation and multi-warehouse capabilities to achieve more fine-grained resource isolation and resource management and control, allowing you to customize resource configuration and usage rules for different workflows.
This article is published by mdnice Multiple platforms