This translation from the official website: Streaming Aggregation https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/tuning/streaming_aggregation_optimization.html
SQL is a language for data analysis of the most widely used. Flink the SQL Table API enabling a user to efficiently and with less time and effort defined flow analysis applications. Moreover, Flink Table API and SQL has been effectively optimized, it integrates a number of query optimization and optimization of operator implementation. But not at all optimizations enabled by default, so for some workloads, you can improve performance by some of the options open.
On this page, we will introduce some useful optimization options and internal flow aggregation principle, which will bring great improvements in some cases.
Note: Currently, only Blink program supports tuning options mentioned on this page.
Note: Currently, only the borderless polymeric support optimized flow aggregation . The future will support the window polymerization of optimization .
- Small batch polymerization
- Local global polymerization
- Segmentation different polymerization
- FILTER modifier used in the different polymerization
By default, a unbounded operator a polymerization process the input record, i.e., (1) read from the state of the accumulator, (2) accumulating the recording / retracted into the accumulator (3) Write accumulator back to the state, a record from (1) (4) re-processing. This processing mode may increase the cost of StateBackend (especially for RocksDB StateBackend). In addition, the production of very common data skew make the problem worse, and make work easier to bear back pressure situation.
Small batch polymerization
Small batch polymeric core idea is to set an internal input buffer in the polymerization operator buffer. When the trigger input to be processed, each key only to access a status of the operation. This can greatly reduce state spending and get better throughput. However, this may add some delay, because it will buffer some records instead of dealing with them immediately. This is a trade-off between throughput and delay.
The following figure illustrates how to reduce small quantities of polymerized state operation.
MiniBatch optimization is disabled by default. In order to make this optimization, you should set up table.exec.mini-batch.enabled , table.exec.mini-batch.allow-latencyand table.exec.mini-batch.size. See Configuration page for more details.
The following example shows how to enable these options.
// instantiate table environment val tEnv: TableEnvironment = ... // access flink configuration val configuration = tEnv.getConfig().getConfiguration() // set low-level key-value options configuration.setString("table.exec.mini-batch.enabled", "true") // enable mini-batch optimization configuration.setString("table.exec.mini-batch.allow-latency", "5 s") // use 5 seconds to buffer input records configuration.setString("table.exec.mini-batch.size", "5000") // the maximum number of records can be buffered by each aggregate operator task
Local global polymerization
The partial polymerization is proposed in two stages to solve the problem of data skew, i.e., the first partial polymerization in the upstream and downstream of the polymerization globally, similar to the MapReduce Combine + Reduce mode. For example, consider the following SQL:
SELECT color, sum(id)
FROM T
GROUP BY color
Recording the data stream may be inclined, so some instances the polymerization operator will be more than the other examples of the recording process, which can lead to hot spots. Local aggregation can help having a certain number of input keys is the same accumulated into a single accumulator. Global summary will receive only reduce the accumulator, rather than a large number of original input. This can greatly reduce the cost of network restructuring and state visits. Each time the accumulated number of local aggregation batch interval based on a minimum input. This means that the local - global small batch polymerization depends enabled optimization.
The following diagram shows how to improve the performance of local global polymerization.
The following example shows how to enable local global aggregation.
// instantiate table environment val tEnv: TableEnvironment = ... // access flink configuration val configuration = tEnv.getConfig().getConfiguration() // set low-level key-value options configuration.setString("table.exec.mini-batch.enabled", "true") // local-global aggregation depends on mini-batch is enabled configuration.setString("table.exec.mini-batch.allow-latency", "5 s") configuration.setString("table.exec.mini-batch.size", "5000") configuration.setString("table.optimizer.agg-phase-strategy", "TWO_PHASE") // enable two-phase, i.e. local-global aggregation
分割不同的聚合
局部全局优化可有效消除常规聚合的数据偏斜，例如SUM，COUNT，MAX，MIN，AVG。但是，在处理不同的聚合时，其性能并不令人满意。
例如，如果我们要分析今天有多少唯一用户登录。我们可能有以下查询：
SELECT day, COUNT(DISTINCT user_id)
FROM T
GROUP BY day
如果distinct key（即user_id）的值稀疏，则COUNT DISTINCT不能减少记录。即使启用了局部全局优化，它也无济于事。因为累加器仍包含几乎所有原始记录，并且全局聚合将成为瓶颈（大多数繁重的累加器由一项任务处理，即在同一天）。
此优化的想法是将不同的聚合（例如COUNT(DISTINCT col)）分为两个级别。第一次聚合由组密钥和其他存储桶密钥混洗。使用来计算存储桶密钥HASH_CODE(distinct_key) % BUCKET_NUM。BUCKET_NUM默认为1024，可以通过table.optimizer.distinct-agg.split.bucket-num选项配置。第二次聚合由原始组密钥改组，并用于SUM聚合来自不同存储桶的COUNT DISTINCT值。由于相同的唯一键将仅在同一存储桶中计算，因此转换是等效的。存储桶密钥充当附加组密钥的角色，以分担组密钥中的热点负担。存储桶关键字使工作具有可伸缩性，以解决不同聚合中的数据偏斜/热点。
拆分非重复聚合后，上述查询将自动重写为以下查询：
SELECT day, SUM(cnt) FROM ( SELECT day, COUNT(DISTINCT user_id) as cnt FROM T GROUP BY day, MOD(HASH_CODE(user_id), 1024) ) GROUP BY day
下图显示了拆分的非重复聚合如何提高性能（假设颜色代表天，字母代表user_id）。
注意：上面是最简单的示例，可以从此优化中受益。除此之外，Flink 支持分裂更复杂的聚集查询，例如，一个以上的具有不同的不同密钥（例如不同的集合COUNT(DISTINCT a), SUM(DISTINCT b)），与其他非重复的聚合工作（例如SUM，MAX，MIN，COUNT）。
注意：但是，当前，拆分优化不支持包含用户定义的AggregateFunction的聚合。
以下示例显示如何启用拆分非重复聚合优化。
// instantiate table environment val tEnv: TableEnvironment = ... tEnv.getConfig // access high-level configuration .getConfiguration // set low-level key-value options .setString("table.optimizer.distinct-agg.split.enabled", "true") // enable distinct agg split
在不同的聚合上使用FILTER修饰符
在某些情况下，用户可能需要从不同维度计算UV（唯一访客）的数量，例如Android的UV，iPhone的UV，Web的UV和总UV。许多用户将选择CASE WHEN支持此功能，例如：
SELECT day, COUNT(DISTINCT user_id) AS total_uv, COUNT(DISTINCT CASE WHEN flag IN ('android', 'iphone') THEN user_id ELSE NULL END) AS app_uv, COUNT(DISTINCT CASE WHEN flag IN ('wap', 'other') THEN user_id ELSE NULL END) AS web_uv FROM T GROUP BY day
但是，在这种情况下，建议使用 FILTER 语法而不是CASE WHEN。因为FILTER它更符合SQL标准，并且将获得更多的性能改进。 FILTER是用于聚合函数的修饰符，用于限制聚合中使用的值。将上面的示例替换为FILTER修饰符，如下所示：
SELECT day, COUNT(DISTINCT user_id) AS total_uv, COUNT(DISTINCT user_id) FILTER (WHERE flag IN ('android', 'iphone')) AS app_uv, COUNT(DISTINCT user_id) FILTER (WHERE flag IN ('wap', 'other')) AS web_uv FROM T GROUP BY day
Flink SQL优化器可以识别同一唯一键上的不同过滤器参数。例如，在上面的示例中，所有三个COUNT DISTINCT都在user_id列上。然后Flink可以只使用一个共享状态实例，而不是三个状态实例，以减少状态访问和状态大小。在某些工作负载中，这可以显着提高性能。
欢迎关注Flink菜鸟公众号，会不定期更新Flink（开发技术）相关的推文
