RocksDB State Backend Details

Incremental Checkpoints

Recovery time of incremental checkpoints may be longer or shorter compared to full checkpoints. If your network bandwidth is the bottleneck, it may take a bit longer to restore from an incremental checkpoint, because it implies fetching more data (more deltas). Restoring from an incremental checkpoint is faster, if the bottleneck is your CPU or IOPs, because restoring from an incremental checkpoint means not re-building the local RocksDB tables from Flink’s canonical key/value snapshot format (used in savepoints and full checkpoints).

与完全检查点相比,增量检查点的恢复时间可能更长或更短。如果网络带宽是瓶颈,那么从增量检查点恢复可能需要更长的时间,因为这意味着需要提取更多数据(更多增量)。如果瓶颈是CPU或IOPs,那么从增量检查点恢复速度更快,因为从增量检查点恢复意味着不用从Flink的标准键/值快照格式重新构建本地RocksDB表。

Limitations of the RocksDBStateBackend:

  • As RocksDB’s JNI bridge API is based on byte[], the maximum supported size per key and per value is 2^31 bytes each. IMPORTANT: states that use merge operations in RocksDB (e.g. ListState) can silently accumulate value sizes > 2^31 bytes and will then fail on their next retrieval. This is currently a limitation of RocksDB JNI.

RocksDBStateBackend限制: 

由于RocksDB的 JNI bridge API 基于byte[],每个键和每个值支持的最大大小为2^31字节。重要提示:在RocksDB中使用合并操作的状态(例如ListState)可以轻易地积累值大小> 2^31字节,然后在下一次检索时失败。这是目前RocksDB JNI的一个限制。

未完待续。

ref:

https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/state/state_backends.html#the-rocksdbstatebackend

猜你喜欢

转载自www.cnblogs.com/zgq25302111/p/13184978.html