In terms of state management, Flink uses Checkpoint and Savepoint to achieve state fault tolerance. When the calculation scale of Flink changes, it can automatically redistribute the state between parallel instances. The bottom layer uses the State Backend strategy to store the calculation state, and the State Backend determines the way and location of state storage.
Flink state in the state management of all operations can be divided Keyed Stateand Operator State.
The state of the Keyed State type is bound to the key one by one, and can only be used in KeyedStream. All non-KeyedStream state operations are called Operator State. When Flink performs state management at the bottom level, it <parallel-operator-instance, key>associates the Keyed State with the Keyed State . Since a key only falls into one of the operator-instances, it can be simply understood that the Keyed State is <operator,key>bound to it, and the Key Group mechanism is used to perform the Keyed State. For management or classification, all keyed-operators may need to interact with 1~n Key Groups when doing state operations.
When Flink distributes the Keyed State state, it does not use the key as the unit, but the Key Group as the smallest unit.
Operator State (also called non-keyed state ), each operator state is bound to a parallel operator instance.
Keyed State and Operator State exist in two forms ( managed and raw ). All operators known by Flink support Managed State, but Raw State is only used when user-defined Operators, and does not support The state is redistributed when the degree of parallelism changes. Therefore, although Flink supports Raw State, in most application scenarios, Managed State is generally used.
Regarding how to use the six keyedStates of flink, the blogger’s previous blog posts have been shared in java and scala. You can click on the link: https://blog.csdn.net/qq_44962429/article/details/104428236 to view.
This article mainly introduces flink's TTL, State backend, and Clear Sate. The examples are described in scala language.
1、flink TTL（Time To Live）
Flink can assign state time to live (TTL) to any type of keyed-state. If TTL is configured and the state value has expired, Flink will try to clear the stored historical state value.
Configuration introduction:
import org.apache.flink.api.common.state.StateTtlConfig
import org.apache.flink.api.common.state.ValueStateDescriptor
import org.apache.flink.api.common.time.Time
val ttlConfig = StateTtlConfig
.newBuilder(Time.seconds(5))
.setUpdateType(StateTtlConfig.UpdateType.OnCreateAndWrite)
.setStateVisibility(StateTtlConfig.StateVisibility.NeverReturnExpired)
.build
val stateDescriptor = new ValueStateDescriptor[String]("text state", classOf[String])
stateDescriptor.enableTimeToLive(ttlConfig)
① newBuilder(Time.seconds(5)): Set the expiration time of the state to expire in 1s.
② setUpdateType(StateTtlConfig.UpdateType.OnCreateAndWrite): Set the modification strategy of the state expiration time. The sample adopts the modification of the expiration time when creating and reading. For example, when the state is created at 10:00:00, it is queried at 10:00:02. Or if the state is modified, the expiration time of the state will be re-timed at 10:00:02, and the state will expire at 10:00:07.
③ setStateVisibility(StateTtlConfig.StateVisibility.NeverReturnExpired): Set the visibility of the expired state, and never regret the expired data.
Note : After enabling TTL, the system will consume additional memory to store the time stamp (Processing Time). If the user has not enabled the TTL configuration before, modify the code to enable TTL before startup, and the system will not start up when the state is restored, and it will throw Compatibility failure and StateMigrationException exception.
2、Expired State
By default, only when the expired state is clearly read, the expired data will be cleared by calling the ValueState.value() method, which means that if the system has not read the expired state, it will not be deleted. This may cause the file storing state data to continue to grow.
Next, we will introduce several clear state strategies supported by flink.
（1）Cleanup in full snapshot
The system will load all the state snapshots from the time point of the last state recovery. During the loading process, the out-of-date data will be removed. This will not affect the state data stored on the disk. The state data will only be used during Checkpoint. Overwrite, but still can't solve the problem of automatically clearing expired and unused data at runtime.
import org.apache.flink.api.common.state.StateTtlConfig
import org.apache.flink.api.common.time.Time
val ttlConfig = StateTtlConfig
.newBuilder(Time.seconds(1))
.cleanupFullSnapshot
.build
Note : Because this clearing method will only be triggered when the state is restored, and can only be used for backend implementation of memory or snapshot state, it does not support RocksDB State Backend, so it is not recommended in the actual production environment.
（2）Cleanup in background
The background clearing strategy can be turned on, and the default clearing strategy is adopted according to the State Backend (the backend storage in different states has different clearing strategies)
import org.apache.flink.api.common.state.StateTtlConfig
val ttlConfig = StateTtlConfig
.newBuilder(Time.seconds(1))
.cleanupInBackground
.build
Based on stateBackend, it is divided into two types, one is based on memory backend , and the other is based on RocksDB
- memory backend
import org.apache.flink.api.common.state.StateTtlConfig
val ttlConfig = StateTtlConfig.newBuilder(Time.seconds(5))
.setUpdateType(UpdateType.OnCreateAndWrite)
.setStateVisibility(StateVisibility.NeverReturnExpired)
.cleanupIncrementally(100,true) // 默认值 5 | false
.build()
The first parameter means that every time cleanup is triggered, the system will process 100 elements at a time. The second parameter is false, which means that as long as the user operates on any state, the system will trigger the cleanup strategy; the second parameter is true, which means that the cleanup strategy will be triggered as long as the system receives the number of records (even if the user has no operating state).
- RocksDB
RocksDB is an embedded key-value storage, where key and value are arbitrary byte streams. The bottom layer is asynchronously compressed, and the data with the same key is compacted (compressed) to reduce the size of the state file, but it does not deal with expired data. The state is cleaned up, so you can configure compactFilter to allow RocksDB to exclude expired state when compacting. This filtering feature of the RocksDB database is disabled by default. If you want to enable it, you can configure the state in flink-conf.yaml. backend.rocksdb.ttl.compaction.filter.enabled:true or set RocksDBStateBackend::enableTtlCompactionFilter in the application's API.
The following pictures describe the process of compact and compactFilter.
import org.apache.flink.api.common.state.StateTtlConfig
val ttlConfig = StateTtlConfig.newBuilder(Time.seconds(5))
.setUpdateType(UpdateType.OnCreateAndWrite)
.setStateVisibility(StateVisibility.NeverReturnExpired)
.cleanupInRocksdbCompactFilter(1000) //默认配置1000
.build()
The 1000 here means that when the system is doing Compact, it will check whether 1000 elements are invalid, and if it fails, the expired data will be cleared.
3、state backend
State Backend determines how Flink stores system state information (in the form of Checkpoint). Currently, Flink provides three implementations of State Backend.
- Memory (JobManagwer): This is the default implementation of Flink. It is usually used for testing. The system will store the calculation state in the memory of JobManagwer. However, in the actual production environment, due to the large number of calculation states, using Memory can easily lead to OOM (Out of memory).
- FileSystem: The system stores the calculation status in the memory of the TaskManager, so it is generally used as a production environment. The system will back up the TaskManager status data on the file system according to the CheckPoin mechanism. If it is a very large-scale cluster, TaskManager memory may also overflow.
- RocksDB: The system will store the calculation status in the memory of the TaskManager. If the TaskManager memory is not enough, the system can use RocksDB to configure the local disk to complete the status management, and support the backup of the local status data to the remote file system. Therefore, RocksDB Backend is recommended s Choice.
Each job can configure its own back-end implementation of state storage
var env=StreamExecutionEnvironment.getExecutionEnvironment
val fsStateBackend:StateBackend = new FsStateBackend("hdfs:///xxx") //MemoryStateBackend、FsStateBackend、RocksDBStateBackend
env.setStateBackend(fsStateBackend)
Considering that the state of the production environment may be relatively large, it is recommended to use RocksDBStateBackend.
Well, the blogger mentioned at the beginning of the article. Flink state is divided into keyedstate and operator state. Keyedstate has been shared in the previous blog post. If you need it, please click https://blog.csdn.net/qq_44962429 /article/details/104428236 View, in the next sharing, the blogger will introduce the operator state.