There are two main reasons why the state should only be maintained for a limited time. Imagine a Flink application, which receives the user login event stream, and stores the related event information and timestamp of the last login for each user to improve the experience of high-frequency access users.
Control the size of the state. The main use scenario of the state survival time feature is to be able to effectively manage the ever-increasing state size. Normally, data only needs to be saved temporarily, for example, the user is in a network connection session. When the user access event ends, there is actually no need to save the user's state to reduce unnecessary state storage space. The background state cleaning mechanism based on time-to-live introduced by Flink 1.8.0 allows us to automatically clean up useless data. Previously, application developers had to take additional actions and explicitly delete useless states to free up storage space. This manual cleaning process is not only error-prone, but also inefficient. Take the above user login case as an example, because the relevant information of these inactive users will be automatically expired and cleared, we no longer need to store the timestamp of the last login.
Meets (sensitive) data protection requirements. With the development of data privacy regulations (such as the General Data Protection Regulation GDPR promulgated by the European Union), compliance with the relevant requirements of such regulations or sensitive data processing has become the top priority of many applications. Typical cases of such usage scenarios include the need to save data only for a certain period of time and prevent the data from being accessed again later. This is a common challenge for companies that provide short-term services to customers. The state lifetime feature ensures that applications can be accessed only for a limited time, which helps to comply with data protection laws and regulations.
Both of these requirements can be solved by the state survival time. This function can periodically and continuously delete the key value in the state when the key value becomes unimportant and no longer needs to be stored in the storage.
Continuous cleaning of application state
Version 1.6.0 of Apache Flink introduced the state lifetime feature. It enables developers of stream processing applications to configure the state of the operator so that it will be cleared after the defined lifetime expires. In Flink 1.8.0, this function has been further expanded to continuously clean up the old data of RocksDB and the backend ( FsStateBackendand MemoryStateBackend) of the state of the heap memory .
In Flink's DataStreamAPI, the application state is defined by a state descriptor. The state lifetime is configured by passing the StateTtlConfigurationobject to the state descriptor. The following Java example demonstrates how to create a state-to-live configuration and provide it to the state descriptor, which saves the user's last login time as a Longvalue:
import org.apache.flink.api.common.state.StateTtlConfig;
import org.apache.flink.api.common.time.Time;
import org.apache.flink.api.common.state.ValueStateDescriptor;
StateTtlConfig ttlConfig = StateTtlConfig
.newBuilder(Time.days(7))
.setUpdateType(StateTtlConfig.UpdateType.OnCreateAndWrite)
.setStateVisibility(StateTtlConfig.StateVisibility.NeverReturnExpired)
.build();
ValueStateDescriptor<Long> lastUserLogin =
new ValueStateDescriptor<>("lastUserLogin", Long.class);
lastUserLogin.enableTimeToLive(ttlConfig);
Flink provides multiple options to configure the state lifetime:
When is the time to live reset? By default, when the state is modified, the time to live will be updated. We can also update the survival time of related items when the read operation accesses the state, but this will take an additional write operation to update the timestamp.
Can the expired data be accessed? The state lifetime mechanism uses a lazy strategy to clear out expired states. This may cause the application to try to read the expired but not yet deleted state. The user can configure whether to return an expired status to such a read request. In either case, the expired status will be deleted immediately afterwards. Although returning the expired state is beneficial to data availability, not returning the expired state is more in line with the requirements of relevant data protection regulations.
Which time semantics is used to define time to live? In Apache Flink 1.8.0, users can only define state survival time based on Processing Time. It is planned to support Event Time in a future version of Flink.
关于状态生存时间的更多信息，可以参考 Flink官方文档。
在实现上，状态生存时间特性会额外存储上一次相关状态访问的时间戳。虽然这种方法增加了一些存储开销，但它允许 Flink 在访问状态、创建检查点、恢复或存储清理过程时可以检查过期状态。
“取走垃圾数据”
在访问状态对象时，Flink 将检查其时间戳，并在状态过期时清除状态（是否返回过期状态，则取决于配置的过期数据可见性）。由于这种访问时才删除的特性，除非被垃圾回收，否则那些永远不被访问过期数据将仍然占用存储空间。
那么，在没有显示处理过期状态的情况下，如何删除这些数据呢？通常，我们可以配置不同的策略进行后台删除。
保证完整快照中不包含过期数据
Flink 1.6.0 已经支持在创建检查点（Checkpoint）或保存点（Savepoint）的完整快照时不包含过期状态。需要注意的是，创建增量快照时并不支持剔除过期状态。完整快照时的过期状态剔除必须如下例所示进行显示启用：
StateTtlConfig ttlConfig = StateTtlConfig
.newBuilder(Time.days(7))
.cleanupFullSnapshot()
.build();
上述配置并不会影响本地状态存储的大小，但是整个作业的完整快照的大小将会减小。只有当用户从快照重新加载其状态到本地时，才会清除用户的本地状态。
由于上述这些限制，在 Flink 1.6.0 中程序仍需要过期后主动删除状态。为了改善用户体验， Flink 1.8.0 引入了两种自主清理策略，分别针对两种状态后端类型：
堆内存状态后端的增量清理
此方法只适用于堆内存状态后端（FsStateBackend和MemoryStateBackend）。其基本思路是在存储后端的所有状态条目上维护一个全局的惰性迭代器。某些事件（例如状态访问）会触发增量清理，而每次触发增量清理时，迭代器都会向前遍历删除已遍历的过期数据。以下代码示例展示了如何启用增量清理：
StateTtlConfig ttlConfig = StateTtlConfig
.newBuilder(Time.days(7))
// check 10 keys for every state access
.cleanupIncrementally(10, false)
.build();
如果启用该功能，则每次状态访问都会触发清除。而每次清理时，都会检查一定数量的状态条目是否过期。其中有两个调整参数。第一个定义了每次清理时要检查的状态条目数。第二个参数是一个标志位，用于表示是否在每条记录处理（Record processed）之后（而不仅仅是访问状态，State accessed），都还额外触发清除逻辑。关于这种方法有两个重要的注意事项：首先是增量清理所花费的时间会增加记录处理的延迟。其次，如果没有状态被访问（State accessed）或者没有记录被处理（Record processed），过期的状态也将不会被删除。
RocksDB 状态后端利用后台压缩来清理过期状态
如果使用 RocksDB 状态后端，则可以启用另一种清理策略，该策略基于 Flink 定制的 RocksDB 压缩过滤器（Compaction filter）。RocksDB 会定期运行异步的压缩流程以合并数据并减少相关存储的数据量，该定制的压缩过滤器使用生存时间检查状态条目的过期时间戳，并丢弃所有过期值。
使用此功能的第一步，需要设置以下配置选项：state.backend.rocksdb.ttl.compaction.filter.enabled。一旦配置使用 RocksDB 状态后端后，如以下代码示例将会启用压缩清理策略：
StateTtlConfig ttlConfig = StateTtlConfig
.newBuilder(Time.days(7))
.cleanupInRocksdbCompactFilter()
.build();
需要注意的是启用 Flink 的生存时间压缩过滤机制后，会放缓 RocksDB 的压缩速度。
使用定时器进行状态清理
另一种手动清除状态的方法是基于 Flink 的计时器，这也是社区评估的一个想法。使用这种方法，将为每个状态访问注册一个清除计时器。这种方法的清理更加精准，因为状态一旦过期就会被立刻删除。但是由于计时器会与原始状态一起存储会消耗空间，开销也更大一些。
未来展望
除了上面提到的基于计时器的清理策略之外，Flink 社区还计划进一步改进状态生存时间特性。可能的改进包括为事件时间（Event Time）添加生存时间的支持（目前只支持处理时间）和为可查询状态（Queryable state）启用状态生存时间机制。
总结
状态可访问时间的限制和应用程序状态大小的控制，是状态流处理领域的常见挑战，Flink 的 1.8.0 版本通过添加对过期状态对象连续性后台清理的支持，显著改进了状态生存时间特性。新的清理机制可以不再需要手动实现状态清理的工作，而且由于惰性清理的机制，执行效率也更高。总得来说，状态生存时间方便用户控制应用程序状态的大小，使得用户可以将精力集中在应用程序的核心逻辑开发上。