A, flink state principle
1.1 What is the status of flink? Why state management?
flink process run computing tasks, there will be a lot of intermediate processing. During the entire run of the task in the middle there are more temporary status, for example, some of the data is performing a operator, but only half the data processing, in addition generally not enough time to deal with, which is a state.
Assuming that during operation, for some reason, the task hang, or more task flink in one task and hung up, then it will state in memory loss, this time if we do not store intermediate calculations state, then it means that when restarting the computing tasks need to start the original processed data to recalculate again. If you store the intermediate state, it can be restored to the intermediate state, and start to continue his mission from the state. This is the meaning of state management. Therefore, a mechanism is needed to save the intermediate state of record execution, this mechanism is the state management mechanism.
1.2 flink classification of state
flink includes two base status: keyed state (keyed state) and operator state (operator status)
1.2.1 keyed state
keyed with key state always together, and can only be used in keyedStream in. The state is to bind with a particular key, and each key on the KeyedStream flow, probably corresponds to a state. Combined into a unique (operator - key, state) form.
1.2.2 operator status
Keyed State with different, Operator State concurrent with a particular operator of a bound instance, the entire operator has only one state. In comparison, in an operator, there may be a number of key, so that a corresponding plurality of keyed state. And operator state can be applied in a non-keyed stream.
For example, Kafka Connector Flink is, use of the operator state. It will be for each connector instance, to save all the instances of consumer topic (partition, offset) mapping.
1.3 state of the form
All the state in the form of two: managed (trust management) and raw (raw).
Escrow is managed by the state management framework flink provided through the interface provided by the framework flink state management, and to update the values of state management. This includes a data structure for storing state data, the existing packaging and the like.
Original way is self-managed by the state of the user specific data structure, when the frame do checkpoint (checkpoint is a state data flink persistent storage mechanism), the use of byte [] to read and write the contents of the state, its internal data structures without a I know.
Usually recommended hosted in the status on DataStream, when implementing a user-defined operator, it will be used to its original state. In general, the hosting state with more.
1.4 managed way to provide interface
flink state data storage offers a lot of classes, class inheritance diagram is as follows:
FIG. 1.4 flink-- state management class inheritance FIG.
Class can be used to store state data as follows:
ValueState<T>*这保留了一个可以更新和检索的值。即类型为T的单值状态。这个状态与对应的key绑定，是最简单的状态了。它可以通过update方法更新状态值，通过value()方法获取状态值。
ListState<T>*这保存了一个元素列表，即key上的状态值为一个列表。可以追加元素并检索Iterable所有当前存储的元素。可以通过add方法往列表中附加值；也可以通过get()方法返回一个Iterable<T>来遍历状态值。
ReducingState<T>*这保留了一个值，该值表示添加到状态的所有值的聚合。接口类似于ListState，但是添加的元素使用add(T)。每次调用add方法添加值的时候，会调用reduceFunction，最后合并到一个单一的状态值。
AggregatingState<IN, OUT>*这保留了一个值，该值表示添加到状态的所有值的聚合。和ReducingState不同的是，聚合类型可能与添加到状态的元素类型不同
FoldingState<T, ACC>*这保留了一个值，该值表示添加到状态的所有值的聚合。违背ReducingState，聚合类型可能与添加到状态的元素类型不同。接口类似于ListState但是添加的元素使用add(T)使用指定的FoldFunction.这个基本弃用，请用AggregatingState代替
MapState<UK, UV>*它保存了一个映射列表。可以将键值对放入状态并检索Iterable所有当前存储的映射。映射使用put(UK, UV)或putAll(Map<UK, UV>)添加元素。使用get(UK)获取元素。映射、键和值的可迭代视图可以分别使用entries(), keys()和values()It should be noted that the above objects of the State, only for interaction (update, delete, empty, etc.) and the state, while the real value of the state, there may be the presence of memory, disk, or other distributed storage system. We just hold the equivalent of the state of the handle (state handle).
Next look, how do we get this state handle. Flink by StateDescriptordefining a state. This is an abstract class, defines the internal state of the basic information, name, type, etc. of the sequence. Corresponding to the above types of states. as follows:
ValueStateDescriptor
ListStateDescriptor
ReducingStateDescriptor
FoldingStateDescriptor
AggregatingStateDescriptor
MapStateDescriptorSecond, the use of state management flink way
2.1 The basic flow state management
以keyed state为例，
1、首先，普通Function接口是不支持状态管理的，也就是一般故障的情况下，状态并没有保存下来，后面需要将所有数据进行重新计算。如果需要支持状态管理，那么我们需要继承实现 RichFunction类。基本常用的function，flink都再封装了对应的RichFunction接口给我们使用，比如普通function中的MapFunction，对应的RichFunction抽象类为RichMapFunction。命名方式对应关系很简单，基本就是 xxxFunciotn -->RichxxxFunction。
2、接着，需要在覆盖实现RichFunction中的对应的算子方法（如map、flatMap等），里面需要实现算子业务逻辑，并将对keyed state进行更新、管理。然后还要重写open方式，用于获取状态句柄。
2.2 使用keyed state例子
下面使用ValueState为例，实现RichFlatMapFunction接口：
public class CountWindowAverage extends RichFlatMapFunction<Tuple2<Long, Long>, Tuple2<Long, Long>> {
/**
* ValueState状态句柄. 第一个值为count，第二个值为sum。
*/
private transient ValueState<Tuple2<Long, Long>> sum;
@Override
public void flatMap(Tuple2<Long, Long> input, Collector<Tuple2<Long, Long>> out) throws Exception {
// 获取当前状态值
Tuple2<Long, Long> currentSum = sum.value();
// 更新
currentSum.f0 += 1;
currentSum.f1 += input.f1;
// 更新状态值
sum.update(currentSum);
// 如果count >=2 清空状态值，重新计算
if (currentSum.f0 >= 2) {
out.collect(new Tuple2<>(input.f0, currentSum.f1 / currentSum.f0));
sum.clear();
}
}
@Override
public void open(Configuration config) {
ValueStateDescriptor<Tuple2<Long, Long>> descriptor =
new ValueStateDescriptor<>(
"average", // 状态名称
TypeInformation.of(new TypeHint<Tuple2<Long, Long>>() {}), // 当个状态数据的类型，这里是tuple,也就是元祖
Tuple2.of(0L, 0L)); // 状态默认值
//获取状态句柄
sum = getRuntimeContext().getState(descriptor);
}
}
// 主程序
env.fromElements(Tuple2.of(1L, 3L), Tuple2.of(1L, 5L), Tuple2.of(1L, 7L), Tuple2.of(1L, 4L), Tuple2.of(1L, 2L))
.keyBy(0)
.flatMap(new CountWindowAverage())
.print();
// the printed output will be (1,4) and (1,5)三、状态管理源码分析
3.1 获取state操作句柄源码分析
以2.2 中的例子为例，看状态句柄是如何获取的。
首先从这里入手：
sum = getRuntimeContext().getState(descriptor);首先RichFlatMapFunction类中没有这个方法，再到父类AbstractRichFunction，如下：
public abstract class AbstractRichFunction implements RichFunction, Serializable {
private static final long serialVersionUID = 1L;
private transient RuntimeContext runtimeContext;
public AbstractRichFunction() {
}
public void setRuntimeContext(RuntimeContext t) {
this.runtimeContext = t;
}
//返回RuntimeContext类的对象
public RuntimeContext getRuntimeContext() {
if (this.runtimeContext != null) {
return this.runtimeContext;
} else {
throw new IllegalStateException("The runtime context has not been initialized.");
}
}
...........................
}getRuntimeContext()这个方法在这里实际上是返回了StreamingRuntimeContext这个子类对象。所以调用了RichFlatMapFunction.getRuntimeContext().getState方法，最终会调用StreamingRuntimeContext.getState方法：
/* StreamingRuntimeContext */
public class StreamingRuntimeContext extends AbstractRuntimeUDFContext {
...........
public <T> ValueState<T> getState(ValueStateDescriptor<T> stateProperties) {
//这里获取到KeyedStateStore对象
KeyedStateStore keyedStateStore = this.checkPreconditionsAndGetKeyedStateStore(stateProperties);
stateProperties.initializeSerializerUnlessSet(this.getExecutionConfig());
return keyedStateStore.getState(stateProperties);
}
...............
}接着看this.checkPreconditionsAndGetKeyedStateStore，
private KeyedStateStore checkPreconditionsAndGetKeyedStateStore(StateDescriptor<?, ?> stateDescriptor) {
Preconditions.checkNotNull(stateDescriptor, "The state properties must not be null");
KeyedStateStore keyedStateStore = this.operator.getKeyedStateStore();
Preconditions.checkNotNull(keyedStateStore, "Keyed state can only be used on a 'keyed stream', i.e., after a 'keyBy()' operation.");
return keyedStateStore;
}再接着看 this.operator.getKeyedStateStore();，
/*AbstractStreamOperator*/
.....
public KeyedStateStore getKeyedStateStore() {
return this.keyedStateStore;
}
.....返回了AbstractStreamOperator.keyedStateStore变量。这个变量的初始化在AbstractStreamOperator.initializeState方法中：
public final void initializeState() throws Exception {
TypeSerializer<?> keySerializer = this.config.getStateKeySerializer(this.getUserCodeClassloader());
StreamTask<?, ?> containingTask = (StreamTask)Preconditions.checkNotNull(this.getContainingTask());
CloseableRegistry streamTaskCloseableRegistry = (CloseableRegistry)Preconditions.checkNotNull(containingTask.getCancelables());
//创建 StreamTaskStateInitializerImpl 对象
StreamTaskStateInitializer streamTaskStateManager = (StreamTaskStateInitializer)Preconditions.checkNotNull(containingTask.createStreamTaskStateInitializer());
//这里创建一个当前operator的state context对象
StreamOperatorStateContext context = streamTaskStateManager.streamOperatorStateContext(this.getOperatorID(), this.getClass().getSimpleName(), this, keySerializer, streamTaskCloseableRegistry);
//通过state context这个上下文对象获取keyed state和operator state的 backend配置
this.operatorStateBackend = context.operatorStateBackend();
this.keyedStateBackend = context.keyedStateBackend();
//初始化keyedStateStore，将从配置中创建的statebackend封装到默认的statebackend
if (this.keyedStateBackend != null) {
this.keyedStateStore = new DefaultKeyedStateStore(this.keyedStateBackend, this.getExecutionConfig());
}
this.timeServiceManager = context.internalTimerServiceManager();
CloseableIterable<KeyGroupStatePartitionStreamProvider> keyedStateInputs = context.rawKeyedStateInputs();
CloseableIterable operatorStateInputs = context.rawOperatorStateInputs();
try {
StateInitializationContext initializationContext = new StateInitializationContextImpl(context.isRestored(), this.operatorStateBackend, this.keyedStateStore, keyedStateInputs, operatorStateInputs);
this.initializeState(initializationContext);
} finally {
closeFromRegistry(operatorStateInputs, streamTaskCloseableRegistry);
closeFromRegistry(keyedStateInputs, streamTaskCloseableRegistry);
}
}In containingTask.createStreamTaskStateInitializer(), that is, StreamTask.createStreamTaskStateInitializer()there are initialization, and see
public StreamTaskStateInitializer createStreamTaskStateInitializer() {
return new StreamTaskStateInitializerImpl(this.getEnvironment(), this.stateBackend, this.timerService);
}Creating a StreamTaskStateInitializerImpl the object, which was introduced to this.stateBackend this parameter, it is estimated that get stateBackend type specified by the user to track this.stateBackendthe parameters in which initialization, and found StreamTask.invokethis method, as follows:
public final void invoke() throws Exception {
..........
this.stateBackend = this.createStateBackend();
............
}Call this.createStateBackend(), that is, StreamTask.createStateBackend()continue to look
private StateBackend createStateBackend() throws Exception {
//从application中获取已经创建的statebackend object
StateBackend fromApplication = this.configuration.getStateBackend(this.getUserCodeClassLoader());
//加载指定的 statebackend类，并返回statebackend对象
return StateBackendLoader.fromApplicationOrConfigOrDefault(fromApplication, this.getEnvironment().getTaskManagerInfo().getConfiguration(), this.getUserCodeClassLoader(), LOG);
}Continue to look at this.configuration.getStateBackendthatStreamConfing.getStateBackend
public StateBackend getStateBackend(ClassLoader cl) {
try {
//从application的config中获取"statebackend"这个key对应的object，也就是已创建的backend对象。
return (StateBackend)InstantiationUtil.readObjectFromConfig(this.config, "statebackend", cl);
} catch (Exception var3) {
throw new StreamTaskException("Could not instantiate statehandle provider.", var3);
}
}Get "statebackend" the key corresponding to the object from the application's config, that is backend object has been created. If the application does not exist, backend read from the configuration type should be used, and then loading. Look at this method StateBackendLoader.fromApplicationOrConfigOrDefault. This method is to read "state.backend" configuration item values from the config configuration, and then load the corresponding backend.
In the end, remember that this.keyedStateStore = new DefaultKeyedStateStore(this.keyedStateBackend, this.getExecutionConfig());this is this.keyedStateStorethe object encapsulated.
Back then StreamingRuntimeContext.getState, we see this.keyedStateStore.getStatethis line, the call is DefaultKeyedStateStore.getState, in fact, re-initialize it, not to proceed here.
3.2 access and update state value source analysis
First, we use here is ValueState, its immediate implementation subclasses are HeapValueState. Its data structure stored in its parent AbstractHeapStatein order to StateTable<K, N, SV> stateTablepresent in the form, where K represents the Key type, N for the namespace State (state so that the same name may belong to a different namespace), SV representative of the type of state value.