Qos分类
- At Most Once：数据最多被处理一次，这种处理机制存在数据丢失的风险；
- At Least Once：数据最少被处理一次，存在数据重复计算的问题；
- Exactly Once：数据仅被处理一次，计算结果精准，代价是延迟增大；
目前Apache Flink 支持At Least Once和Exactly Once两种Qos。
适用场景
- At Least Once：实时性要求高，结果存在一定误差的场景（可能偏大）；
- Exactly Once：结算结果要求精准，容忍一定延迟的场景；
实现原理
- At Least Once：Operator跳过流对齐，接受到数据后就开始计算，计算完成后数据流向下游，这就导致checkpoint n+1的数据可能被划分到checkpoint n中，因此如果中间过程找出现错误，重新恢复时可能出现重复计算，由于没有流对齐，因此实时性好。
When the alignment is skipped, an operator keeps processing all inputs, even after some checkpoint barriers for checkpoint n arrived. That way, the operator also processes elements that belong to checkpoint n+1 before the state snapshot for checkpoint n was taken. On a restore, these records will occur as duplicates, because they are both included in the state snapshot of checkpoint n, and will be replayed as part of the data after checkpoint n.
- Exactly Once：Operator会进行流对齐，Barrier先到的输入流数据先被缓存起来直到所有数据流的Barrier到达才开始进行计算，由于存在等待的过程，因此延迟增高；
As soon as the operator receives snapshot barrier n from an incoming stream, it cannot process any further records from that stream until it has received the barrier n from the other inputs as well. Otherwise, it would mix records that belong to snapshot n and with records that belong to snapshot n+1.Streams that report barrier n are temporarily set aside. Records that are received from these streams are not processed, but put into an input buffer.Once the last stream has received barrier n, the operator emits all pending outgoing records, and then emits snapshot n barriers itself.
端到端Exactly-once
端到端Exactly-once是指从Source到Sink整条处理链路保证Exactly-once语义，需要外部Soruce和Sink的支持。
- Source支持重置消费位点，回溯消费；
- Sink 需要感知整体Checkpoint的完成，并在整体Checkpoint完成时候将计算结果发送出去；
参考：