We know that Flink provides a fault-tolerant mechanism that can resume tasks when the application fails. This mechanism is mainly achieved by continuously generating snapshots. Flink snapshots mainly include two parts of data, one part is data stream data, and the other part is operator status data. The implementation of the corresponding snapshot mechanism consists of two main parts, one is the barrier and the other is the state. Because of the data flow processed by Flink, the data continues to flow in the DAG topology of multiple operators. If you want to realize that a snapshot at a certain time can be used for system failure recovery, you must ensure that this snapshot can fully determine the state at a certain time, before this time. All of the data have been processed, and none of the subsequent data has been processed. Here the concept of barrier is introduced. Here we mainly introduce the barrier implementation.
1. Barrier
A core element in Flink distributed snapshots is the stream barrier. These barriers are injected into the data stream and flow with the data as part of the data stream. The barrier does not hold any data, but flows linearly like data. You can see that the barrier divides the data stream into two parts of data (in fact, multiple consecutive parts), one is the data of the current snapshot, and the other is the data of the next snapshot. Each barrier will carry its snapshot ID. The data of this snapshot is in front of this barrier. From the figure, the data moves from left to right (the one on the right enters the system first), then the data contained in snapshot n is the data from the right to the next barrier (n-1), one of the two gray vertical lines in the figure Part of checkpoint n. In addition, the barrier does not interrupt the flow of data, so the barrier is very lightweight. At the same time, multiple snapshots can be in the same data stream, which means that multiple snapshots can be generated at the same time.
In the case of multiple input data streams, barriers of multiple data streams will be inserted into the data stream at the same time. The point where the barrier of snapshot n is inserted into the data stream (we call it Sn) is a certain position in the data stream (including all the data before the current moment), which is the snapshot of this part of the data contained . For example, in Kafka, this location is the offset of the last record of the partition. This position Sn will be reported to the checkpoint coordinator (Flink JobManager).
Then the barrier began to flow downwards. When an intermediate operator receives the snapshot n barrier of all its input sources, it will transmit a snapshot n barrier to all its output streams. Once a sink operator receives the barrier n of all input data streams, it Will send a snapshot n confirmation to the checkpoint coordinator. When all sinks have confirmed snapshot n, the system considers that the data of the current snapshot has been completed.
Once the snapshot n has been executed, the task will no longer request the data before Sn, because at this moment, these data have completely passed the data flow topology.
Second, the alignment mechanism
Operators that receive more than one data input need to align the input data stream based on barriers . Details are as follows: the
entire flow chart is shown below
Then we look at one by one:
-
When the operator receives the barrier n of the snapshot, it cannot directly process the subsequent data, but needs to wait for the barrier n of other input snapshots. Otherwise, the data of snapshot n will be mixed with the data of snapshot n+1. The first one in the figure shows that the operator is about to receive data stream 1 (the above is regarded as data stream 1 (6, 5, 4, 3, 2, 1), and the following is regarded as data stream 2). , 1, 2, 3 arrive at the operator after barrier n. At this time, if the data stream 1 continues to be processed, then the operator will contain the data (1, 2, 3) after the barrier n, but the operator is receiving and processing at this moment Data stream 2, data (a, b, c) will be mixed with data stream 1 (1, 2, 3).
- The data stream of snapshot n will be temporarily put aside. The data obtained from these data streams will not be processed, but stored in a buffer. The first one in the figure shows that because the barrier n of data stream 2 has not yet arrived, the operator continues to receive 1, 2, and 3 but does not do any processing. But you need to store 1, 2, and 3 in the buffer. At this time, the second data stream receives a, b, then sends it directly, and sends c after receiving c.
- Once the last data stream receives snapshot n, opertor will send out all blocked data and send out its own barrier. As shown in the third figure, the operator finally receives the barrier n of another data stream, and then sends out a, b, c (c, b, a in the operator in the figure), and then sends out its own barrier. This time A 4 is added to the buffer, which becomes (4, 3, 2, 1).
- After that, the operator will restart processing all input data streams, first process the data in the buffer, and then process the data of the input data stream after processing. As shown in the fourth figure, the 1, 2, 3, and 4 in the buffer are processed first, and the data from these two data sources is received and processed.
