Beyond Storm, SparkStreaming - Flink how stateful computing

file

Calculated flow stateless and state into the two cases. No calculation of each individual event status observation, Storm computing framework is no state, each message to the future and back and forth does not matter, one is a. For example, our sensors power system data reception, alarm on when the voltage exceeds 240v, which is stateless data. However, if we need to determining a plurality of voltages, such as three-phase circuit, we have determined the three-phase is higher than a certain value, then it is necessary to save the state is calculated. This is because three records were sent over.

file

Storm need to implement with a state of the computation, for example by means of a memory variable or custom redis systems, ensuring low delay case is determined to achieve a state in their calculations, but does not require such Flink, and as a new generation of stream processing system, Flink very seriously.

consistency

In fact, message passing is correct. In stream processing, consistency divided into three levels.

at-most-once: at most once, you may be lost.
at-least-once: at least once, may be repeated, and when the calculation of multiple operations might affect the results.
exactly-once: to ensure exactly once, the results obtained in this way is the most accurate.

The first guarantee exactly-once system (Storm Trident and Spark Streaming), but paid a high price in terms of performance and expressiveness of these two aspects. In order to ensure exactly-once, these systems can not apply separately to each record application logic, but at the same time handle multiple (batch) records, to ensure that either all succeed each batch processing, or all fail. This leads before to get results, you must wait for a number of the recording process ends. Thus, users often have to use two stream processing frame (for a guarantee exactly-once, another process used to make latency for each element), resulting in more complicated infrastructure.

However, Flink solve this problem.

Checkpoint mechanism

Flink checkpoint is one of the most valuable innovation because it allows Flink can guarantee exactly-once, and without sacrificing performance.

Flink central role of the checkpoint is to ensure the proper state, even in the face interrupted program, but also correct. After remember this basic point, we look at checkpoint is an example of how to run. Flink provides the user with tools to define states. For example, this Scala programs are grouped by the first field (a character string) input record count state and maintains the second field.

val stream: DataStream[(String, Int)] = ... 
 
val counts: DataStream[(String, Int)] = stream   
.keyBy(record => record._1)   
.mapWithState((in: (String, Int), count: Option[Int]) =>     
  count match {       
    case Some(c) => ( (in._1, c + in._2), Some(c + in._2) )       
    case None => ( (in._1, in._2), Some(in._2) )     
})

The program has two operators: keyBy operator used to group records in accordance with the first element (a character string), the data based on the re-partition key, and then send the recording to the next operator: stateful the map operator (mapWithState). Operators map data upon receipt of each element, the second input record field added to the conventional total, then the updated elements emitted.

file

6 the input stream is recorded checkpoint barrier (checkpoint barrier) apart, all of the map operator states are 0 (count has not yet started). All key of a record will be the top map handle operator, all recording key map b will be operator of the intermediate layer processing, all the c key record will be the bottom of the map operator process.

If the input stream from the message transmission system Kafka, this isolated position is offset.

file

Checkpoint recorded as normal as a barrier to flow between the operator. When processed before the map operator 3 receives the checkpoint record and time barriers, they will state asynchronously written to stable storage.

file

When no fault occurs, Flink checkpoint overhead is extremely small, the available bandwidth determined by the speed of the checkpoint stable storage.

If the checkpoint operation fails, Flink will discard the checkpoint and continue to perform well, because after a certain checkpoint may succeed.

file

In this case, Flink will re-topology (may get a new execution resources), the input stream poured back to the last checkpoint, and then restore the state began to calculate the value and continue from there.

file

Flink input stream rewound on a checkpoint barrier position, while restoring the state value map operator. Then, Flink restart the process from here. When doing so ensure consistent after the record is processed, the status value of the map operator failure does not occur.

Flink official name of the checkpoint barrier snapshot algorithm is asynchronous (asynchronous barrier snapshotting).

Save Point

Status Version Control

Flink generated automatically by the checkpoint, the processing for re-recording on failure to correct the status. Flink user can also consciously manage state version by another feature, this feature is called save-point (savepoint).

Save points and check points work exactly the same way, except that it is triggered by the user through Flink command-line tool or the Web console manually, rather than automatically triggered by Flink, the user can restart the job from the save point, rather than starting from scratch. Another point to save the understanding that it is in the clear version of time saving application state.

file

In the figure, v.0 a certain application is running version. We were at the time t1 and t2, triggering a save point. Thus, it is possible at any time to return these two time points, and restarting the program. More importantly, you can start being modified version of the program from the save point. For example, you can modify the application code (referred to assume new version v.1), and then began to run from the time t1 altered code.

file

Use the Save Point Flink updated version of the application. The new version can be started from a save point at the old version generated.

To-end consistency

file

In this application architecture, stateful application Flink consumption data from a message queue, and writes the data output system, for the query.

Input data from Kafka, during the state of content delivery to the output of the storage system, how to ensure exactly-once it? This is called the end consistency. There are essentially two ways to achieve, which method depends on the type of the output requirements of the storage system, and applications.

(1) The first method is to buffer all output links in the sink, and upon receipt of the checkpoint record sink, the output "atomic commitment" to the storage system. This method ensures that the output result storage systems exist only guaranteed consistency, and duplication of data does not occur. In essence, the output of the storage system will be involved in the checkpoint Flink. To do this, the output storage system needs to have the ability to "atomic commitment" of.

(2) The second method is to write data to output eagerly storage system, bearing in mind that these data may be "dirty", and the need to re-process in case of failure. If a failure occurs, you need to output, input and Flink job all rolled back, so that the "dirty" data coverage, and delete already written to the output of the "dirty" data. Note that, in many cases, the deletion did not actually happen. For example, if the new record simply overwrite the old record (rather than adding to the output), then the "dirty" data is only a brief existence between checkpoints, and will eventually be corrected with the new data.

Depending on the type of output of the storage system, and Flink the corresponding connector may together end to end to ensure the consistency of, and support for multiple isolation levels.