Principle parsing | Exactly-Once processing Apache Flink combined Kafka build end

Article Directory:

Apache Flink application Exactly-Once semantics
Flink application Exactly-Once-end semantics
Flink example application starts pre-submission stage
Flink implemented in the two-phase commit Operator
to sum up

Apache Flink since December 2017 released version 1.4.0 began, introduced an important milestone for the flow characteristic calculation: TwoPhaseCommitSinkFunction (related Jira). It extracts the general logic of the two-phase commit protocol, such that the program Exactly-Once constructed by end Flink possible. Some data support the source (source) and output terminal (sink), comprising Apache Kafka 0.11 and later. It provides a layer of abstraction, the user need only a few methods can be achieved from end to end semantics Exactly-Once.

See the documentation for the use of TwoPhaseCommitSinkFunction: TwoPhaseCommitSinkFunction. Or you can directly read Kafka 0.11 sink documents: kafka.

The next will be a detailed analysis of this new functionality and implementation logic Flink, divided into the following points.

Description Flink checkpoint mechanism how Exactly-Once Flink is to ensure that the results of the program
Flink commit protocol shows how the data source and data output by the two-stage interaction, providing guaranteed end-Exactly-Once
Through a simple example of how to use the file output TwoPhaseCommitSinkFunction achieve Exactly-Once the

A, Apache Flink application Exactly-Once semantics

When we say " Exactly-Once when" refers to each event entered once only affect the final outcome. Even if the machine or software failure, neither duplicate data, it will not lose data.

It provides Exactly-Once semantics Flink before long. In the past few years, we have had in-depth description of the checkpoint mechanism Flink, which is the ability to provide core Flink Exactly-Once semantics. Flink document provides a comprehensive overview of the function.

Before continuing, look under the brief introduction to the checkpoint mechanism, which is on the back of understanding of the subject is essential.

Second checkpoint is consistent snapshot of the following:
The current state of the application
The position of the input stream

Flink a fixed time point may be configured to periodically generate checkpoint, the checkpoint data is written to persistent storage system, e.g. S3 or HDFS. The checkpoint data is written to the persistent store occurs asynchronously, which means Flink application can continue processing data in the checkpoint process.

If the machine or software failure, restart, Flink application recovery process from the latest checkpoint point; Flink will restore application state, will enter the last checkpoint rolled back into the saved position, and then start running again. This means Flink can be like that never happened as the result of the calculation fault.

Before Flink 1.4.0, Exactly-Once semantics Flink limited to internal applications, and does not extend to the majority of the external data processing system Flink after transmission. Flink various applications and interact with the data output terminal, the developer needs to have the ability to maintain context component to ensure Exactly-Once semantics.

To Exactly-Once-end semantics - that is, except the application internal Flink, Flink external system is also required to meet the writing Exactly-Once semantics - these external systems to provide a method must commit or rollback, then Flink the checkpoint mechanism to coordinate.

Distributed systems, a common method is to coordinate the commit and rollback two-phase commit protocol. In the next section, we will discuss how the TwoPhaseCommitSinkFunction Flink is the use of two-phase commit protocol to provide end-to-Exactly-Once semantics.

Two, Flink application Exactly-Once-end semantics

We will introduce two-phase commit protocol, as well as how to read and write it in a Flink program Kafka's end to end Exactly-Once semantics. Kafka is a popular messaging middleware, often used in conjunction with Flink. Kafka adds support for transactions in the most recent version 0.11. This means that now read and write Kafaka by Flink, and provide end-Exactly-Once semantics with the necessary support.

Flink support for end-to-Exactly-Once semantics is not limited to Kafka, you can source it provides / outputs necessary coordination mechanisms with any used together. For example Pravega, apart from Origin media storage system DELL / EMC, but also to support end to end semantics Exactly-Once through the TwoPhaseCommitSinkFunction Flink.

In this example, the program of today's discussion, we have:

Kafka read from a data source (Flink built KafkaConsumer)
Polymerization window
Kafka write data back to the data output terminal (Flink built KafkaProducer)

For data output providing assurance Exactly-Once, all the data that must be submitted to Kafka by a transaction. Bundled with all submitted data to be written between the two checkpoint. This ensures that the data in the event of a failure can be rolled back written. However, in a distributed system, usually have multiple tasks running concurrently write simple commit or rollback is not enough, because all components must be "consistent" when committed or rolled back to ensure consistent results. Flink using two-phase commit protocol and submit a pre-stage to solve this problem.

In the checkpoint beginning, that is, two-phase commit protocol "pre-commit" phase. When the checkpoint started, Flink JobManager the checkpoint will Barrier (the recorded data stream into the current checkpoint into a checkpoint with the next) injection stream.

brarrier passed between operator. For each operator, it triggers operator status snapshot written to the state backend.

Kafka consumption saving the data source offset (offset), then pass to the next checkpoint barrier operator.

In this way only apply to operator has "internal" status. The so-called internal state storage means and Flink state backend management - sum value, for example, in the second window operator counted out polymerization. When a process has its internal state, in addition to the data written to the state backend changes before the checkpoint, no need to pre-submission stage to do anything else. Flink is responsible for submitting written in the case of these successful checkpoint correct, or suspend these write in case of failure.

Third, the example Flink application starts pre-submission stage

However, when the process has "external" state, you need to make some additional processing. Normally occurs in the form of an external write external system (e.g., Kafka) a. In this case, in order to provide assurance Exactly-Once, the system must support external affairs, so as to integrate and two-phase commit protocol.

In the example herein, data needs to be written Kafka, and therefore the data output terminal (Data Sink) external state. In this case, the pre-commit phase, in addition to its state write state backend, the data output terminal must also submit the transaction in advance outside thereof.

When the checkpoint barrier in all operator are passed again, and trigger a checkpoint callback is successfully completed, the pre-submission phase ends. All state trigger snapshots are considered part of the checkpoint. checkpoint is a snapshot of the state of the entire application, including outside the state submitted in advance. If a failure occurs, we can roll back to a point in time snapshot of the last successful completion.

The next step is to notify all operator, checkpoint has been successful. This is a two-phase commit protocol submission stage, JobManager issue checkpoint has completed callback for the application of each operator.

Widnow operator data source and no external state, during the commit phase, the operator need not perform any operation. However, the data output terminal (Data Sink) has an external state where the external transaction should be submitted.

We summarize the above knowledge points:

Once all the operator completed pre-submission, submitted a commit.
If there is at least a pre-commit fails, all other submissions will be discontinued, we will roll back on a successful completion of the checkpoint.
After a successful pre-commit, commit to submit the need to ensure the ultimate success - operator and external systems are required to protect this. If the commit fails (eg, due to intermittent network problems), Flink entire application fails, the application will be restarted according to the restart policy for the user, but also try to re-submit. This process is crucial, because if the commit ultimately unsuccessful, will result in data loss.

Therefore, we can determine the operator all agree that the end result checkpoint: All operator agree that data has been submitted, or the submission is aborted and rolled back.

Fourth, implement two-phase commit Operator in the Flink

Complete implementation of two-phase commit protocol may be a bit complicated, which is why it's common Flink logical reason to extract the abstract class TwoPhaseCommitSinkFunction.

Then a simple example of the output to a file-based, explains how to use TwoPhaseCommitSinkFunction. Users only need to implement four functions can be achieved Exactly-Once semantics of data output:

beginTransaction -, we create a temporary file in the temporary directory of the target file system before the start of the transaction. Then, we can write data to the file when processing data.
preCommit - in the pre-submission stage, we reload the files to the memory, close the file, not re-written. We will also belong to any subsequent written document next checkpoint to start a new business.
commit - file in the commit phase, we will pre-submission stage atomically move to the real destination directory. It should be noted that this delay will increase the visibility of the output data.
abort - in stoppage stage, we delete the temporary files.

We know that if any fault, Flink will take place in the state to restore the application to the latest in a checkpoint point. An extreme case is that pre-submission was successful, but has failed to inform the commit before reaching the operator. In this case, the operator of the state Flink will have to return to pre-commit, but the state has not really committed.

We need to save in the pre-submission phase checkpoint enough information to state, in order to correct abort or commit the transaction after the restart. In this case, the information is the path to a temporary file and the target directory.

TwoPhaseCommitSinkFunction have put this into account, and state and recovery from checkpoint point, will give priority to issue a commit. We need to realize submit to power, etc., generally speaking, this is not difficult. In this example, we can identify this situation: Temporary files are not in the temporary directory, but has moved to the target directory.

In TwoPhaseCommitSinkFunction, there are a number of other border situation will be taken into account, please refer Flink documentation for more information.

to sum up

This article summarizes some key points covered:

Flink's checkpoint mechanism is to support two-phase commit protocol and provide end to end basis Exactly-Once semantics.
The advantage of this scheme is: Flink Unlike some other systems such as, for storing data transmitted through the network - not like the program calculates the most batch as written to disk at each stage.
Flink's TwoPhaseCommitSinkFunction extract the general logic of the two-phase commit protocol, based on the combination of this Flink and external transactional systems, build end-Exactly-Once possible.
From Flink 1.4.0, Pravega and Kafka 0.11 producer provides Exactly-Once semantics; Kafka was first introduced in version 0.11 of the transaction, offers the possibility of providing Exactly-Once semantics for use in Kafka producer Flink program.
Kafaka 0.11 producer of the transaction is implemented in TwoPhaseCommitSinkFunction basis, and at-least-once producer compared to an increase of only very low overhead.

This is an exciting feature, and look forward Flink TwoPhaseCommitSinkFunction support more data in the future the receiving end.

via： www.ververica.com/blog/end-to…

作者: Peter Nowojski

Translation | Zhou Kaibo

Zhou Kaibo, Alibaba technical expert, Master of Sichuan University, graduating in 2010 后加入阿里 search division, engaged in research and development platform for offline search, the search will participate in reconstruction of background data processing architecture of MapReduce to Flink. Currently Computing Platform Division at Ali, focused on the one-stop computing platform based on Flink's construction.