Table of contents
Gracefully stop the job to create the final savepoint
Three modes to submit and view assignments
Submit and view assignments in Session mode
Submit and view jobs in Per-Job Cluster mode
Application mode to submit and view jobs
Flink provides a command-line interface (CLI) bin/Flink for running programs packaged as JAR files and controlling their execution. The CLI is part of any Flink setup and can be used both in local single-node setups and in distributed setups. It connects to the running JobManager specified in conf/flink-conf.yaml.
CLI list
| operate | use |
| yarn-session.sh | Use yarn-session.sh to start a resident Flink cluster to accept tasks submitted by clients. |
| run | Submit Flink jobs. Jobs can be submitted to a resident Flink cluster or run in stand-alone mode. |
| run-application | Execute jobs in Application mode. |
| info | Optimized execution graph for print jobs. Again, the jar for the job needs to be passed . |
| list | List all jobs (including JobID ) |
| savepoint | Used to create or process savepoints for a given job. If the state.savepoints.dir parameter is not specified in conf/flink-conf.yaml , it may be necessary to specify a savepoints directory outside the JobID . |
| cancel | cancel job |
| stop | stop job (streaming jobs only) |
Job lifecycle management
A prerequisite for using the Flink CLI is to have a running Flink cluster deployed on Kubernetes , YARN or any other available option. You can start the Flink cluster locally and try commands on your own machine.
submit a job
Submitting a job means uploading the job's JAR and related dependencies to the Flink cluster and starting the job execution. For example, we chose a long-running job such as examples/streaming/StateMachineExample.jar . You can choose any other jar file from the examples/ folder to deploy your own job .
$ ./bin/flink run \
--detached \
./examples/streaming/StateMachineExample.jarSubmitting a job with --detached will cause the command to return after the commit is complete, with the output containing the ID of the newly submitted job .
Job has been submitted with JobID cca7bc1061d61cf15238e92312c2fc20
The printed usage information lists job-related parameters, which can be added to the end of the job submission command if desired.
There is also a run-application command that can be used to run jobs in Application Mode , which works similarly to run .
The run and run-application commands support accepting additional configuration parameters via -D . For example, the maximum degree of parallelism for a job can be set by setting -Dpipeline.max parallelism=120 . This parameter is useful in per-job or application mode, because you can pass any configuration parameters to the cluster without changing configuration files.
When submitting a job in session mode, only configuration parameters are supported.
Job detection
You can detect any running Job with the following command :
$ ./bin/flink listAll jobs that have been submitted but not executed are listed under Scheduled Jobs.
Create Savepoints
Savepoints can be created to save the current state of a job. Just JobID:
$ ./bin/flink savepoint \
$JOB_ID \
/tmp/flink-savepointsThe savepoint folder is optional and needs to be specified if state.savepoints.dir is not set.
The path to the savepoint can be used later to restart the Flink job.
delete savepoint
To delete a savepoint, --need to add the command --dispose with the corresponding savepoint path :
$ ./bin/flink savepoint \
--dispose \
/tmp/flink-savepoints/savepoint-cca7bc-bb1e257f0dab \
$JOB_IDIf using a custom state instance (such as a custom reducing state or a RocksDB state), you must specify the path to the program JAR used to trigger the savepoint . Otherwise, you will encounter a ClassNotFoundException :
$ ./bin/flink savepoint \
--dispose <savepointPath> \
--jarfile <jarFile>Triggering Savepoint processing via the Savepoint operation not only deletes the data from storage, but also causes Flink to clean up the savepoint- related metadata.
terminate job
Gracefully stop the job to create the final savepoint
This is a more graceful way to end a running streaming job and thus the flow from source to sink . When the user requests to stop the job, all sources will be requested to send the last checkpoint barrier , thereby triggering the checkpoint , and after successfully completing the checkpoint , they will complete the termination of the job by calling cancel () .
$ ./bin/flink stop \
--savepointPath /tmp/flink-savepoints \
$JOB_IDSuspending job "cca7bc1061d61cf15238e92312c2fc20" with a savepoint.
Savepoint completed. Path: file:/tmp/flink-savepoints/savepoint-cca7bc-bb1e257f0dab
If state.savepoints.dir is not set , the savepoint folder must be specified with --savepointPath .
If the --drain flag is specified , MAX_WATERMARK will be emitted before the last checkpoint barrier . This will trigger all registered event time timers, clearing any state waiting for a specific watermark, such as windows. The job will run until all sources are properly shut down. This allows the job to finish processing all in-flight data so that when stopped, data after the savepoint can be processed.
If you want to terminate the job permanently, use the --drain flag. Do not use --drain if you wish to resume the job at a later time , as this may lead to erroneous results.
cancel job
Canceling a job can be done with the cancel operation:
$ ./bin/flink cancel $JOB_IDCancelling job cca7bc1061d61cf15238e92312c2fc20.
Cancelled job cca7bc1061d61cf15238e92312c2fc20.
The status of the corresponding job will transition from "Running" to "Canceled". Any calculations will stop.
Start a job from a savepoint
A job can be started from a savepoint using the run (and run-application ) action.
$ ./bin/flink run \
--detached \
--fromSavepoint /tmp/flink-savepoints/savepoint-cca7bc-bb1e257f0dab \
./examples/streaming/StateMachineExample.jarIt looks like how this command is identical to the initial run command except for the --fromSavepoint parameter, which is used to reference the status of a previously stopped job, which will generate a new job ID that can be used for maintenance jobs .
By default, we try to match the entire savepoint state to the job being submitted. If you wish to allow skipping savepoint states that cannot be restored, you can set the --allowNonRestoredState flag. If an operator that was part of the program was removed from the program when the savepoint was triggered, and you still want to use the savepoint, you need to allow this.
$ ./bin/flink run \
--fromSavepoint <savepointPath> \
--allowNonRestoredState ...This is useful if your program removes operators that were part of savepoints.
Three job submission modes
| model |
describe |
features |
| Session model |
The Seesion mode will create a Flink cluster according to the resource parameters you set , and all jobs will be submitted to this cluster to run. The cluster will not be released automatically after the job finishes running. For example, if an exception occurs in a job and a Task Manager is shut down, all other jobs running on the Task Manager will fail. In addition, since there is only one Job Manager in the same cluster , as the number of jobs increases, the pressure on the Job Manager will increase accordingly. |
Advantages: When submitting jobs, the time overhead caused by resource allocation is smaller than other modes. Disadvantages: Since all jobs run in the cluster, there will be competition for resources and interaction between jobs. Based on the above characteristics, this mode is suitable for deploying jobs that require short startup time and relatively short running time. |
| Per-Job Cluster model |
When using the Per-Job Cluster mode, each time a Flink job is submitted , YARN will start a new Flink cluster for the job, and then run the job. When the job finishes running or is canceled, the Flink cluster to which the job belongs will also be released. |
Advantages: resource isolation between jobs, abnormal behavior of one job will not affect other jobs. Because each job has a one-to-one correspondence with a Job Manager , there will be no problem that a Job Manager will be overloaded due to running multiple Jobs . Disadvantage: Every time a job is run, a dedicated Flink cluster must be started, and the overhead of starting the job is greater. Based on the above characteristics, this mode is usually suitable for long-running jobs. |
| Application mode |
When using the Application mode, each time a Flink Application is submitted (an Application contains one or more jobs), YARN will start a new Flink cluster for the Application . When the Application ends or is canceled, the Flink cluster to which the Application belongs will also be released. The difference between this mode and the Per-Job mode is that the main() method in the JAR package corresponding to the Application will be executed in the Job Manager in the cluster . If the submitted JAR package contains multiple jobs, these jobs will be executed in the cluster to which the Application belongs. |
Advantages: It can reduce the burden on the client when submitting jobs. Disadvantages: Every time a Flink Application is run , a dedicated Flink cluster must be started, and the time spent on starting the Application will be greater. |
Three modes to submit and view assignments
Submit and view assignments in Session mode
- Execute the following command to start the YARN session.
yarn-session.sh --detached- Execute the following command to submit the job.
flink run /opt/apps/FLINK/flink-current/examples/streaming/TopSpeedWindowing.jarNote: This article uses the TopSpeedWindowing example provided by Flink itself, which is a long-running streaming job.
After the submission is successful, the YARN Application ID of the submitted Flink job will be returned. Information similar to the following is returned.
- Execute the following command to view the job status.
flink list -t yarn-session -Dyarn.application.id=<application_XXXX_YY>You can also view job status through the Web UI.
- Execute the following command to stop the job.
flink cancel -t yarn-session -Dyarn.application.id=<application_XXXX_YY> <jobId>Submit and view jobs in Per-Job Cluster mode
- Execute the following command to submit the job.
flink run -t yarn-per-job --detached /opt/apps/FLINK/flink-current/examples/streaming/TopSpeedWindowing.jarAfter the submission is successful, the YARN Application ID of the submitted Flink job will be returned. Information similar to the following is returned.
- Execute the following command to view the job status.
flink list -t yarn-per-job -Dyarn.application.id=<application_XXXX_YY>Note that <application_XXXX_YY> in the example in this article is the Application ID returned after the job runs.
You can also view job status through the Web UI. For details, see Viewing Job Status Through the Web UI.
- Execute the following command to stop the job.
flink cancel -t yarn-per-job -Dyarn.application.id=<application_XXXX_YY> <jobId>Application mode to submit and view jobs
- Execute the following command to submit the job.
flink run-application -t yarn-application /opt/apps/FLINK/flink-current/examples/streaming/TopSpeedWindowing.jarAfter the submission is successful, the YARN Application ID of the submitted Flink job will be returned. Information similar to the following is returned.
- Execute the following command to view the job status.
flink list -t yarn-application -Dyarn.application.id=<application_XXXX_YY>You can also view job status through the Web UI. For details, see Viewing Job Status Through the Web UI.
- Execute the following command to stop the job.
flink cancel -t yarn-application -Dyarn.application.id=<application_XXXX_YY> <jobId>