[Big Data] Detailed Explanation of Flink (6): Source Code Part I

This series includes:

• [Big Data] Detailed Explanation of Flink (1): Basics
• [Big Data] Detailed Explanation of Flink (2): Core Part I
• [Big Data] Detailed Explanation of Flink (3): Core Part II
• [Big Data] Detailed Explanation of Flink (4): Core Part III
• [Big Data] Detailed Explanation of Flink (5): Core Part IV
• [Big Data] Detailed Explanation of Flink (6): Source Code Part I

---

55. What is the submission process of Flink jobs?

Flink's submission process:

• In , start the function in , generate Flink StreamGraph and JobGraph Flink Clientthrough reflection , and submit the JobGraph to the Flink cluster.jarmain
• After the Flink cluster receives the JobGraph ( JobManagerreceived), it translates the JobGraph into an ExecutionGraph, then starts scheduling, and starts consuming data after the startup is successful.

To sum up: Flink core execution process, calls to user APIs can be converted to StreamGraph→ JobGraph→ ExecutionGraph.

56. How many ways are there for Flink job submission?

Flink's job submission is divided into two ways:

• Local mode : that is, the local submission mode, which runs the code directly in IDEA.
• Remote submission method : divided intostandalonemethod,yarnmethod,K8smethod. Among them,yarnthe method is divided into three submission modes:yarn-per-jobmode,yarn-sessionmode,yarn-applicationmode.

57. When is the Flink JobGraph generated?

StreamGraph and JobGraph are all Flink Clientgenerated on the client side, that is, before submitting to the cluster. The schematic diagram is as follows:

58. What process did JobGraph go through before submitting the cluster?

• Users start the Flink cluster, use the command line to submit jobs, and run them flink run -c WordCount xxx.jar.
• After running the command line, the entry will be invoked through runthe script CliFrontend, which will trigger the method in the file CliFrontendsubmitted by the user , and then hand it over to the method, and finally trigger a specific execution according to the submission mode.jarmainPipelineExecuteorexecutePipelineExecutor
• According to the specific PipelineExecutorexecution, the user's code will be compiled to generate StreamGraph, and Jobgraph will be generated after optimization.

The specific flow chart is as follows:

59. Seeing that you mentioned PipeExecutor, what implementation classes does it have?

PipeExecutorIn Flink, it is called a pipeline executor . It is an interface and Flink Clientan important part of submitting jobs to the cluster after JobGraph is generated. As mentioned earlier, there are several ways to submit jobs to the cluster, the most commonly used method is yarnthe method, yarnwhich includes $3$ submission modes, mainly usingsessionmode,per-jobmode. applicationIn the mode, the JobGraph is generated in the cluster.

So PipeExecutorthe implementation class of is as shown in the figure below: (press in the code and CTRL+Hit will come out)

In addition to the above two modes in the red box, use it when running Flink MiniCluster in the IDEA environment for debugging LocalExecutor.

60. What are the characteristics of the Local submission mode, and how is it realized?

Local is the submission method running in the local IDEA environment. Not on the cluster. Mainly used for debugging, the schematic diagram is as follows:

• Flink programs JobClientare submitted by .

• JobClientSubmit assignments to JobManager.

• JobManagerResponsible for coordinating resource allocation and job execution. After the resource allocation is completed, the task will be submitted to the corresponding TaskManager.

• TaskManagerStarts a thread to start executing, TaskManagerand will JobManagerreport status changes to , such as started, in progress, or completed.

• After the job execution is complete, the results are sent back to the client.

Source code analysis: through Flink $\mathrm{1.12.2}$ The source code is analyzed.

(1) Create and get the corresponding StreamExecutionEnvironmentobject: LocalStreamEnvironment.

Invoke StreamExecutionEnvironmentthe object's executemethod.

(2) Get the StreamGraph.

(3) Execute specific PipeLineExecutorGet localExecutorFactory.

(4) Get the JobGraph.

Generate JobGraph according to the implementation class LocalExecutor of localExecutorFactory.

The above part is all Flink Clientgenerated in . Since it is submitted in Local mode, a MiniCluster cluster will be created next, and miniCluster.submitJobthe jobGraph to be submitted will be specified by .

(5) Instantiate the MiniCluster cluster.

(6) Return to JobClientthe client.

After executing the above miniCluster.submitJobto submit the JobGraph to the local cluster, a JobClientclient will be returned, which JobClientcontains some detailed information of the application, including the JobID, the status of the application, and so on. Finally, return to the previous layer of code execution, and the corresponding class is StreamExecutionEnvironment.

The above is the source code execution process of the Local mode.

61. What are the remote submission modes?

Remote submission method: divided into Standalone method , Yarn method , and K8s method .

• Standalone : Containssessionthe schema.
• The Yarn method is divided into three submission modes:yarn-per-jobmode,yarn-Sessionmode,yarn-applicationmode.
• K8s way : containssessionmode.

62. Briefly introduce the Standalone mode?

The Standalone mode is the stand-alone version submission method of the Flink cluster . Only one node is used for submission, and the Session mode is commonly used.

Submit the command as follows:

bin/flink run org.apache.flink.WordCount xxx.jar

• ClientThe client submits tasks to JobManager.
• JobManagerResponsible for applying for the resources required for task operation and managing tasks and resources.
• JobManagerDistribute tasks to TaskManagerexecute.
• TaskManagerRegularly JobManagerreport status to .

63. Tell me about the yarn cluster submission method?

Submission through the yarn cluster is divided into $3$ ways to submit:

• sessionmodel
• per-jobmodel
• applicationmodel

64. What are the characteristics of yarn-session mode?

Submit the command as follows:

./bin/flink run -t yarn-session \
-Dyarn.application.id=application_XXXX_YY xxx.jar

yarn-session mode : All jobs share cluster resources, poor isolation, JM load bottleneck, andmainmethods are executed on the client. It is suitable for short tasks with short execution time and frequent execution. There is only one job in the clusterJobManager. In addition, Jobs are randomly assignedTaskManager.

Features: session-clusterThe mode needs to start the cluster first, then submit the job, and then apply for a space from Yarn, and the resources will remain unchanged forever. If the resource is full, the next job cannot be submitted, and the next job can only be submitted normally after one of the jobs in Yarn is completed and the resources are released. All jobs share Dispatcherand ResourceManagershare resources, which is suitable for jobs with small scale and short execution time.

65. What are the characteristics of yarn-per-job mode?

Commit command:

./bin/flink run -t yarn-per-job --detached xxx.jar

yarn-per-job mode : Each job starts the cluster independently, with good isolation, JM load balancing, andmainthe method is executed on the client. Inper-jobmode, there is one for each JobJobManager,TaskManagerand only a single Job for each.

Features: A task corresponds to a job. Each submitted job will apply for resources from Yarn separately according to its own situation until the job is completed. The failure of a job will not affect the normal submission and operation of the next job. Exclusive Dispatcherand ResourceManager, resource applications are accepted on demand. Suitable for large-scale long-running jobs.

66. What are the characteristics of yarn-application mode?

Submit the command as follows:

./bin/flink run-application -t yarn-application xxx.jar

yarn-application mode : Each job starts the cluster independently, with good isolation, JM load balancing, andmainmethodsJobManagerare executed on .

In yarn-per-jobboth the and yarn-sessionmodes, the client needs to perform the following three steps, namely:

• Get the dependencies required by the job;
• By performing an environmental analysis and obtaining a logical plan, ie StreamGraph→ JobGraph;
• Upload dependencies and JobGraph to the cluster.

Only after these are completed, env.execute()the Flink runtime will be triggered to actually start executing the job through the method. If all users submit jobs on the same client, larger dependencies will consume more bandwidth, and the translation of more complex job logic into JobGraph will also consume more CPU and memory, and the resources of the client will instead become bottleneck.

In order to solve it, the community implemented the Application mode on the basis of the traditional deployment mode. The three things that originally required the client to do have been transferred to JobManager, that is to say, main()the method is executed in the cluster (where the entry point is located ApplicationClusterEntryPoint), and the client only needs to be responsible for initiating the deployment request.

To sum up, the Flink community recommends using yarn-per-jobthe or yarn-applicationmode to submit applications.

67. Tell me about the yarn-session submission process in detail?

The submission flow chart is as follows:

1. Start the cluster

• Flink ClientYarn ResourceManagerSubmit task information to .
  • Flink ClientUpload the application configuration ( Flink-conf.yaml, logback.xml, log4j.properties) and related files (Flink Jar, configuration class files, user Jar files, JobGraph objects, etc.) to the distributed storage HDFS.
  • Flink ClientYarn ResourceManagerSubmit task information to .
• Yarn starts the Flink cluster, do $2$ steps:
  • By Yarn Clientsubmitting Yarn ResourceManageran application for Flink to create a cluster to , Yarn ResourceManagerallocate Container resources, and notify the corresponding to NodeManagerstart one ApplicationMaster(every time a Flink Job is submitted, one will be started ApplicationMaster), ApplicationMasterwhich will include the currently to be started JobManagerand the internal use of Flink itself ResourceManager.
  • JobManagerRuns in the process as YarnSessionClusterEntryPointthe entry point for cluster startup. Initialization Dispatcher, which is used internally by Flink ResourceManager, starts related RPC services, and waits for Flink ClientJobGraph to be submitted through the Rest interface.

2. Job submission

• Flink ClientDispatcherSubmit the compiled JobGraph to via Rest . DispatcherIt is a Rest interface and is not responsible for the actual scheduling and assignment work.

• DispatcherAfter receiving the JobGraph, create one for the job JobMaster, hand over the job JobMaster, JobMasterbe responsible for job scheduling, manage the life cycle of the job and Task, and build the ExecutionGraph (the parallelized version of the JobGraph, the core data structure of the scheduling layer).

After the above two steps are executed, the job enters the scheduling execution stage.

3. Job scheduling execution

• JobMasterApply for resources from ResourceManagerand start scheduling ExecutionGraph.

• ResourceManagerAdd resource requests to the waiting queue, and YarnResourceManagerstart the process by applying for a new Container through heartbeat TaskManager.

• YarnResourceManagerStart, and then load Jar files and other required related resources from HDFS, start in the container TaskManager, and TaskManagerstart TaskExecutor.

• TaskManagerAfter starting, ResourceManagerregister with , and report your Slot resource status to ResourceManager.

• ResourceManagerTake out the Slot request from the waiting queue, TaskManagerconfirm the availability of resources to , and tell to TaskManagerwhich Slot is allocated to JobMaster.

• TaskManagerReply to JobMastera Slot of yourself that belongs to your task, and JobMaserthe Slot will be cached in the SlotPool.

• JobMasterSchedule the Task to TaskMnagerbe executed on the Slot.

68. Tell me about the yarn-per-job submission process in detail?

Submit the command as follows:

./bin/flink run -t yarn-per-job --detached xxx.jar

The submission flow chart is as follows:

1. Start the cluster

• Flink ClientYarn ResourceManagerSubmit task information to .
  • Flink ClientUpload the application configuration ( Flink-conf.yaml, logback.xml, log4j.properties) and related files (Flink Jar, configuration class files, user Jar files, JobGraph objects, etc.) to the distributed storage HDFS.
  • Flink ClientYarn ResourceManagerSubmit task information to .
• Yarn starts the Flink cluster, do $2$ steps operation.
  • By Yarn Clientsubmitting Yarn ResourceManageran application for Flink to create a cluster to , Yarn ResourceManagerallocate Container resources, and notify the corresponding to NodeManagerstart one ApplicationMaster(every time a Flink Job is submitted, one will be started ApplicationMaster), ApplicationMasterwhich will include the currently to be started JobManagerand the internal use of Flink itself ResourceManager.
  • JobManagerRuns in the process as YarnJobClusterEntryPointthe entry point for cluster startup. Initialization Dispatcher, which is used internally by Flink ResourceManager, starts related RPC services, and waits for Flink ClientJobGraph to be submitted through the Rest interface.

2. Job submission

• ApplicationMasterStart Dispatcher, Dispatcherstart ResourceManagerand JobMaster(this step JobMasteris different from Session, it is Dispatcherpulled by rather than passed by Client). JobMasterResponsible for job scheduling, managing the life cycle of jobs and Tasks, and constructing ExecutionGraph (the parallelized version of JobGraph, the core data structure of the scheduling layer).

After the above two steps are executed, the job enters the scheduling execution stage.

3. Job scheduling execution

• JobMasterResourceManagerApply for Slot resources from and start scheduling ExecutionGraph.

• ResourceManagerAdd resource requests to the waiting queue, and YarnResourceManagerstart the process by applying for a new Container through heartbeat TaskManager.

• YarnResourceManagerStart, then load Jar files and other required related resources from HDFS, and start in the container TaskManager.

• TaskManagerStart internally TaskExecutor.

• TaskManagerAfter starting, ResourceManagerregister with , and report your Slot resource status to ResourceManager.

• ResourceManagerTake out the Slot request from the waiting queue, TaskManagerconfirm the availability of resources to , and tell to TaskManagerwhich Slot is allocated to JobMaster.

• TaskManagerReply to JobMastera Slot of yourself that belongs to your task, and JobMaserthe Slot will be cached in the SlotPool.

• JobMasterSchedule the Task to TaskMnagerbe executed on the Slot.