Analysis [Flink] Flink Chapter 1 Flink architecture

I. Introduction

Flink runtime architecture mainly includes several parts: Client, JobManager (master node) and TaskManger (slave node).

 

 


 Second, the division of labor

Client: Flink jobs in which machine the above submission, then the current machine called Client. Program code for user-developed, it will build a DataFlow graph, and then submitted to the JobManager by the Client.

JobManager: main (master) node, which corresponds to the YARN ResourceManager, build environment generally can be done and HA. JobManager will split the task scheduler to perform TaskManager above.

TaskManager: from node (slave), TaskManager is the real part of the task.

Client submit jobs to the JobManager, you need to communicate with JobManager, Akka framework or library to use it for communication, the other Client data exchange with JobManager, using the Netty framework. Akka communication may send instructions to the JobManager based Actor System, Client, such Submit job or Cancel / update job. JobManager can also feedback to the Client, such as status updates, Statistics and results.

Client提交给JobManager的是一个Job,然后JobManager将Job拆分成task,提交给TaskManager(worker)。JobManager与TaskManager也是基于Akka进行通信,JobManager发送指令,比如Deploy/Stop/Cancel Tasks或者触发Checkpoint,反过来TaskManager也会跟JobManager通信返回Task Status,Heartbeat(心跳),Statistics等。另外TaskManager之间的数据通过网络进行传输,比如Data Stream做一些算子的操作,数据往往需要在TaskManager之间做数据传输。

当Flink系统启动时,首先启动JobManager和一至多个TaskManager。JobManager负责协调Flink系统,TaskManager则是执行并行程序的worker。当系统以本地形式启动时,一个JobManager和一个TaskManager会启动在同一个JVM中。当一个程序被提交后,系统会创建一个Client来进行预处理,将程序转变成一个并行数据流的形式,交给JobManager和TaskManager执行。

 

Guess you like

Origin www.cnblogs.com/szss/p/11011083.html