The system architecture is the picture of the whole system, we have to understand is that the relationship between the accused and the various parts of the various modules and each module of the system contains. flink architecture as shown below:
As shown in FIG: flink system consists of three parts: client, jobmanager, taskmanager.
client
to write programs using each operator (map, flatmap the like) are resolved to operate, Flink using a Chain of Responsibility pattern similar manner operator combination, client according to the Chain of Responsibility pattern generator configured topology FIG Operate StreamGraph, and converted to JobGraph, that is, task list, and then submit JobGraph to jobmanager, and return.jobmanager
is responsible for scheduling and coordinating Task Job done checkpoint. After receiving from the Client to Job JAR package and other resources, it will generate the optimized execution plan and to schedule Task to the respective units to perform TaskManager.Taskmanager
关键词：task， slot，pipeline
flink中的执行单元是task slot， 每个taskmanger中包含一个或者多个slot，Taskmanger在启动的时候就设置好了槽位数slot。每个slot能启动一个 Task线程，可以运行pipeline或并行任务。 一个pipeline包含多个连续的任务，比如第n个并行任务的MapFunction和ReduceFunction。flink通常执行的都是连续任务，无论是流式计算还是批处理都是经常发生的。Taskmanager从 JobManager 处接收需要部署的 Task，部署启动后，与自己的上游建立 Netty 连接，接收数据并处理。
下面的图中，一个程序包含一个source，一个mapfunction和一个reducefunction。source和mapfunction执行的并行度为4，reducefunction的并行度为3.一个pipeline的执行顺序为source->map>reduce. 在一个2个包含3个slotTaskmanager的集群上，程序的执行图如下所示：
参考文档