Flink blog series, based on Flink1.6 , intend divided into three parts: Principles, source code, and examples of API usage analysis, after the latter get another complete series of blog directory.
This series of blog is my own learning process, some understanding, if not correct, inaccuracies welcome everyone to share the message. Cited in the text have been marked, if infringement, please contact me immediately delete!
1 Introduction
Speaking before Flink basic structure, we first know what Flink that? Chinese official website explanation: Apache Flink is distributed and a frame processing engine for borderless bounded and for calculating a state where the data flow [1]. There are no boundaries on and define the boundaries of the data stream can be explained with reference to the official website, can understand from the explanation is a framework and Flink calculation engine is used to handle the data stream. Flink processes the data stream does not mean that we usually refer only be used for stream processing system, where the data flow primarily to illustrate the way Flink processing data, in the form of streams conducted. In fact, not only stream processing, Flink can do a batch that can be achieved Flink batch / flow integration, as to how to achieve it, just to mention concepts, follow-up blog will slowly come.
2, Flink basic architecture
2.1, Flink basic architecture diagram [2]:
From the figure, Flink The whole system consists of two components: JobManager, TaskManager assembly, which is to follow the main architecture - is, for the master node from the JobManager point design principle, the TaskManager the Slave node (also called work node), by means of communications between components Akka Framework.
1) JobManager: responsible for scheduling and resource allocation across Flink cluster tasks. After obtaining jobs submitted from a client, JobManager TaskManager situation according to resource use (TaskSlots), allocate resources and command TaskManager start the task. In this process, JobManager will trigger the checkpoint operation, TaskManager perform checkpoint operation in which all of the checkpoint is done in a coordinated process of JobManager. In addition, if the mission failed, it failed to restore mission coordinated by the JobManager.
2) TaskManager: responsible for resource management and application specific task execution and nodes, data exchange between multiple nodes is executed on TaskManager. Flink cluster, each worker (taskManager) corresponds to a JVM process.
In other words, JobManager allocation of resources, tasks, TaskManager have the resources to start the task. Generally in a production environment, where nodes TaskManager JobManager and should be isolated, and its main purpose is to ensure TaskManager (calculated based on the memory) does not grab the JobManager resources.
3) client client: After not part of the runtime, in other words, Flink cluster startup task client submitted, client-side when the client can be disconnected, is not required. Unlike JobManager client node and corresponds TaskManager flink cluster (or a physical machine or a virtual machine, or the container), it is executed by a trigger abstraction, where the program when executed in the node JobManager, called client on the JobManager node, similarly, it may be TaskManager junction point.
Submit a task of the normal process is: client and JobManager build Akka connection, will submit the task to the JobManager, JobManager according to the resource has been registered in the JobManager in TaskManager case (TaskSlot), assign tasks to the availability of resources TaskManager, and ordered TaskManager start task, TaskManager from JobManager need to accept the task of subordinates, using slot resources to start task, to establish a data access network connection, and then accept the data and begin processing.
2.2、taskSlot
Each task slot is part of the TaskManager, if a taskManager three taskSlot, then this will be divided equally three taskSlot TaskManager of this resource ( memory only, not including the CPU ). There are multiple slot means that the same JVM have multiple sub-tasks, which will share a TCP connection and heartbeat JVM. This is to be described, the slot number of the number of subtask is not one to one, there may be a plurality of slot subtask. By default, a job in the same sub-task (subtask) can be shared in a slot. This involves the concept of slot sharing, follow-up blog analysis.
reference:
[1]https://flink.apache.org/zh/flink-architecture.html
[2]https://ci.apache.org/projects/flink/flink-docs-release-1.6/concepts/runtime.html