The Flink runtime architecture mainly includes four different components, which will work together when running stream processing applications;
job manager (JobManager), resource manager (ResourceManager), task manager (TaskManager), and dispatcher (Dispatcher) ). Because Flink is implemented in Java and Scala, all components will run on the Java virtual machine. The responsibilities of each component are as follows:
Job Manager (JobManager)
The main process that controls the execution of an application, that is, each application will be controlled by a different JobManager for execution.
JobManager will first receive the application to be executed. This application will include: job graph (JobGraph), logic The data flow graph (logical dataflow graph) and the jar package
JobManager that packs all the classes, libraries and other resources will convert the JobGraph into a physical data flow graph. This graph is called the "Execution Graph" and contains All tasks that can be executed concurrently.
The JobManager will request the resource manager (ResourceManager) for the resources necessary to perform the task, that is, the slot on the Task Manager (TaskManager). Once it has obtained enough resources, it will distribute the execution graph to the taskmanager that actually executes them. During the operation, the JobManager will be responsible for all operations that require central coordination, such as the coordination of checkpoints.
Resource Manager (ResourceManager)
It is mainly responsible for managing the slot of the TaskManager. The TaskManager slot is a processing resource unit defined in flink.
flink provides different resource managers for different environments and resource management tools, such as YARN, Mesos, and K8S have been deployed standalone.
When JobManager applies for slot resources2, ResourceManager will assign TaskManagers with free slots to JobManager. If the ResourceManager does not have enough slots to satisfy the JobManager request, it can also initiate a session to the resource providing platform to provide a container to start the TaskManager process
Dispatcher
It can run across jobs. It submits a REST interface for applications.
When an application is submitted for execution, the dispatcher will start and transfer the application to a JobManager.
Dispatcher will also start a Web UI to conveniently display and monitor job execution information.
Dispatcher may not be necessary in the architecture, depending on how the application is submitted and run
Principle of task scheduling:
The concept of parallelism:
TaskManager and Slots
Parallel subtask allocation
Program and data flow (DataFlow)
Execution Graph (ExecutionGraph)
Data transmission form
Task Chain (Operator Chains)
