Table of contents
SlotSelectionStrategy (Slot selection strategy)
Resource abstraction
The resources involved in Flink are divided into two levels: cluster resources and Flink's own resources. Cluster resources manage hardware resources, including CPU, memory, GPU, etc., which are managed by the resource management framework (yarn, k8s, Mesos). Flink applies for and releases resources from the resource management framework.
Flink applies for a resource container (Yarn's Container or K8S's Pod) from the resource management framework, and runs one TaskManager process in one container. Container resources are relatively coarse-grained for Flink. For example, a single container can use 1 CPU core and 8G memory. Due to different computing types, one task occupying one container may not fully utilize the resources, so a single container will be used by multiple containers. Flink task sharing
Flink divides the applied resources, and each part is called Task Slot, as shown below
In versions before 1.14, it was coarse-grained resource management (version 1.14 added fine-grained resource management, which was still a beta version in Flink, see FLIP-169, FLIP-156 and FLIP-56 for details), and the resources of one TaskMnanager were allocated the same amount. Divided into n parts (n comes from the flink-conf.yaml configuration file), that is to say, up to n Tasks can be run on the TaskManager. In the above figure, 1 TaskManager contains 3 TaskSlots.
Generally speaking, there are three main roles in Flink's resource scheduling framework, namely JobMaster (hereinafter referred to as JM), ResourceManager (hereinafter referred to as RM) and TaskManager. The tasks written by the user will first be compiled into JobGraph, and then the resources will be injected and submitted to JM. The role of JM is to manage the resource application and execution deployment of JobGraph.
The scheduling-related component in JM is Scheduler, which generates a series of SlotRequests based on JobGraph, then aggregates these SlotRequests to generate a ResourceRequirement and sends it to RM. After RM receives the resource declaration, it will first check the existing resource capabilities in the cluster. If it can meet its needs, it will send a request to TM to ask him to offer slot to the corresponding JM (the allocation of slots here is completed by the SlotManager component). If the existing resources are not enough, it will apply for new resources from external K8s or Yarn through the internal driver. Finally, after JM receives enough slots, it will start deploying operators and the job can run.
ResourceManager
effect:
-
Apply for a container to start a new TM, or apply for a slot for a job
-
Handle the abnormal exit of JobManager and TaskManager
-
Cache the TaskManager (i.e. container) and wait for a period of time before releasing unused containers to avoid repeated application and release of resources.
-
Heartbeat awareness of JobManager and TaskManager
There are 3 ResourceManager Drivers built into Flink, which correspond to different resource management frameworks.
SlotManager
SlotManager faces different objects and provides different functions
-
Provide management actions such as registration, unregistration, and idle exit for TaskManager.
-
For Flink jobs, receive Slot requests and releases, resource reports, etc. When resources are insufficient, SlotManager temporarily stores resource requests in the waiting queue. SlotManager notifies ResourceManager to apply for more resources and start a new TaskManager. After TaskManager registers with SlotManager, SlotManager will have new resources, which are sequentially obtained from the waiting queue. Allocating resources.
SlotSelectionStrategy (Slot selection strategy)
When Flink decides which TaskManager a Task will run on, it will make a selection based on the strategy. There are different selection strategies when selecting Slot. SlotSelecionStrategy is the interface defined by the strategy. Its class system is as shown below.
Selection strategies are generally divided into two categories
-
LocationPreferenceSlotSelectionStrategy
Position-first strategies are divided into two categories:
-
Default strategy: DefaultLocationPreferenceSlotSelectionStrategy. This strategy does not consider the even distribution of resources. It will select the first one from the available Sloy set that meets the conditions, and so on.
-
Balanced strategy: EvenlySpreadOutLocationPreperenceSlotSelectionStrategy. This strategy considers the balanced allocation of resources. It will select the Slot with the most remaining resources from the available Slot set that meets the conditions, and try to allow each TaskManager to bear the computing pressure in a balanced manner.
2. Slot priority selection strategy PreviousAllocationSlotSelectionStrategy has been assigned
If there are currently no free allocated slots, the location-limited strategy will still be used to allocate and apply for slots.
SlotPool (Slot resource pool)
JobMaster's scheduler first obtains Slots from SlotPool to schedule tasks. When SlotPool does not have enough Slot resources to execute the job, it will first try to obtain resources from ResoucureManager. If ResourceManager is currently unavailable, ResouceManager rejects the resource request or the request times out, the resource If the application fails, the job startup fails.
After the JobMaster applies for resources, it will hold the Slot locally to avoid the failure of the job due to ResourceManager exceptions. For batch processing, holding the resource JobMaster can first avoid requesting resources from the ResourceManager multiple times. At the same time, the unavailability of the ResourceManager will not affect the continued execution of the job. Only insufficient resources will cause the job execution to fail.
When the job has been executed or the job is fully started and there are remaining resources, the JobMaster will return the remaining resources to the ResourceManager
Slot sharing group
Shared slot conditions: subtasks of different tasks of the same job
Flink has two types of sharing groups
1.SlotSharingGroup
Non-mandatory sharing constraints. Slot sharing checks whether there is a shared Slot based on the JobVerties ID in the group. Just make sure that the same JobVertex ID cannot appear in a shared Slot.
Among the Slots that meet the resource requirements, find a Slot that does not have the same JobVertex ID, and select a Slot according to the Slot selection strategy. If there is no Slot that meets the conditions, apply for a new Slot.
2.CoLocationGroup
CoLocationGroup, also known as local constraint sharing group, has mandatory slot sharing restrictions. CoLocationGroup is used in iterative operations, that is, called in the API of IterativeStream. Tasks in iterative operations must share the same Slot of TaskManager. CoLocationGroup can be seen as a special case of SlotSharingGroup.
It should be noted here that during the conversion process from JobGraph to ExecutionGraph, each ExecutionVertex is given a number written according to the degree of parallelism. The iteratively calculated ExecutionVertex with the same number will be put into the local shared constraint group and share the same CoLocationContraint object. During scheduling When , you can find the Slot information of other Tasks in this group based on the number.
CoLocation sharing checks whether the assigned Slot with the same CoLocationConstraint constraint is available based on the CoLocationConstraint associated with each ExecutionVertex in the group. When scheduling job execution, first find the TaskManager deployed by other Tasks in this constraint. If not, apply for a new Slot. If there is one, share the Slot on the TaskManager.