Three scheduling strategies for yarn
FIFO
FIFO Scheduler arranges applications into a queue in the order of submission. This is a first-in first-out queue. When resource allocation is performed, resources are allocated to the top application in the queue first, and then the requirements of the top application are met. Give the next assignment, and so on.
Capacity
Capacity (capacity) scheduler, there is a dedicated queue for running small tasks, but setting up a queue for small tasks will pre-occupy certain cluster resources, which causes the execution time of large tasks to lag behind when using the FIFO scheduler time.
Fair
In the Fair (fair) scheduler, we do not need to occupy certain system resources in advance. The Fair scheduler will dynamically adjust system resources for all running jobs. When the first big job is submitted, only this job is running, and it has all the cluster resources; when the second small job is submitted, the Fair scheduler will allocate half of the resources to this small task, so that these two Tasks share cluster resources fairly.
It should be noted that in the Fair scheduler shown below, there will be a certain delay from the submission of the second task to the acquisition of resources, because it needs to wait for the first task to release the occupied Container. After the execution of the small task is completed, the resources occupied by it will be released, and the large task will obtain all the system resources. The final effect is that the Fair scheduler not only obtains high resource utilization, but also ensures that small tasks are completed in time.