Hadoop learning tutorial (YARN) (5)

1. YARN resource scheduler

Yarn is a resource scheduling platform that is responsible for providing server computing resources for bit computing programs. It is equivalent to a distributed operating system platform, and computing programs such as MapReduce are equivalent to applications running on the operating system.

1.1. Yarn infrastructure

YARN is mainly composed of ResourceManager, NodeManager, ApplicationMaster, Container and other components.
1) The main functions of ResourceManager (RM)
(1) Process client requests
(2) Monitor NodeManager
(3) Start or monitor ApplicationMaseter
(4) Resource allocation and scheduling
2) The main functions of NodeManager (NM) are as follows
(1) Manage a single node Resources
(2) Process commands from ResourceManager
(3) Process commands from ApplicationMaster
3) Main functions of ApplicationManager (AM)
(1) Apply for resources for applications and assign them to internal tasks
(2) Task monitoring and fault tolerance
4) Container
Container is a resource abstraction in YARN. It encapsulates multi-dimensional resources on a certain node, such as memory, CPU, disk network, etc.

1.2. Yarn working mechanism

(1) The MR program is submitted to the node where the client is located
(2) YarnRunner applies for an Application from ResourceManager.
(3) RM returns the resource path of the application to YarnRunner.
(4) The program submits the resources required for operation to HDFS.
(5) After submitting the program resources, apply to run mrAppMaster.
(6) RM initializes the user's request into a task.
(7) One of the NodeManagers receives the task.
(8) The NodeManager creates a container Container and generates MRAppmaster.
(9) Container copies resources from HDFS to local.
(10) MRAppmaster applies to RM for running MapTask resources.
(11) RM assigns the task of running MapTask to two other NodeManagers, and the other two NodeManagers receive the tasks and create containers respectively.
(12) MR sends the startup program script to the two NodeManagers that received the task, and the other two NodeManagers receive the tasks and create containers respectively.
(13) MrAppMaster waits for all MapTasks to finish running and applies for a container from RM to run the ReduceTask.
(14) ReduceTask obtains the data of the corresponding partition from MapTask.
(15) After the program is completed, MR will apply to RM for cancellation.

1.3. The whole process of homework submission

(1) Job submission
1. CLient calls the job, waitForCompletion method to submit MapReduce jobs to the entire cluster.
2. Client applies for a job ID from RM.
3. RM returns the submission path and job ID of the job resource to the Client.
4. Client submits the jar package, slicing information and configuration files to the specified resource submission path.
5. After the Client submits the resources, it applies to RM to run MrAppMaster.
(2) Job initialization
6. After RM receives the Client's request, it adds the job to the capacity scheduler.
7. An idle NM receives the job.
8. The NM creates a Container and generates an MRAppmaster.
9. Download the resources submitted by the Client to the local area.
(3) Task allocation
10. MrAppMaster applies to RM for resources to run multiple MapTask tasks.
11. RM assigns the task of running MapTask to the other two NodeManagers, and the other two NodeManagers receive the tasks and create containers respectively.
(4) Task running
12. MR sends the program startup script to the two NodeManagers that received the task. The two NodeManagers start MapTask respectively, and MapTask sorts the data partitions.
13. MrAppMaster waits for all MapTasks to finish running, then applies for a container from RM and runs the ReduceTask.
14. ReduceTask obtains the data of the corresponding partition from MapTask.
15. After the program is completed, MR will apply to RM for cancellation.
(5) Progress and task status updates.
Tasks in YARN return their progress and status (including counter) to the application manager. The client requests progress updates from the application manager every second (set by mapreduce.client.progressmonitor.pollinterval). displayed to the user.
(6) Job completion
In addition to requesting job progress from the application manager, the client will call waitForCompletion() every 5 seconds to check whether the job is completed. The time interval can be set through mapreduce.client.completion.pollinterval. After the job is completed, the application manager and Container will clean up the job status. Job information will be stored by the job history server for user inspection.

1.4. Yarn scheduler and algorithm

There are currently three main types of Hadoop job schedulers, FIFO, capacity (Capacity Scheduler) and fairness (Fair Scheduler). The default resource scheduler of Apache Hadoop3.1.3 is the Capacity Scheduler.
The default scheduler of the CDH framework is Fair Scheduler.
For detailed settings, see: yarn-default.xml file

<property>
    <description>The class to use as the resource scheduler.</description>
    <name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
</property>

1.4.1. First-in-first-out scheduler (FIFO)

FIFO scheduler (First In First Out): Single queue, first come first served according to the order in which jobs are submitted.

Disadvantages: Does not support multiple queues and is rarely used in production environments.

1.4.2. Capacity Scheduler

Capacity Scheduler is a multi-user scheduler developed by Yahoo.
Capacity scheduler features:
1. Multiple queues, each queue can be configured with a certain amount of resources, and each queue adopts FIFO scheduling strategy.
2. Capacity guarantee: Administrators can set minimum resource guarantees and resource usage upper limits for each queue.
3. Flexibility: If there are remaining resources in a queue, they can be temporarily shared with queues that need resources. Once a new application is submitted to the queue, the resources borrowed by other queues will be returned to the queue.
4. Multi-user: Supports multiple users to share the cluster and multiple applications to run at the same time.
In order to prevent one user's jobs from monopolizing resources in the queue, the scheduler limits the amount of resources occupied by jobs submitted by the same user.

Capacity Scheduler Scheduler Resource Allocation Algorithm
1) Queue resource allocation:
Starting from the root, the depth first algorithm is used, and the queue with the lowest resource occupancy rate is given priority to allocate resources.
2) Job resource allocation
By default, resources are allocated according to the priority of submitted jobs and the order of submission time.
3) Container resource allocation Allocate
resources according to the priority of the container. If the priorities are the same, the principle of data locality is followed:
(1) Tasks and data are on the same node
(2) Tasks and data are on the same rack
(3) Tasks and data are not on the same rack The same node is no longer the same rack

1.4.3. Fair Scheduler

Fair Scheduler is a multi-user scheduler developed by Facebook.

1) The same points as the capacity scheduler
(1) Multi-queue: supports multiple queues and multiple jobs.
(2) Capacity guarantee: Administrators can set minimum resource guarantees and resource usage upper limits for each queue.
(3) Flexibility: If there are remaining resources in a queue, they can be temporarily shared with those queues that need the resources. Once a new application is submitted to the queue, the resources borrowed by other queues will be returned to the queue.
(4) Multi-tenancy: Supports multi-user shared clusters and simultaneous running of multiple applications: In order to prevent one user's job from monopolizing resources in the queue, the scheduler will limit the resources occupied by jobs submitted by the same user.
2) Differences from the capacity scheduler:
(1) The core scheduling strategy is different.
The capacity scheduler: gives priority to queues with low resource utilization.
Fair scheduler: gives priority to queues with a large proportion of resource shortages.
(2) Each queue can be set independently. Resource allocation method
Capacity scheduler: FIFO, DFR
Fair scheduler: FIFO, FAIR, DFR
3) Fair scheduler -

The design goal of the shortfall fair scheduler is to have fair resources for all jobs on the time scale. At a certain time, a job should The gap between the acquisition of resources and the actual acquisition of resources is called a gap.
The scheduler will give priority to allocating resources to jobs with large shortages.

1.4.3.1. Fair scheduler queue resource allocation method

1) FIFO strategy
If FIFO is selected as the resource allocation strategy for each queue of the fair scheduler, the fair scheduler is equivalent to the capacity scheduler.
2) Fair strategy
Fair strategy (default) is a resource multiplexing method based on the max-min fairness algorithm. By default, this method is used to allocate resources within each queue. This means that if a queue has two applications running at the same time, each application can get 1/2 of the resources, and three applications can each get 1/3 of the resources.
The specific resource allocation process is consistent with the resource scheduler.
1) Queue resource allocation.
Starting from the root, the depth first algorithm is used, and the queue with the lowest resource occupancy rate is given priority to allocate resources.
2) Job resource allocation
By default, resources are allocated according to the priority of submitted jobs and the order of submission time.
3) Container resource allocation Allocate
resources according to the priority of the container. If the priorities are the same, the principle of data locality is followed:
(1) Tasks and data are on the same node
(2) Tasks and data are on the same rack
(3) Tasks and data are not on the same rack The same node is no longer in the same rack
. Each of the above three steps is to allocate resources according to a fair strategy.
Actual minimum resource share: mindshare = Min (resource demand, minimum configured resource);
whether hungry: inNeedy = resource usage <mindshare (Actual minimum resource share);
Resource allocation ratio: minShareRatio = resource usage/Max (mindshare, 1);
Resource weight ratio: useToWeightRatio = resource usage/weight;

1.4.3.2. Fair scheduler resource allocation algorithm

(1) Queue resource allocation
The total resources of the cluster are 100, and there are three queues. The resource requirements are A-20, B-50, and C-30. First calculation:
100/3=33.33 A points 33.33, and B
points 13.33 more
The score is 33.33, which is 16.67 points less.
C score is 33.33, which is 3.33 more points.
The second calculation (13.33+3.33)/1=16.66
A points 20
B points 33.33+16.66≈50
C points 30

(2) Job resource allocation
(a) Unweighted (the focus is the number of jobs)
There is a queue with 12 total resources and 4 jobs. The resource requirements are:
A-1, B-2, C -6, D-5
First calculation: 12/4=3
A points 3 more and 2
B points 3 more than 1
C point 3 less than 3
D points 3 less than 2
Second calculation: 3/2=1.5
A points 1
B points 2
C points 3+1.5=4.5
D points 3+1.5=4.5
Repeat the calculation until there are no free resources

(b) Weighting (the focus is the weight of the job)
There is a queue with a total resource of 16, and the resource requirements of 4 jobs
are
A-4, B-2, C-10, and D-4.
The weight of each job First calculation for
A-5, B-8, C-1, D-2 : 16/(5+8+1+2)=1 A points 5 more, 1 B point 8 more, 6 points C 1 less 9 D points, 2 points less, second calculation: (1+6)/(1+2)=7/3 A points, 4 points, B points, 2 points, C points, 1–>points 7/3 (2.33)–>6.67 less D points 2–>points 14/3(4.66)–>more than 2.66. Third calculation: 2.66/1=2.66 A points 4 B points 2 C points 1–>points 2.66/1–>points 2.66 D points 4

1.5. Yarn common commands

In addition to viewing the Yarn commands on the hadoop101:8088 page, you can also operate them through commands.
Common commands are as follows
(1) List all Applications:

yarn application -list

(2) Filter yarn application -list -appStates according to Application status (all states ALL, NEW, NEW_SAVING, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED)

yarn application -list -appStates FINISHED

(3) Kill Application

yarn application -kill application_xxxxxxxxxxxx_0001

1.5.2, yarn logs view logs

(1) Query the Application log:

yarn logs -applicationId <ApplicationId>

(2) Query Container logs

yarn logs -applicationId applicationId  -containerId containerId

1.5.3. yarn applicationattempt to view the tasks that are attempted to be run.

(1) List all Application attempts

yarn applicationattempt -list <ApplicationId>

(2) Print the status of ApplicationAttemp

yarn applicationattempt -status <ApplicationAttemptId>

1.5.4, yarn container view container

(1) List all Containers

yarn container -list <ApplicationAttemptId>

(2) Print Container status

	yarn container -status <ContainerId>
	注：只有在任务跑的途中才能看到container的状态

1.5.4. yarn node check node status

List all nodes:

yarn node -list -all

1.5.6, yarn rmadmin update configuration

Load queue configuration

yarn rmadmin -refreshQueues

1.5.7, yarn queue view queue

Print queue information

yarn queue -status default

1.6. Core parameters of Yarn production environment

2. Yarn case

Take a Linux snapshot before operation to facilitate rollback

2.1. Yarn production environment core parameter configuration

1) Requirement: From 1G data, count the number of occurrences of each word. There are 3 servers, each equipped with 4G memory, 1 core CPU, and 2 threads.
2) Demand analysis:
1G / 128m = 8 MapTask; 1 ReduceTask; 1 mrAppMaster.
Each node runs an average of 10 / 3 units ≈ 3 tasks (4 3 3)
3) Modify the yarn-site.xml configuration parameters as follows :

<!-- 选择调度器，默认容量 -->
<property>
	<description>The class to use as the resource scheduler.</description>
	<name>yarn.resourcemanager.scheduler.class</name>
	<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
</property>

<!-- ResourceManager处理调度器请求的线程数量,默认50；如果提交的任务数大于50，可以增加该值，但是不能超过3台 * 2线程 = 6线程（去除其他应用程序实际不能超过4） -->
<property>
	<description>Number of threads to handle scheduler interface.</description>
	<name>yarn.resourcemanager.scheduler.client.thread-count</name>
	<value>4</value>
</property>

<!-- 是否让yarn自动检测硬件进行配置，默认是false，如果该节点有很多其他应用程序，建议手动配置。如果该节点没有其他应用程序，可以采用自动 -->
<property>
	<description>Enable auto-detection of node capabilities such as
	memory and CPU.
	</description>
	<name>yarn.nodemanager.resource.detect-hardware-capabilities</name>
	<value>false</value>
</property>

<!-- 是否将虚拟核数当作CPU核数，默认是false，采用物理CPU核数 -->
<property>
	<description>Flag to determine if logical processors(such as
	hyperthreads) should be counted as cores. Only applicable on Linux
	when yarn.nodemanager.resource.cpu-vcores is set to -1 and
	yarn.nodemanager.resource.detect-hardware-capabilities is true.
	</description>
	<name>yarn.nodemanager.resource.count-logical-processors-as-cores</name>
	<value>false</value>
</property>

<!-- 虚拟核数和物理核数乘数，默认是1.0 -->
<property>
	<description>Multiplier to determine how to convert phyiscal cores to
	vcores. This value is used if yarn.nodemanager.resource.cpu-vcores
	is set to -1(which implies auto-calculate vcores) and
	yarn.nodemanager.resource.detect-hardware-capabilities is set to true. The	number of vcores will be calculated as	number of CPUs * multiplier.
	</description>
	<name>yarn.nodemanager.resource.pcores-vcores-multiplier</name>
	<value>1.0</value>
</property>

<!-- NodeManager使用内存数，默认8G，修改为4G内存 -->
<property>
	<description>Amount of physical memory, in MB, that can be allocated 
	for containers. If set to -1 and
	yarn.nodemanager.resource.detect-hardware-capabilities is true, it is
	automatically calculated(in case of Windows and Linux).
	In other cases, the default is 8192MB.
	</description>
	<name>yarn.nodemanager.resource.memory-mb</name>
	<value>4096</value>
</property>

<!-- nodemanager的CPU核数，不按照硬件环境自动设定时默认是8个，修改为4个 -->
<property>
	<description>Number of vcores that can be allocated
	for containers. This is used by the RM scheduler when allocating
	resources for containers. This is not used to limit the number of
	CPUs used by YARN containers. If it is set to -1 and
	yarn.nodemanager.resource.detect-hardware-capabilities is true, it is
	automatically determined from the hardware in case of Windows and Linux.
	In other cases, number of vcores is 8 by default.</description>
	<name>yarn.nodemanager.resource.cpu-vcores</name>
	<value>2</value>
</property>

<!-- 容器最小内存，默认1G -->
<property>
	<description>The minimum allocation for every container request at the RM	in MBs. Memory requests lower than this will be set to the value of this	property. Additionally, a node manager that is configured to have less memory	than this value will be shut down by the resource manager.
	</description>
	<name>yarn.scheduler.minimum-allocation-mb</name>
	<value>1024</value>
</property>

<!-- 容器最大内存，默认8G，修改为2G -->
<property>
	<description>The maximum allocation for every container request at the RM	in MBs. Memory requests higher than this will throw an	InvalidResourceRequestException.
	</description>
	<name>yarn.scheduler.maximum-allocation-mb</name>
	<value>2048</value>
</property>

<!-- 容器最小CPU核数，默认1个 -->
<property>
	<description>The minimum allocation for every container request at the RM	in terms of virtual CPU cores. Requests lower than this will be set to the	value of this property. Additionally, a node manager that is configured to	have fewer virtual cores than this value will be shut down by the resource	manager.
	</description>
	<name>yarn.scheduler.minimum-allocation-vcores</name>
	<value>1</value>
</property>

<!-- 容器最大CPU核数，默认4个，修改为2个 -->
<property>
	<description>The maximum allocation for every container request at the RM	in terms of virtual CPU cores. Requests higher than this will throw an
	InvalidResourceRequestException.</description>
	<name>yarn.scheduler.maximum-allocation-vcores</name>
	<value>2</value>
</property>

<!-- 虚拟内存检查，默认打开，修改为关闭 -->
<property>
	<description>Whether virtual memory limits will be enforced for
	containers.</description>
	<name>yarn.nodemanager.vmem-check-enabled</name>
	<value>false</value>
</property>

<!-- 虚拟内存和物理内存设置比例,默认2.1 -->
<property>
	<description>Ratio between virtual memory to physical memory when	setting memory limits for containers. Container allocations are	expressed in terms of physical memory, and virtual memory usage	is allowed to exceed this allocation by this ratio.
	</description>
	<name>yarn.nodemanager.vmem-pmem-ratio</name>
	<value>2.1</value>
</property>

4) Distribute configuration.
Note: If the hardware resources of the cluster are inconsistent, each NodeManager must be configured separately.
5) Restart the cluster.

sbin/stop-yarn.sh
sbin/start-yarn.sh

6) Execute the WordCount program

hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /input /output

7) Observe the Yarn task execution page
http://hadoop102:8088/cluster/apps

2.2. Capacity scheduler multi-queue submission case

1) How to create a queue in a production environment?
(1) The scheduler has a default queue by default, which cannot meet production requirements.
(2) Put the tasks of each framework into the designated queue according to the framework hive/spark/flink
(3) According to the business module, login and registration, shopping cart, order, business department
2) The benefits of creating multiple queues
(1) Worry about employees Accidentally exhaust resources
(2) Realize the downgrade of tasks and ensure sufficient resources for important task queues during special periods

2.2.1. Configuring a multi-queue capacity scheduler

1) Configure the following configuration in capacity-scheduler.xml:
(1) Modify the following configuration

<!-- 指定多队列，增加hive队列 -->
<property>
    <name>yarn.scheduler.capacity.root.queues</name>
    <value>default,hive</value>
    <description>
      The queues at the this level (root is the root queue).
    </description>
</property>

<!-- 降低default队列资源额定容量为40%，默认100% -->
<property>
    <name>yarn.scheduler.capacity.root.default.capacity</name>
    <value>40</value>
</property>

<!-- 降低default队列资源最大容量为60%，默认100% -->
<property>
    <name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
    <value>60</value>
</property>

(2) Add necessary attributes to the new queue:

<!-- 指定hive队列的资源额定容量 -->
<property>
    <name>yarn.scheduler.capacity.root.hive.capacity</name>
    <value>60</value>
</property>

<!-- 用户最多可以使用队列多少资源，1表示 -->
<property>
    <name>yarn.scheduler.capacity.root.hive.user-limit-factor</name>
    <value>1</value>
</property>

<!-- 指定hive队列的资源最大容量 -->
<property>
    <name>yarn.scheduler.capacity.root.hive.maximum-capacity</name>
    <value>80</value>
</property>

<!-- 启动hive队列 -->
<property>
    <name>yarn.scheduler.capacity.root.hive.state</name>
    <value>RUNNING</value>
</property>

<!-- 哪些用户有权向队列提交作业 -->
<property>
    <name>yarn.scheduler.capacity.root.hive.acl_submit_applications</name>
    <value>*</value>
</property>

<!-- 哪些用户有权操作队列，管理员权限（查看/杀死） -->
<property>
    <name>yarn.scheduler.capacity.root.hive.acl_administer_queue</name>
    <value>*</value>
</property>

<!-- 哪些用户有权配置提交任务优先级 -->
<property>
    <name>yarn.scheduler.capacity.root.hive.acl_application_max_priority</name>
    <value>*</value>
</property>

<!-- 任务的超时时间设置：yarn application -appId appId -updateLifetime Timeout
参考资料：https://blog.cloudera.com/enforcing-application-lifetime-slas-yarn/ -->

<!-- 如果application指定了超时时间，则提交到该队列的application能够指定的最大超时时间不能超过该值。 
-->
<property>
    <name>yarn.scheduler.capacity.root.hive.maximum-application-lifetime</name>
    <value>-1</value>
</property>

<!-- 如果application没指定超时时间，则用default-application-lifetime作为默认值 -->
<property>
    <name>yarn.scheduler.capacity.root.hive.default-application-lifetime</name>
    <value>-1</value>
</property>

2) Distribute the configuration file
3) Restart Yarn or execute yarn rmadmin -refreshQueues to refresh the queue, and you will see two queues:

2.2.3. Submit tasks to hive queue

1) Hadoop jar method

hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount -D mapreduce.job.queuename=hive /input /output
-D表示运行时改变参数值

2) How to build jar packages.
The default task submissions are submitted to the default queue. If you want to submit tasks to other queues, you need to declare in the Driver:

public class WcDrvier {
    
    

    public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
    
    

        Configuration conf = new Configuration();

        conf.set("mapreduce.job.queuename","hive");

............
..........
.......
    }
}

2.2.4. Task priority

The capacity scheduler supports the configuration of task priorities. When resources are tight, tasks with high priority will be given priority to obtain resources. By default, Yarn limits the priority of all tasks to 0. If you want to use it, please open the restriction first.
1) Modify the yarn-site.xml file and add the following parameters

<property>
    <name>yarn.cluster.max-application-priority</name>
    <value>5</value>
</property>

2) Distribute the configuration and restart Yarn

xsync yarn-site.xml
sbin/stop-yarn.sh
sbin/start-yarn.sh

3) To simulate a resource tight environment, you can continuously submit the following tasks until the newly submitted tasks cannot apply for resources.

hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar pi 5 2000000

4) Resubmit the high-priority task again

hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar pi  -D mapreduce.job.priority=5 5 2000000

5) You can also modify the priority of the task being executed through the following command.

yarn application -appID <ApplicationID> -updatePriority 优先级

2.3. Fair scheduler case

2.3.1. Demand

Create two queues, test and XXX (named after the group to which the user belongs). It is expected to achieve the following effects: if the user specifies a queue when submitting a task, the task will be submitted to the specified queue for running; if no queue is specified, the task submitted by the test user will be run in the root.group.test queue, and the task submitted by XXX will be run in the root.group.XXX Queue operation (note: group is the group to which the user belongs).
The configuration of the fair scheduler involves two files, one is yarn-site.xml, and the other is the fair scheduler queue allocation file fair-scheduler.xml (the file name can be customized).
(1) Configuration file reference:
https://hadoop.apache.org/docs/r3.1.3/hadoop-yarn/hadoop-yarn-site/FairScheduler.html
(2) Task queue placement rule reference:
https:/ /blog.cloudera.com/untangling-apache-hadoop-yarn-part-4-fair-scheduler-queue-basics/

2.3.2. Configuring a multi-queue fair scheduler

1) Modify the yarn-site.xml file and add the following parameters

<property>
    <name>yarn.resourcemanager.scheduler.class</name>
    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
    <description>配置使用公平调度器</description>
</property>

<property>
    <name>yarn.scheduler.fair.allocation.file</name>
    <value>/opt/module/hadoop-3.1.3/etc/hadoop/fair-scheduler.xml</value>
    <description>指明公平调度器队列分配配置文件</description>
</property>

<property>
    <name>yarn.scheduler.fair.preemption</name>
    <value>false</value>
    <description>禁止队列间资源抢占</description>
</property>

2) Configure fair-scheduler.xml

<?xml version="1.0"?>
<allocations>
  <!-- 单个队列中Application Master占用资源的最大比例,取值0-1 ，企业一般配置0.1 -->
  <queueMaxAMShareDefault>0.5</queueMaxAMShareDefault>
  <!-- 单个队列最大资源的默认值 test atguigu default -->
  <queueMaxResourcesDefault>4096mb,4vcores</queueMaxResourcesDefault>

  <!-- 增加一个队列test -->
  <queue name="test">
    <!-- 队列最小资源 -->
    <minResources>2048mb,2vcores</minResources>
    <!-- 队列最大资源 -->
    <maxResources>4096mb,4vcores</maxResources>
    <!-- 队列中最多同时运行的应用数，默认50，根据线程数配置 -->
    <maxRunningApps>4</maxRunningApps>
    <!-- 队列中Application Master占用资源的最大比例 -->
    <maxAMShare>0.5</maxAMShare>
    <!-- 该队列资源权重,默认值为1.0 -->
    <weight>1.0</weight>
    <!-- 队列内部的资源分配策略 -->
    <schedulingPolicy>fair</schedulingPolicy>
  </queue>
  <!-- 增加一个队列atguigu -->
  <queue name="atguigu" type="parent">
    <!-- 队列最小资源 -->
    <minResources>2048mb,2vcores</minResources>
    <!-- 队列最大资源 -->
    <maxResources>4096mb,4vcores</maxResources>
    <!-- 队列中最多同时运行的应用数，默认50，根据线程数配置 -->
    <maxRunningApps>4</maxRunningApps>
    <!-- 队列中Application Master占用资源的最大比例 -->
    <maxAMShare>0.5</maxAMShare>
    <!-- 该队列资源权重,默认值为1.0 -->
    <weight>1.0</weight>
    <!-- 队列内部的资源分配策略 -->
    <schedulingPolicy>fair</schedulingPolicy>
  </queue>

  <!-- 任务队列分配策略,可配置多层规则,从第一个规则开始匹配,直到匹配成功 -->
  <queuePlacementPolicy>
    <!-- 提交任务时指定队列,如未指定提交队列,则继续匹配下一个规则; false表示：如果指定队列不存在,不允许自动创建-->
    <rule name="specified" create="false"/>
    <!-- 提交到root.group.username队列,若root.group不存在,不允许自动创建；若root.group.user不存在,允许自动创建 -->
    <rule name="nestedUserQueue" create="true">
        <rule name="primaryGroup" create="false"/>
    </rule>
    <!-- 最后一个规则必须为reject或者default。Reject表示拒绝创建提交失败，default表示把任务提交到default队列 -->
    <rule name="reject" />
  </queuePlacementPolicy>
</allocations>

3) Distribute configuration and restart Yarn

2.3.3. Test submission task

1) Specify the queue when submitting the task. According to the configuration rules, the task will go to the specified root.test queue.

hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar pi -Dmapreduce.job.queuename=root.test 1 1

2) Do not specify a queue when submitting a task. According to the configuration rules, the task will go to the root.atguigu.XXX queue.

hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar pi 1 1

After all the above operations are completed, take a snapshot or manually modify the configuration file to its previous state. Because the resources are insufficient, it is divided into so many that it is inconvenient for future testing.