capacity调度器和fair调度器队列配置

1、fair调度器配置：

队列default：wkz：wc=1:3:6（资源占比）

<allocations>
<queue name="root">
<aclSubmitApps> </aclSubmitApps>
<aclAdministerApps> </aclAdministerApps>
<queue name="wkz">
<minResources>24649mb,7 vcores</minResources>
<maxResources>246495mb,72 vcores</maxResources>
<maxRunningApps>100</maxRunningApps>
<weight>1</weight>
<minSharePreemptionTimeout>30</minSharePreemptionTimeout>
<schedulingMode>fair</schedulingMode>
<aclSubmitApps>wkz</aclSubmitApps>
<aclAdministerApps>wkz</aclAdministerApps>
</queue>
<queue name="wc">
<minResources>12324mb,4 vcores</minResources>
<maxResources>123247mb,36 vcores</maxResources>
<maxRunningApps>100</maxRunningApps>（最大运行的applications数量）
<weight>1</weight>（权重，说是如果有多余的资源就按照权重比例分配给各个队列，好像还需要配置什么，没深究）
<minSharePreemptionTimeout>300</minSharePreemptionTimeout>（最小共享量抢占时间。如果一个资源池在该时间内使用的资源量一直低于最小资源量，则开始抢占资源。）
<schedulingMode>fair</schedulingMode>
<aclSubmitApps>wc</aclSubmitApps>（队列提交用户限制，*是全部，空格是无，或者具体的用户名，限制那个用户能在本队列提交任务）
<aclAdministerApps>wc</aclAdministerApps>（队列admin权限拥有者，*是全部，空格是无，或者具体的用户名，限制能够删除队列等）
</queue>
<queue name="default">
<minResources>4108mb,1 vcores</minResources>
<maxResources>41082mb,12vcores</maxResources>
<maxRunningApps>100</maxRunningApps>
<weight>1</weight>
<minSharePreemptionTimeout>1000</minSharePreemptionTimeout>
<schedulingMode>fair</schedulingMode>
<aclSubmitApps>*</aclSubmitApps>
<aclAdministerApps>*</aclAdministerApps>
</queue>
</queue>
</allocations>

2、capacity调度器配置：

default：wkz：wc=30:45:25

<configuration>
<property>
<name>yarn.scheduler.capacity.root.queues</name>

<value>default,wkz,wc</value>（root队列下的子队列，这些就是我们默认用的队列，如果不设置这些，capacity默认有一个default队列。如果你想要在CM上把defalut队列删除，那么只是简单的在这块把值设置为wkz，wc，刷新yarn后，你会发现yarn的rm页面并没有去掉队列default，在CM上若想完全去除default对列，必须重启yarn服务，在进入yarn的rm页面就会发现default队列已经没有了）

</property>
<property>
<name>yarn.scheduler.capacity.root.capacity</name>
<value>100</value>（队列的资源占比，因为root是父队列，也是根队列，故占比100%，这块不设置也行，自己亲测过。）
</property>
<property>
<name>yarn.scheduler.capacity.root.acl_administer_queue</name>
<value>root</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.acl_submit_applications</name>
<value>root</value>(这块是控制能否实现用户只能向指定队列提交任务的权限，如果改为*，则队列的用户限制无法实现。)
</property>
<property>
<name>yarn.scheduler.capacity.root.default.capacity</name>
<value>30</value>(default队列资源占比)
</property>
<property>
<name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
<value>35</value>（default队列使用的资源上限）
</property>
<property>
<name>yarn.scheduler.capacity.root.wkz.capacity</name>
<value>45</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.wkz.maximum-capacity</name>
<value>50</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.wc.capacity</name>
<value>25</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.wc.maximum-capacity</name>
<value>30</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.acl_administer_queue</name>
<value>root</value>（default队列的管理者）
</property>
<property>
<name>yarn.scheduler.capacity.root.default.acl_submit_applications</name>
<value>*</value>（设置default队列任何用户都可以提交任务）
</property>
<property>
<name>yarn.scheduler.capacity.root.wkz.acl_administer_queue</name>
<value>wkz</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.wkz.acl_submit_applications</name>
<value>wkz</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.wc.acl_administer_queue</name>
<value>wc</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.wc.acl_submit_applications</name>
<value>wc</value>
</property>
<property>
<name>yarn.scheduler.capacity.resource-calculator</name><value>org.apache.hadoop.yarn.util.resource.DominantResourceCalculator</value>（设置资源计算方法，这个方法会根据cpu和内存计算资源）
</property>

</configuration><allocations>

注：fifo调度器，无需配置，只有一个默认队列defalut，先进先出，capacity对列就是多个先进先出队列，fair有点乱，很多没实现。

配置文件如下：

fair

<allocations>
<queue name="root">
<aclSubmitApps> </aclSubmitApps>
<aclAdministerApps> </aclAdministerApps>
<queue name="wkz">
<minResources>24649mb,7 vcores</minResources>
<maxResources>246495mb,72 vcores</maxResources>
<maxRunningApps>100</maxRunningApps>
<weight>1</weight>
<minSharePreemptionTimeout>1000</minSharePreemptionTimeout>
<schedulingMode>fair</schedulingMode>
<aclSubmitApps>wkz</aclSubmitApps>
<aclAdministerApps>wkz</aclAdministerApps>
</queue>
<queue name="wc">
<minResources>12324mb,4 vcores</minResources>
<maxResources>123247mb,36 vcores</maxResources>
<maxRunningApps>100</maxRunningApps>
<weight>1</weight>
<minSharePreemptionTimeout>1000</minSharePreemptionTimeout>
<schedulingMode>fair</schedulingMode>
<aclSubmitApps>wc</aclSubmitApps>
<aclAdministerApps>wc</aclAdministerApps>
</queue>
<queue name="default">
<minResources>4108mb,1 vcores</minResources>
<maxResources>41082mb,12vcores</maxResources>
<maxRunningApps>100</maxRunningApps>
<weight>1</weight>
<minSharePreemptionTimeout>1000</minSharePreemptionTimeout>
<schedulingMode>fair</schedulingMode>
<aclSubmitApps>*</aclSubmitApps>
<aclAdministerApps>*</aclAdministerApps>
</queue>
</queue>
</allocations>

capacity

<configuration>
<property>
<name>yarn.scheduler.capacity.root.queues</name>
<value>default,wkz,wc</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.capacity</name>
<value>100</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.acl_administer_queue</name>
<value>root</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.acl_submit_applications</name>
<value>root</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.capacity</name>
<value>30</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
<value>35</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.wkz.capacity</name>
<value>45</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.wkz.maximum-capacity</name>
<value>50</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.wc.capacity</name>
<value>25</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.wc.maximum-capacity</name>
<value>30</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.acl_administer_queue</name>
<value>root</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.acl_submit_applications</name>
<value>*</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.wkz.acl_administer_queue</name>
<value>wkz</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.wkz.acl_submit_applications</name>
<value>wkz</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.wc.acl_administer_queue</name>
<value>wc</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.wc.acl_submit_applications</name>
<value>wc</value>
</property>
<property>
<name>yarn.scheduler.capacity.resource-calculator</name><value>org.apache.hadoop.yarn.util.resource.DominantResourceCalculator</value>
</property>
</configuration>