Overview
Node labels are a way to group nodes with similar characteristics, and applications can specify where to run. Right now we only support node partitions:
- A node can only have one node partition, so a cluster is divided into multiple mutually exclusive sub-clusters through node partitions. By default, nodes belong to the DEFAULT partition (partition="").
- Users need to configure how many resources each partition can use from different queues. Please refer to the following sections for more details.
- There are two types of node partitions:
- Mutually exclusive: Containers are assigned to nodes with exactly matching node partitions (eg requesting partition="x" will be assigned to nodes with partition="x", requesting DEFAULT partition will be assigned to nodes with DEFAULT partition.
- Non-exclusive: If a partition is non-exclusive, then it shares idle resources with containers requesting the DEFAULT partition.
The user can specify the set of node labels that can be accessed by each queue, and an application can only use a subset of the node labels that can be accessed by the queue to which it belongs.
characteristic
Currently Node Labels supports the following features:
- Partition cluster - Each node can specify a label, so the cluster will be divided into multiple small mutually exclusive partitions.
- ACL of node-labels on queues - User can set the node labels that each queue can access, so only some nodes can be accessed by specific queues.
- Specifies the percentage of resources of the partition that can be accessed by the queue - the user can set the percentage like this: queue A can access 30% of the resources of a node with label lable=hbase. Such percentage settings will be consistent with existing resource managers.
- Specify the required Node Label in the resource request and it will be assigned to the node with the same label. If the required node label is not specified, then resource requests will be allocated to nodes belonging to the DEFAULT partition.
- Operability
- Node Label and node label mapping can be restored in RM restart.
- Update node labels - Update node labels - Administrators can update the labels of nodes and queues during RM running.
configuration information
Set up the ResourceManager to enable Node Labels:
Set the following properties in yarn-site.xml
| Property | Value |
| yarn.node-labels.fs-store.root-dir | hdfs://namenode:port/path/to/store/node-labels/ |
| yarn.node-labels.enabled | true |
Remark:
- Make sure yarn.node-labels.fs-store.root-dir is created and ResourceManager has permission to access it.
- If the user wants to store the node label to the local file system (replacing hdfs), a path like file:///home/yarn/node-label can be used.
Add/modify node label list and node-to-labels mapping
- Add a list of cluster node labels:
- Execute yarn rmadmin -addToClusterNodeLabels "label_1(exclusive=true/false),label_2(exclusive=true/false)" to add node labels.
- If the user does not specify "(exclusive=...)", exclusive defaults to true.
- 运行 yarn cluster --list-node-labels 检查添加的node labels在集群中已经是可见的。
- 添加label到node
- 执行 yarn rmadmin -replaceLabelsOnNode “node1[:port]=label1 node2=label2”，增加label1到node1，label2到node2.如果用户没有指定port，它将会添加到运行在node上的所有NodeManager。
调度器与node label相关的配置项
Capacity Scheduler 配置
| Property | Value |
| yarn.scheduler.capacity.<queue-path>.capacity | 设置相应队列能够访问属于DEFAULT partition的节点的capacity百分比。每个parent之下的所有直接children的DEFAULT capacity总和必须是100. |
| yarn.scheduler.capacity.<queue-path>.accessible-node-labels | 管理员需要指定每个队列能够访问的label，以英文逗号分隔，像这种形式；“hbase,storm” 表示队列能够访问hbase和storm label。所有队列都能访问没有label的node，用户不需要额外指定。如果用户没有指定该字段，它将会从parent继承。如果用户想明确指定一个队列只能访问没有label的node，只需要设置空格作为值。 |
| yarn.scheduler.capacity.<queue-path>.accessible-node-labels.<label>.capacity | 设置队列能够访问<label> partition的node的百分比。每个父节点下直接children的<label> capacities的总和必须是100.默认是0. |
| yarn.scheduler.capacity.<queue-path>.accessible-node-labels.<label>.maximum-capacity | 类似于yarn.scheduler.capacity.<queue-path>.maximum-capacity，它是每个队列对于label的maximum-capacity。默认值是100. |
| yarn.scheduler.capacity.<queue-path>.default-node-label-expression | 值类似于"hbase",它的意思是：如果提交到队列的application，在其资源请求中没有指定node label，它会使用hbase作为default-node-label-expression。默认值为空，所以application会从没有node label的节点中获取容器。 |
node label配置的示例：
假定我们有这样一个队列结构
root
/ | \
engineer sales marketing
我们在集群中有5个node（hostname=h1..h5），其中的每一个有24G memory，24vcores。5个node之中的1个有GPU（假定是h5）.所以管理员增加GPU label到h5.
Assume user have a Capacity Scheduler configuration like: (key=value is used here for readability)
假定用户的Capacity Scheduler配置如下： (此处的key=value处于可读性考虑)
yarn.scheduler.capacity.root.queues=engineering,marketing,sales yarn.scheduler.capacity.root.engineering.capacity=33 yarn.scheduler.capacity.root.marketing.capacity=34 yarn.scheduler.capacity.root.sales.capacity=33 yarn.scheduler.capacity.root.engineering.accessible-node-labels=GPU yarn.scheduler.capacity.root.marketing.accessible-node-labels=GPU yarn.scheduler.capacity.root.engineering.accessible-node-labels.GPU.capacity=50 yarn.scheduler.capacity.root.marketing.accessible-node-labels.GPU.capacity=50 yarn.scheduler.capacity.root.engineering.default-node-label-expression=GPU
你可以看到 root.engineering/marketing/sales.capacity=33，所以它们其中的每一个授权具有partition的1/3的资源。所以它们中的每一个能够使用h1..h4的1/3的资源，也就是24 * 4 * (1/3) = (32G mem, 32 v-cores)。另外只有engineering/marketing队列有权限访问GPU partition。
engineering/marketing中的每一个被授权具有
partition=GPU
label的资源的1/2.所以它们之中的每一个能够使用h5资源的1/2，也就是 24 * 0.5 = (12G mem, 12 v-cores).
备注：
- 完成CapacityScheduler的配置之后，执行yarn rmadmin -refreshQueues使变更生效。
- 到RM Web UI的调度页面检查你的配置是否成功设置。
为application指定node label
application可以使用如下Java APIs指定请求的node label。(API的描述就不翻译了)
- ApplicationSubmissionContext.setNodeLabelExpression(..) to set node label expression for all containers of the application.
- ResourceRequest.setNodeLabelExpression(..) to set node label expression for individual resource requests. This can overwrite node label expression set in ApplicationSubmissionContext
- Specify setAMContainerResourceRequest.setNodeLabelExpression in ApplicationSubmissionContext to indicate expected node label for application master container.
监控
通过Web UI监控
通过web UI可以看到以下label相关的字段：
- Nodes page: http://RM-Address:port/cluster/nodes, 你可以获取每个node的label。
- Node labels page: http://RM-Address:port/cluster/nodelabels, 你可以获取type(exclusive/non-exclusive),活跃的node manager数量，每个partition的资源总量。
- Scheduler page: http://RM-Address:port/cluster/scheduler, 你可以获取每个队列的label相关设置，以及每个队列partition的资源使用情况。
通过命令行监控
- 使用yarn cluster --list-node-labels 获取集群中的label。
- 使用yarn node -status <NodeId> 获取指定node的状态，状态信息包含labels