Some resource configuration parameters of yarn-site.xml are mainly related to memory

vim yarn-site.xml

Each nodemanager can allocate 100G of memory: yarn.nodemanager.resource.memory-mb
Each nodemanager can allocate 20 CPU cores: yarn.nodemanager.resource.cpu-vcores

  <property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>102400</value>
    <description>Amount of physical memory, in MB, that can be allocated for containers.</description>
  </property>

  <property>
    <name>yarn.nodemanager.resource.cpu-vcores</name>
    <value>20</value>
    <description>Number of vcores that can be allocated for containers.</description>
  </property>

yarn.nodemanager.resource.memory-mb
represents the total amount of physical memory available for YARN on the node, the default is 8192 (MB), note that if your node memory resources are not enough 8GB, you need to reduce this value, and YARN Will not intelligently detect the total physical memory of the node.

The yarn.nodemanager.vmem-pmem-ratio
task uses 1MB of physical memory, and the maximum amount of virtual memory can be used. The default is 2.1.

yarn.nodemanager.pmem-check-enabled
Whether to start a thread to check the amount of physical memory used by each task, if the task exceeds the allocated value, it will be killed directly, the default is true.

yarn.nodemanager.vmem-check-enabled
Whether to start a thread to check the amount of virtual memory used by each task, if the task exceeds the allocated value, it will be killed directly, the default is true.

yarn.scheduler.minimum-allocation-mb
The minimum amount of physical memory that a single task can apply for. The default is 1024 (MB). If the amount of physical memory requested by a task is less than this value, the corresponding value will be changed to this number.

yarn.scheduler.maximum-allocation-mb
The maximum amount of physical memory that a single task can apply for. The default is 8192 (MB).

Guess you like

Origin blog.51cto.com/fengyunshan911/2595489