YARN
Since hadoop2.0, we can use apache yarn to manage cluster resources. Yarn can divide and isolate resources (memory, CPU) in the form of Containers. YARN will manage the available computing resources of all machines in the cluster. Based on these resources, YARN will schedule resource requests from applications (such as MapReduce), and then YARN will allocate Containers to provide processing capabilities to each application. The Container is YARN The basic unit of processing power is the package (container) for memory, CPU, etc.
ResourceManager: hereinafter referred to as RM. The central control module of YARN is responsible for the unified planning of the use of resources.
NodeManager: Hereinafter referred to as NM. The resource node module of YARN is responsible for starting the management container.
ApplicationMaster: hereinafter referred to as AM. Each application in YARN will start an AM, which is responsible for applying for resources from RM, requesting NM to start the container, and telling the container what to do.
Container: Resource container. All applications in YARN run on the container. AM also runs on the container, but the AM container is applied for by RM.
After understanding the basic concepts above, you can start to optimize the configuration of the cluster
Configure NM's registered resources
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>30</value>
<discription>每个nodemanager可分配的cpu总核数</discription>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>122880</value>
<discription>每个nodemanager可分配的内存总量</discription>
</property>Optimization suggestions:
1. The number of cpu cores = the number of logical cores-the number of other applications (datanode? work? zk? etc.)
cat /proc/cpuinfo | grep "processor" | wc -lYou can check the number of logical cores in the cluster.
2. The memory is recommended to be an integer multiple of the CPU, and enough memory is reserved for the system
ApplicationMaster configuration
<property>
<name>yarn.app.mapreduce.am.resource.cpu-vcores</name>
<value>1</value>
</property>
<property>
<name>yarn.app.mapreduce.am.resource.mb</name>
<value>4096</value>
<discription>ApplicationMaster的占用的内存大小</discription>
</property>
Optimization suggestion
1. The ratio of cpu and memory is consistent with the allocation ratio of nm
Container configuration optimization
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>16384</value>
<discription>单个任务可申请最大内存，默认8192MB</discription>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-vcores</name>
<value>4</value>
<discription>单个任务可申请的最多虚拟CPU个数</discription>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-vcores</name>
<value>1</value>
<discription>单个任务可申请的最小虚拟CPU个数</discription>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>4096</value>
<discription>container最小可申请的内存</discription>
</property>Optimization suggestion
1. In the scheduler, many resource calculation parts will be converted into N times the minimum value for calculation. Therefore, when setting allocable memory and other resources, it is best to be just a multiple of this minimum value.
2. The cpu/memory ratio remains the same
. 3. YARN uses thread monitoring to determine whether the task is overusing memory. Amount, kill it directly. Due to the lack of flexibility in the control of memory by Cgroups (that is, the task cannot exceed the upper limit of memory at any time, if it exceeds, it will be directly killed or reported OOM), and the Java process will double the memory at the moment of creation, and then drop to the normal value. In this case, the thread monitoring method is more flexible (when it is found that the memory of the process tree instantly doubles and exceeds the set value, it can be considered a normal phenomenon and the task will not be killed), so YARN does not provide a Cgroups memory isolation mechanism To control the container.
mapreduce parameter settings
<property>
<name>mapreduce.map.memory.mb</name>
<value>4096</value>
<discription>map的内存大小</discription>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx3072M</value>
<discription>用户设定的map/reduce阶段申请的container的JVM参数。最大堆设定要比申请的内存少一些，用于JVM的非堆部分使用0.80-0.85建议</discription>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>8192</value>
</property>
<property>
<name>mapreduce.reduce.java.opts</name>
<value>-Xmx6144M</value>
</property>
Optimization reference
1. If the cluster mainly uses mr for calculation, it is recommended that the memory of the map and the minimum of the cpu and the container are equal.
2. How many maps can be run in a container at most? yarn.scheduler.maximum-allocation-mb/mapreduce.map.memory.mb=4
The question is
how to control the number of Containers in a nodemanager?
<property>
<name>yarn.scheduler.fair.assignmultiple</name>
<value>true</value>
<discription>是否允许NodeManager一次分配多个容器</discription>
</property>
<property>
<name>yarn.scheduler.fair.max.assign</name>
<value>20</value>
<discription>如果允许一次分配多个,一次最多可分配多少个,这里按照一个最小分配yarn.scheduler.minimum-allocation-mb4gb来计算总共内存120/4=30给20即可</discription>
</property>Fari Scheduler configuration example
24 nodes, 120GB memory per node, 30 logical CPUs
<?xml version="1.0"?>
<allocations>
<queue name="mapreduce">
<minResources>368640 mb,90 vcores</minResources><!--3 nodes-->
<maxResources>2334720 mb,570 vcores</maxResources><!--19 nodes-->
<maxRunningApps>70</maxRunningApps>
<weight>5</weight>
<queue name="vipquery">
<minResources>122880 mb,30 vcores</minResources><!--1 nodes-->
<maxResources>1966080 mb,480 vcores</maxResources><!--16 nodes-->
<maxRunningApps>20</maxRunningApps>
<weight>8</weight>
</queue>
<queue name="hive">
<minResources>122880 mb,30 vcores</minResources><!--1 nodes-->
<maxResources>1966080 mb,480 vcores</maxResources><!--16 nodes-->
<maxRunningApps>20</maxRunningApps>
<weight>7</weight>
</queue>
<queue name="hadoop">
<minResources>122880 mb,30 vcores</minResources><!--1 nodes-->
<maxResources>1966080 mb,480 vcores</maxResources><!--16 nodes-->
<maxRunningApps>30</maxRunningApps>
<weight>6</weight>
</queue>
</queue>
<queue name="default">
<minResources>122880 mb,30 vcores</minResources><!--1 nodes-->
<maxResources>614400 mb,150 vcores</maxResources><!--5 nodes-->
<maxRunningApps>20</maxRunningApps>
<weight>1</weight>
</queue>
</allocations>to sum up
Yarn can be effectively controlled through reasonable configuration, resource preemption, and peak concurrency issues.