想把Hadoop基于Yarn部署到小内存主机上，默认的配置会由于资源不足导致简单的任务也无法执行成功。
2019-12-02 14:28:33,989 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1575266850654_0002
2019-12-02 14:28:33,989 INFO mapreduce.JobSubmitter: Executing with tokens: []
2019-12-02 14:28:34,316 INFO conf.Configuration: resource-types.xml not found
2019-12-02 14:28:34,317 INFO resource.ResourceUtils: Unable to find ‘resource-types.xml’.
2019-12-02 14:28:34,393 INFO impl.YarnClientImpl: Submitted application application_1575266850654_0002
2019-12-02 14:28:34,470 INFO mapreduce.Job: The url to track the job: http://vm-0-2-centos:8088/proxy/application_1575266850654_0002/
2019-12-02 14:28:34,471 INFO mapreduce.Job: Running job: job_1575266850654_0002
2019-12-02 14:28:43,766 INFO mapreduce.Job: Job job_1575266850654_0002 running in uber mode : false
2019-12-02 14:28:43,767 INFO mapreduce.Job: map 0% reduce 0%
2019-12-02 14:30:23,061 INFO mapreduce.Job: map 26% reduce 0%
2019-12-02 14:30:24,107 INFO mapreduce.Job: map 44% reduce 0%
2019-12-02 14:38:58,133 INFO mapreduce.Job: Task Id : attempt_1575266850654_0002_m_000001_0, Status : FAILED
AttemptID:attempt_1575266850654_0002_m_000001_0 Timed out after 600 secs
2019-12-02 14:38:58,148 INFO mapreduce.Job: Task Id : attempt_1575266850654_0002_m_000002_0, Status : FAILED
有时候还会发现Hadoop的常驻进程自动挂了，通过journal -xe查看日志可以看到由于资源的原因，被内核强制停了。
Dec 02 14:30:23 vm-0-2-centos kernel: Out of memory: Kill process 16044 (java) score 94 or sacrifice child
Dec 02 14:30:23 vm-0-2-centos kernel: Killed process 16044 (java), UID 1000, total-vm:2440728kB, anon-rss:174572kB, file-rss:2148kB, shme
导致Yarn无法正常执行任务的主要资源限制是在内存上，所以可以调整限制Yarn所能使用的物理内存。
<!-- yarn-site.xml -->
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>1024</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>128</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>1024</value>
</property>
配置后如果有超过该资源的请求，则会被拒接
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException): Invalid resource request! Cannot allocate containers as requested resource is greater than maximum allowed allocation. Requested resource type=[memory-mb], Requested resource=<memory:1536, vCores:1>, maximum allowed allocation=<memory:80, vCores:4>, please note that maximum allowed allocation is calculated by scheduler based on maximum resource of registered NodeManagers, which might be less than configured maximum allocation=<memory:80, vCores:4>
默认情况下AM的请求1.5G的内存，降低am的资源请求配置项到分配的物理内存限制以内。
<!-- mapred-site.xml -->
<property>
<name>yarn.app.mapreduce.am.resource.mb</name>
<value>256</value>
</property>
默认对mapred的内存请求都是1G，也降低和合适的值。
<!-- mapred-site.xml -->
<property>
<name>mapreduce.map.memory.mb</name>
<value>256</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>256</value>
</property>
但如果太低也会出现OFM的问题。
2019-12-02 15:43:45,837 ERROR [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:1000)
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:408)
at org.apache.hadoop.mapred.MapTask.access NewOutputCollector.(MapTask.java:710)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:782)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
以上配置在2G的内存的主机上可以正常执行自带sample中的grep任务。
