Today records about the use of a problem Hql encountered map = 100% and reduce = 100% over the thought that can perform a result, the last reported a sudden Bug, life really is full of surprises and unexpected, I can only say this ... there is no happy surprise
After I saw this Bug, view the log found the hive, the hive with the implementation of the interface is the same newspaper, see what things out, then use hadoop's history: 8088 view aggregated log and found the UI here is the start of the 4 a map a reduce, reduce no logs appears to be no map is not complete, leading to reduce not work, point them into a map View log discover the source of the newspaper wrong, the problem is, heap overflow, and virtual memory the cause of map was blown away
Several digital-focused to do something simple Comments
110.2 MB of 1 GB
3.7 GB of 2.1 GB virtual memory used. Killing container
1,110.2MB is the actual physical memory for this task
2,1GB is mapreduce.map.memory.mb set (default)
3,3.7GB task is to take up the virtual memory
4,2.1GB is mapreduce.map.memory.db multiplied yarn.nodemanager.vmem-pmem-ratio obtained
5, wherein yarn.nodemanager.vmem-pmem-ratio is the ratio of virtual memory and physical memory, the yarn-site.xml provided, the default is 2.1
if I is locally set 3, which is the calculated value 1 * 3 = 3GB
In summary logs tell us is the task to be virtual memory is only 3.7G and 2.1G of system virtual memory beyond the limit does not work, here's just beyondVirtual MemoryThere are differences with the actual memory of their own Baidu do not understand this knowledge
So the solution in the following areas
In the yarn-site.xml set off the virtual memory checking (the default is True open state)
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
<description>Whether virtual memory limits will be enforced for containers.</description>
</property>
According to my own configuration, and the amount of data processed discovery memory is enough, because this is in general virtual memory checking mechanism locked out so I configured only this, you can continue execution of the task
In the yarn-site.xml physical memory is checked can be closed
<property>
<name>yarn.nodemanager.pmem-check-enabled </name>
<value>false</value>
<description>Whether virtual memory limits will be enforced for containers.</description>
</property>
However, such a setting is generally not recommended because if a program has a memory leak and other issues, cancel the check, could lead to the collapse of the cluster.
There is a shortage of physical memory may cause the program to fail to start, increase physical memory
In mapred-site.xml appropriately adjusted value of the following parameters according to the amount of data processing:2048 、-Xmx1024MSuitable self-set value
<property>
<name>mapreduce.map.memory.mb</name>
<value>2048</value>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx1024M</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>2048</value>
</property>
<property>
<name>mapreduce.reduce.java.opts</name>
<value>-Xmx2560M</value>
</property>
It can be adjusted yarn.nodemanager.vmem pmem--ratio size, because virtual memory is affected by the percentage of physical memory
There may be a process running in the MR data in the tilt, memory leaks and other reasons that only a detailed analysis of the specific reasons
After changing the yarn-site.xml configuration file Remember to restart the cluster yarn
