spark on yarn:Container is running beyond physical memory limits

After installing hadoop and spark in the virtual machine. Execute start-all.sh (hadoop command) to start hdfs and yarn services. After the service is successfully started, in order to test the executable of spark on yarn, enter the command:

./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode client ./examples/jars/spark-examples_2.11-2.1.1.jar 100

After execution, we found that the program reported an error:

Application application_1447669815913_0002 failed 2 times due to AM Container for appattempt_1447669815913_0002_000002 exited with exitCode: -104 For more detailed output, check application tracking page:http://myserver.com:8088/proxy/application_1447669815913_0002/
Then, click on links to logs of each attempt. 
Diagnostics: Container [pid=47384,containerID=container_1447669815913_0002_02_000001] is running beyond physical memory limits. Current usage: 17.9 GB of 17.5 GB physical memory used; 18.7 GB of 36.8 GB virtual memory used. Killing container.

The reason for this error is that the virtual machine has insufficient resources to create a container that meets the requirements. Since our execution is spark on yarn, in order to increase the memory of the virtual machine, we can configure it in the hadoop configuration directory

Configure yarn-site.xml. We add the following configuration under the original configuration:

<property>
   <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
    <description>Whether virtual memory limits will be enforced for containers</description>
  </property>
<property>
   <name>yarn.nodemanager.vmem-pmem-ratio</name>
    <value>4</value>
    <description>Ratio between virtual memory to physical memory when setting memory limits for containers</description>
</property>

The first configuration is whether to force the container to start when the virtual machine memory is insufficient. If it is set to no, memory errors can be detected as soon as possible.

The second configuration is to increase the mapping ratio of virtual machine memory to physical memory. The default is 2.1. 

Restart the service, resubmit the task, and the execution is successful. (Be sure to restart the service)