Spark way to throw an exception when running in yarn

Spark throws an exception when running in yarn ways:

cluster.YarnClientSchedulerBackend: Yarn application has already exited with state FINISHED!

solution:

Mainly to allocate less memory node, yarn kill the spark application.

1. Stop the cluster hadoop

[root@node06 bin]# stop-all.sh

2. increase the allocation to the yarn-site.xml:

<property>
    <name>yarn.nodemanager.pmem-check-enabled</name>
    <value>false</value>
</property>
<property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
    <description>Whether virtual memory limits will be enforced for containers</description>
</property>
<property>
    <name>yarn.nodemanager.vmem-pmem-ratio</name>
    <value>4</value>
    <description>Ratio between virtual memory to physical memory when setting memory limits for containers</description>
</property>

3. The covering yarn-site.xm files on the changes to the master l each node slaves

[root@node06 hadoop]# scp yarn-site.xml node07:`pwd`
yarn-site.xml                                                                                                              100% 1565     1.5KB/s   00:00    
[root@node06 hadoop]# scp yarn-site.xml node08:`pwd`
yarn-site.xml                                                                                                              100% 1565     1.5KB/s   00:00    
[root@node06 hadoop]# scp yarn-site.xml node09:`pwd`

 

4. Restart the cluster hadoop

[root@node06 hadoop]# start-all.sh

5. Re-submit the task to spark yarn

[root@node06 bin]# ./spark-submit --master yarn --class org.apache.spark.examples.SparkPi ../lib/spark-examples-1.6.0-hadoop2.6.0.jar 100