spark-submit 提交任务到yarn集群报错

bin/spark-submit --class spark.KafkaAccountReport --master yarn --deploy-mode cluster --driver-memory 2g --executor-memory 2g --executor-cores 3 –-total-executor-cores 5 ./jSpark-jar-with-dependencies.jar

集群模式提交任务到yarn，但是状态一直是ACTIVED或者FAILED,并且报如下错误

Application application_1557392692629_0008 failed 2 times due to AM Container for appattempt_1557392692629_0008_000002 exited with exitCode: 13

Failing this attempt.Diagnostics: [2019-05-29 20:51:14.623]Exception from container-launch.

Container id: container_e01_1557392692629_0008_02_000001

Exit code: 13

[2019-05-29 20:51:14.624]Container exited with a non-zero exit code 13. Error file: prelaunch.err.

Last 4096 bytes of prelaunch.err :

Last 4096 bytes of stderr :

onfun$run$1(ApplicationMaster.scala:245)

at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)

at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:779)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)

at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:778)

at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:245)

at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:803)

at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)

)

java.lang.InterruptedException

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)

at org.apache.spark.streaming.ContextWaiter.waitForStopOrError(ContextWaiter.scala:63)

at org.apache.spark.streaming.StreamingContext.awaitTermination(StreamingContext.scala:618)

at org.apache.spark.streaming.api.java.JavaStreamingContext.awaitTermination(JavaStreamingContext.scala:565)

at spark.KafkaAccountReport.main(KafkaAccountReport.java:113)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:684)

2019-05-29 20:51:14,103 INFO streaming.StreamingContext: Invoking stop(stopGracefully=false) from shutdown hook

2019-05-29 20:51:14,104 INFO scheduler.ReceiverTracker: ReceiverTracker stopped

2019-05-29 20:51:14,104 INFO scheduler.JobGenerator: Stopping JobGenerator immediately

2019-05-29 20:51:14,105 INFO util.RecurringTimer: Stopped timer for JobGenerator after time 1559134270000

2019-05-29 20:51:14,109 INFO scheduler.JobGenerator: Stopped JobGenerator

2019-05-29 20:51:14,111 INFO scheduler.JobScheduler: Stopped JobScheduler

2019-05-29 20:51:14,114 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@76543490{/streaming,null,UNAVAILABLE,@Spark}

2019-05-29 20:51:14,114 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@5c31fedc{/streaming/batch,null,UNAVAILABLE,@Spark}

2019-05-29 20:51:14,115 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@72e54550{/static/streaming,null,UNAVAILABLE,@Spark}

2019-05-29 20:51:14,115 INFO streaming.StreamingContext: StreamingContext stopped successfully

2019-05-29 20:51:14,116 INFO spark.SparkContext: Invoking stop() from shutdown hook

2019-05-29 20:51:14,119 INFO server.AbstractConnector: Stopped Spark@7037120f{HTTP/1.1,[http/1.1]}{0.0.0.0:0}

2019-05-29 20:51:14,119 INFO ui.SparkUI: Stopped Spark web UI at http://dw-10-10-10-22.haisu.com:46730

2019-05-29 20:51:14,124 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!

2019-05-29 20:51:14,130 INFO memory.MemoryStore: MemoryStore cleared

2019-05-29 20:51:14,131 INFO storage.BlockManager: BlockManager stopped

2019-05-29 20:51:14,131 INFO storage.BlockManagerMaster: BlockManagerMaster stopped

2019-05-29 20:51:14,135 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!

2019-05-29 20:51:14,139 INFO spark.SparkContext: Successfully stopped SparkContext

2019-05-29 20:51:14,139 INFO yarn.ApplicationMaster: Deleting staging directory hdfs://haisu-v2/user/dw/.sparkStaging/application_1557392692629_0008

2019-05-29 20:51:14,174 INFO util.ShutdownHookManager: Shutdown hook called

2019-05-29 20:51:14,174 INFO util.ShutdownHookManager: Deleting directory /data/hdfs/data/tmp/nm-local-dir/usercache/dw/appcache/application_1557392692629_0008/spark-cc902437-f2b1-45bc-a450-ab6d0aa991db