Submit job to Spark Yarn using Intellij IDEA in window7 (mode)

使用window提交到Spark cluster中出现下面错误：
Exit code: 1
Exception message: /bin/bash: line 0: fg: no job control
Stack trace: ExitCodeException exitCode=1: /bin/bash: line 0: fg: no job control
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent .ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non -zero exit code 1

troubled for a week, all kinds of Baidu, google, can't solve the problem.

Finally I opened a small program through EClipse and found that the
<dependency>
            <groupId>org.apache.spark</groupId> referenced by mvn
            <artifactId>spark-yarn_2.11</artifactId>
            <version>2.1.0</version>
            < scope>compile</scope>
        </dependency>
This jar package conflicts with the jar package of jars under spark, delete this reference, the problem is solved!

Record it here.

When running the program, I found that the program will always package all the dependent jars into zip and upload them to hdfs, which is a waste of time

. I package the spark jars into zip under hdfs, and upload them to hdfs
for use in the program.
conf.set("spark.yarn.archive", "hdfs:////input/spark/spark-jars.zip")
can be set