spark-submit --master yarn configuration requires increased WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set

supplement to last blog :https://blog.csdn.net/yao09605/article/details/103971966
solving the WARN message:
Neither spark.yarn.jars nor spark.yarn.archive is set

$ jar -cv0f spark-libs.jar -C $SPARK_HOME/jars/ .
$ dfs -mkdir /system
$ dfs -mkdir /system/SparkJars
$ dfs -mkdir /system/SparkJars/jar
$ hdfs dfs -put spark-libs.jar /system/SparkJars/jar
$ cd $SPARK_HOME/conf
$ cp spark-defaults.conf.template spark-defauls.conf

add

spark.yarn.archive=hdfs:///system/SparkJars/jar/spark-libs.jar

run

$ spark-submit --master yarn app.py

should be ok by now

Published 78 original articles · won praise 7 · views 10000 +

Guess you like

Origin blog.csdn.net/yao09605/article/details/103975849