spark-submit --master yarn 需要增加的配置WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set

supplement to last blog :https://blog.csdn.net/yao09605/article/details/103971966
solving the WARN message:
Neither spark.yarn.jars nor spark.yarn.archive is set

$ jar -cv0f spark-libs.jar -C $SPARK_HOME/jars/ .
$ dfs -mkdir /system
$ dfs -mkdir /system/SparkJars
$ dfs -mkdir /system/SparkJars/jar
$ hdfs dfs -put spark-libs.jar /system/SparkJars/jar
$ cd $SPARK_HOME/conf
$ cp spark-defaults.conf.template spark-defauls.conf

add

spark.yarn.archive=hdfs:///system/SparkJars/jar/spark-libs.jar

run

$ spark-submit --master yarn app.py

should be ok by now

发布了78 篇原创文章 · 获赞 7 · 访问量 1万+

猜你喜欢

转载自blog.csdn.net/yao09605/article/details/103975849