Submit job resource parameter tuning

https://www.cnblogs.com/wwcom123/p/10561806.html

 

　spark submit jobs, yarn-cluster pattern examples:

　　./bin/spark-submit\

　　--class com.ww.rdd.wordcount \

　　--master yarn \

　　--deploy-mode cluster \ 
　　--executor-memory 4G \
　　--num-executors 50 \

　　--driver-memory 1G \

　　--conf spark.default.parallelism=1000 \

　　--conf spark.memory.fraction=0.75 \

　　--conf spark.memory.storageFraction=0.5 \

　　/home/spark/wordcount.jar \
　　1000 # to the Senate