用spark-submit提交任务给集群时涉及的参数:
用bin/spark-submit提交,查看spark-submit用法
bin/spark-submit --help
Usage:spark-submit [options] <app jar | python file> [app arguments]
如local模式时:
#将程序运行在local mode,启动2个Thread
bin/spark-submit --master local[2] 1.py
将脚本运行在集群上:
/bin/spark-submit\
--master spark://master:7077\ (指定运行在哪里;运行在集群上)
--deploy-mode client\ (部署模式,客户端还是集群)
--driver-memory 512M\
--executor-memory 1G\ (每个executor的内存)
--executor-cores 1\ (每个executor的核数)
--total-executor-cores 2\ (所有executor的核数)
1.py