Link to the official website of Spark submission parameters
Official website screenshot: read more in English
- Pi case:
SPARK_HOME=/export/server/spark
${SPARK_HOME}/bin/spark-submit \
--master yarn \
--deploy-mode client \
--driver-memory 512m \
--executor-memory 512m \
--num-executors 1 \
--total-executor-cores 2 \
--class org.apache.spark.examples.SparkPi \
${SPARK_HOME}/examples/jars/spark-examples_2.11-2.4.5.jar \
10
- **Basic parameter configuration:** When submitting to run Spark Application, some basic parameters need to be passed values, as shown below:
Dynamically load the runtime parameters of Spark Applicaiton, and specify through -conf
- Driver Program parameter configuration:
Each Spark Application runtime has a Driver Program , which belongs to a JVM Process process and can set the number of memory and CPU core cores (resources)
- Executor parameter configuration : (understand that if the amount of data is large, use it to modify the memory CPU)
to sum up:
Yarn：
--num-executors NUM 去启动的executors的数量，默认为2
--executor-cores NUM 每一个executor有多少cpu cores，yarn默认为1
假如：num-executors=10个executors数量，每个executor有executor-cores=2，所有的executor的cpucores的数量10*2=20个
#Executor 的内存，真正执行计算的内存
--executor-memory MEM Memory per executor (e.g. 1000M, 2G) (Default: 1G). 每个Executor的内存，1G
假如：num-executors=10个executors数量，每个executor的内存--executor-memory=2G，所有参与计算的executors的内存是10*2=20G
--driver-cores NUM Driver端使用的CPuCores集合，默认为1
#Driver 申请资源执行计算
--driver-memory MEM Memory for driver (e.g. 1000M, 2G) (Default: 1024M). Driver的内存，1G
- Main code execution process:
except for the opening and closing of SparkCOntext, all other parts are completed in the executor