Parameter meaning
The common submission commands for spark on yarn are as follows:
${SPARK_HOME}/bin/spark-submit --class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode cluster \
--driver-memory 2g \
--executor-memory 1g \
--executor-cores 4 \
--num-executors 3 \
--queue default \
${SPARK_HOME}/examples/jars/spark-examples*.jar \
10
- num-executors refers to the number of executors to be used.
- executor-memory refers to the memory size allocated by each executor.
- executor-cores refers to the number of CPU cores allocated to each executor.
- driver-memory refers to the memory size allocated by the driver.
The submitted application runs as a driver in AM. It builds sparkContext objects, DAGScheduler objects, TaskScheduler objects, parses RDD operations into directed acyclic graphs, divides stages according to wide and narrow dependencies, builds tasks, encapsulates taskSets, etc., etc. All operations require memory space, and driver-memory allocates memory for it.
RDD is eventually parsed into tasks and executed in the executor, and each CPU core executes one task. There are executors on the cluster num-executors, and each executor has executor-coresCPU cores, so a maximum of tasks can be processed in parallel at the same timenum-executors * executor-cores . That is to say, these two parameters adjust the maximum parallel processing capability of tasks. The actual degree of parallelism is also related to the number of data partitions. .
The memory structure of the executor is shown in the figure below, which consists of two parts: yarn overhead memory and JVM Heap memory. JVM Heap memory can be divided into three parts: RDD cache memory, shuffle memory, and work heap, indicating the size of the JVM Heap executor-memory. One of the reasons spark is faster than mapreduce is that it keeps the intermediate results of calculations in memory instead of writing them back to disk. Therefore, when the memory is larger, RDD can cache more data, and the shuffle operation can use more memory and write as little data as possible to the disk, thereby reducing disk IO; when the task is executed, the larger memory space means By doing garbage collection less frequently, performance can be improved from this point on. Of course, the bigger the memory space, the better. If the memory space is too large, the cluster cannot allocate it, and yarn will kill the task directly. However, increasing the resource application to a certain extent can indeed improve the efficiency of task execution.
How to make full use of the applied resources
Only by making full use of resources can tasks be executed more efficiently. This is actually simple, in one sentence: The number of task tasks matches the number of executor cores . Assuming executor-cores=4 and num-executors=3, 12 tasks can be executed in parallel. However, if there are less than 12 tasks when executing a certain stage, this will result in a waste of applied resources and relatively low execution efficiency.
# 总数据量 12G 时：
分区 每个分区数据量 使用的 cores 效率
10 1.2G 10 低
12 10G 12 高
The official document has this description
Clusters will not be fully utilized unless you set the level of parallelism for each operation high enough.
Spark automatically sets the number of “map” tasks to run on each file according to its size (though you can
control it through optional parameters to SparkContext.textFile, etc), and for distributed “reduce” operations,
such as groupByKey and reduceByKey, it uses the largest parent RDD’s number of partitions. You can pass the
level of parallelism as a second argument (see the spark.PairRDDFunctions documentation), or set the config
property spark.default.parallelism to change the default. In general, we recommend 2-3 tasks per CPU core in
your cluster.
You can read the details yourself. During reduce operations, such as reduceByKey and groupByKey, you can modify the number of partitions through parameters, or you can use spark.default.parallelism in the configuration file to modify the number of partitions. Generally, it is recommended to allocate 2-3 tasks to each CPU core, that is, the number of tasks is 2-3 times of all cores.