Spark - how to properly set the executor-memory, executor-cores, num-executors

Parameter Description

executor-memoryRepresents the memory allocated to each executor, the default is 1G.
executor-coresIt represents a number assigned to each core audit executor i.e., the default mode is a spark on yarn.
num-executorsExpressed as a number of current application start executor. If you turn on dynamic allocation ( spark.dynamicAllocation.enablednumber executor of default is false), the initial start-up is the num-executorsnumber of settings.

The following four suggestions to keep in mind

• Hadoop/Yarn/OS Deamons: When we cluster managerwhen to run tasks such as yarn spark cluster, there will be some background process to perform the NameNode、Secondary NameNode、DataNode、ResourceManager、NodeManagerlike. Therefore, before determining the above three parameters, we need to make sure to leave enough for those cores daemon ( 一般每个节点留一个core).
• Yarn ApplicationMaster (AM): ApplicationMasterResponsible for the ResourceManagerapplications and resources and NodeManagersthe implementation and monitoring together containersand their resource consumption. If we spark on yarn mode, then we need to ApplicationMasterset aside some resources ( 1G和1个Executor).
• HDFS Throughput: HDFS client will be multi-threaded concurrency issues. Studies have shown that HDFS while simultaneously executed five task each Executor write throughput is the best. Therefore, it is recommended executor-coresvalue remains at 5 or less .
• MemoryOverhead: We spark on yarn and deploy-mode = cluster Example. Before version 2.3,
  by spark.yarn.executor.memoryOverheadthe definition of the executor to memoryOverhead. Official ratio to 10%, usually between 6% and 10%, most of them chose to 7%. Specific reference 1. 6.3-spark-properties

Full memory requested to yarn per executor =
          spark-executor-memory + spark.yarn.executor.memoryOverhead.
 spark.yarn.executor.memoryOverhead = 
        	Max(384MB, 7% of spark-executor-memory)

In version 2.3, it is spark.executor.memoryOverheaddefined. Wherein for memoryOverhead VM overheads, interned strings, other native overheadslike.

GC usually cause great delays in the implementation of the assigned Executor has a great memory.
Executor performed very small (such as 一个coreand enough memory to run a task required), so that in a JVM ( 每个Executor就是一个独立的JVM实例) can run simultaneously advantage of multiple task is lost.

Configuration parameters

First, assume a cluster configuration as follows

10 Nodes
16 cores per Node
64GB RAM per Node

Method a: Tiny executors (One Executor per core)

--num-executors =  16 x 10 = 160
--executor-cores = 1 (one executor per core)
--executor-memory = `mem-per-node/num-executors-per-node`
                     = 64GB/16 = 4GB

In this One Executor per corecase the configuration, we can not take advantage of simultaneously run multiple task in a JVM inside. Meanwhile, the shared variable variable broadcast (broadcast variables) and accumulator (Accumulators) will be replicated on every node on each core a is copied 16 times on each node ( 因为共享变量会复制到每个executor上). And we do not set aside sufficient resources to background processes such as NameNode etc., nor the resources to count up ApplicationManager. very bad! ! !

Method two: Fat executors (One Executor per node)

--num-executors =  10
--executor-cores = 16 
--executor-memory = `mem-per-node/num-executors-per-node`
                     = 64GB/1 = 64GB

Each node on the Executor to get all core, except ApplicationManagerand background processes aside, HDFS throughput will receive the impact and accompanied by verification of GC. It is also very bad! ! !

Method three: Balance between Fat (vs) Tiny

First, according to the parameters recommended above, we assign to each of the five core Executor that is executor-cores=5, so the throughput of HDFS will be more friendly.
Second, as a background process remain a core, the number of available core each node is a 16 - 1 = 15. So the total number of available core clusters are 15 x 10 = 150.
Third, the number of Executor on each node is 15 / 5 = 3the cluster of the total number of Executor is available 3 * 10 = 30. To ApplicationManagerleave a Executor, then num-executors=29.
Fourth, each node can be assigned to each memory is Executor (64GB-1GB) / 3 = 21GB(subtracted daemon is left with 1GB), removed MemoryOverHead=max(384MB, 7% * 21GB)=2GB, so executor-memory=21GB - 2GB = 19GB.
So last parameter configuration is

--num-executors =  29
--executor-cores = 5 
--executor-memory = 19GB

This approach not only ensures the advantages can be executed simultaneously in the task in a JVM instance, but also to ensure the throughput of hdfs.

Method four: three on the basis of the method does not require so much memory each executor

The method three, the memory allocated to each Executor is 19GB, 10GB assumed memory is sufficient. So at this time we can executor-coresreduce as reduced to 3, each node can have 15 / 3 = 5a Executor, then the total number of Executor is available (5 * 10) - 1 =49, on each node can be assigned to each Executor memory is (64GB-1GB) / 5 = 12GBremoved
MemoryOverHead=max(384MB, 7% * 12GB)=1GB, so executor-memory=12GB - 1GB = 11GB
it is the last configuration parameters are

--num-executors =  49
--executor-cores = 3 
--executor-memory = 11GB

Reference Site

distribution_of_executors_cores_and_memory_for_spark_application
how-to-tune-spark-executor-number-cores-and-executor-memory
resource-allocation-configuration-spark-yarn