Resource parameter tuning
After understanding the basic principles of Spark job finished running, the relevant parameters of the resource is easy to understand. Spark so-called resource parameter tuning, in fact, mainly for the operation of various parts of the process Spark use of resources, by adjusting various parameters to optimize the efficiency of resource use, so as to enhance execution performance Spark job. The following parameters Spark is the major resource parameters, each parameter corresponds to a part of the job of operating principles, we also give a tuning reference value.
num-executors
Parameter Description : This parameter is used to set the number of jobs Spark Executor to execute the process of total use. Driver when applying resources to the cluster manager YARN, YARN cluster manager will try to follow your settings to work on each node of the cluster, start the appropriate number of Executor process. This parameter is very important, if not set, the default will only give you a small amount of Executor start the process, then your Spark job running speed is very slow.
Parameter tuning recommendations : Spark job to run each set of generally about 50 to 100 Executor process is appropriate, too little or too much Executor setting process is not good. Too few settings can not make full use of cluster resources; too many settings, then most of the queue may not be given sufficient resources.
executor-memory
Parameter Description : This parameter is used to set the memory of each Executor process. Executor memory size, often directly determines the performance Spark job, but with the common JVM OOM exceptions, there is a direct correlation.
Parameter tuning recommendations : Executor provided for each memory 4G ~ 8G process is more appropriate. But this is only a reference value, specific settings still have to be determined according to different sectors of resource queues. The maximum memory limit can look at their team is the number of resource queues, num-executors multiplied by the executor-memory, it represents the total amount of memory your application to the Spark job (that is, the sum of all Executor memory processes), the amount It can not exceed the maximum amount of memory queue. In addition, if you are sharing this resource queue with others in the team, then the total amount of memory the application should not exceed a maximum total memory resource queue 1/3 ~ 1/2, to avoid your own Spark job takes up all of the queue resources, leading to other students job can not run.
performer-colors
Parameter Description : This parameter is used to set the number of each Executor process CPU core. This parameter determines the capabilities of each Executor process parallel execution task threads. Because each CPU core can only perform one task at the same time thread, so the more the number of CPU core per Executor process, the more quickly perform all assigned to complete their task thread.
Parameter tuning recommendations : the number of CPU core Executor is set to 2 to 4 is more appropriate. According to the same resource queues have different departments to set, you can see the maximum CPU core restriction own resources is the number of queues, and then based on the number of Executor set to determine each Executor process can be assigned to several CPU core. Also suggested that if the queue is shared with others, then the num-executors * executor-cores do not exceed the total cohort CPU core is about 1/3 ~ 1/2 is appropriate, but also to avoid affecting the job to run other students.
driver-memory
Parameter Description : This parameter is used to set the memory Driver process.
Parameter tuning recommendations : Driver memory usually is not set, or set about 1G should be enough. The only thing to note is that if you need to use the RDD operators collect all the data to pull Driver for processing, it must ensure Driver memory is large enough, otherwise there will be problems OOM memory overflow.
spark.default.parallelism
Parameter Description : This parameter is used to set the default number for each stage of the task. This parameter is extremely important, if not set may directly affect your job performance Spark.
Parameter tuning recommendations : Spark job default task number 500 to 1000 is more appropriate. Many students often make a mistake is not to set this parameter, then the time will lead to Spark themselves to set the number of task based on the number of HDFS block bottom, the default is a HDFS block corresponds to a task. Generally speaking, the number of Spark default setting is less than normal (for example, on dozens of task), if the number of task too few, it would result in front of you to set the parameters of good Executor all come to naught. Imagine, no matter your Executor process how many, how much memory and CPU, but only one task or 10, then 90% of the Executor process may simply not perform the task, which is a waste of resources! Thus Spark official website of the recommended settings principle is to set the parameters for the num-executors * 2 ~ 3 times executor-cores is more appropriate, such as the total number of CPU core Executor is 300, then set the 1000 task is possible, at this time you can make full use of resources Spark cluster.
spark.storage.memoryFraction
Parameter Description : This parameter is used in the data memory can Executor proportion RDD persistence provided, the default is 0.6. In other words, the default Executor 60% of memory, can be used to store persistent data RDD. Depending on the persistence strategy you choose, and if not enough memory, data is not likely to persist, or the data is written to disk.
Tuning parameters recommended : if Spark operation, there are more persistent RDD operation, the value of the parameter number can be appropriately improved to ensure data persistence can be accommodated in memory. Avoid not enough memory to cache all data, resulting in data can only be written to disk, reducing the performance. However, if more Spark job class shuffle operations, and long-lasting operation is relatively small, then the values of the parameters appropriate to reduce some of the more appropriate. In addition, if you find a job due to frequent gc cause slowness (it can be observed through the time-consuming to spark web ui gc jobs), mean task executing user code memory is not enough, it is also recommended to lower the value of this parameter.
spark.shuffle.memoryFraction
Parameter Description : This parameter memory Executor ratio can be used when a task after the process shuffle pull task to the output of the last stage, the polymerization is carried out an operation for setting the default is 0.2. In other words, Executor default only 20% of the memory is used for this operation. When the polymerization shuffle operation is performed, using the memory if it is found beyond this limit of 20%, the excess will overflow data is written to the disk file, then performance will be greatly reduced.
Parameter tuning recommendations : Spark job if RDD persistence fewer operations, more shuffle operation, it is recommended to reduce the proportion of persistent memory operation, increase the proportion accounted shuffle memory operation, during shuffle avoid excessive data memory is not enough with, overflow must be written to disk, reducing the performance. In addition, if you find a job due to frequent gc cause slowness means that task executing user code memory is not enough, it is also recommended to lower the value of this parameter.
Tuning resource parameters, not a fixed value, requires the students according to their actual conditions (including job gc where the number of shuffle operation Spark job, persistence number of operations RDD and the spark web ui shown), while referring to the present principle and tuning advice given in the article, a reasonable set these parameters.
Resource Parameter Reference Example
The following is an example of a spark-submit the command, we can refer to, and make adjustments according to their actual situation:
./bin/spark-submit \ --master yarn-cluster \ --num-executors 100 \ --executor-memory 6G \ --executor-cores 4 \ --driver-memory 1G \ --conf spark.default.parallelism=1000 \ --conf spark.storage.memoryFraction=0.5 \ --conf spark.shuffle.memoryFraction=0.3 \
Parameter setting unreasonable
This technique is relatively common, before we look at the core parameters:
num-executors=10 || 20 ，executor-cores=1 || 2， executor-memory= 10 || 20，driver-memory=20，spark.default.parallelism=64
Assuming our spark queue resources as follows:
memory=1T，cores=400
How to set parameters here are some tips, first you have to understand the principles of resource allocation and use spark:
By default, non-dynamic resource allocation scenarios, spark a pre-application resources, the task will not start monopolizing resources, until the entire job task all over, such as your stepping stones played a spark-shell has not quit, did not perform the task that will always occupy all resources application. (If you set num-executors, dynamic resource allocation will fail)
Note that the above sentence, spark resource allocation and use mapreduce / hive is very different, if you do not understand the problem will lead to other problems on the parameter settings.
For example, executor-cores set the number of suitable? Not less mission parallelism, more exclusive resources will run out of the entire cohort, other students can not perform the task, that task such as the above, the num-executors = 20 executor-cores = case 1 executor-memory = 10 to will monopolize 20 cores, 200G memory, lasted three hours.
For in this case that task, combined with our existing resources, how to set up the five core parameters?
1) executor_cores * num_executors not too small or too large! Generally does not exceed 25% of the total cohort cores, such as queue total cores 400, the largest not more than 100, the minimum is not recommended below 40, unless the log is very small.
2) executor_cores not 1! Otherwise, the number of threads in the work process is too small, usually 2 to 4 is appropriate.
3) executor_memory general 6 ~ 10g is appropriate, the largest not more than 20G, otherwise it will lead to GC too costly, or serious waste of resources.
4) spark_parallelism usually 1 to 4 times executor_cores * num_executors, the system default 64, do not set it will lead to a lot of the task when they were executed serially in batches, or a large number of idle cores, serious waste of resources.
5) driver-memory Earlier students set 20G, in fact, without any driver computing and storage, but issued a task resource management and task interact with yarn, unless you are a spark-shell, or generally 1-2g enough.
Spark Memory Manager：
6) spark.shuffle.memoryFraction (default 0.2), also known as ExecutionMemory. This memory area in order to solve shuffles, joins, sorts and aggregations process in order to avoid frequent IO buffer needed. If your application has a large number of such operations may be appropriate to increase.
7) spark.storage.memoryFraction (default 0.6), also known as StorageMemory. This memory area to solve the block cache (that is, you show the call dd.cache, rdd.persist and other methods), there is the broadcasts, and task results stored. By parameters, if you call a lot of persistence operations or broadcast variables, it may be appropriate to increase it.
8) OtherMemory, reserved to the system, because the program itself is also a need to run memory, (the default is 0.2). Other memory in 1.6 has been adjusted to ensure that there is at least 300m available. You can also set spark.testing.reservedMemory manually. Then the actual available memory minus the reservedMemory get usableMemory. ExecutionMemory and StorageMemory share usableMemory * 0.75 of memory. 0.75 spark.memory.fraction be provided by a new parameter. Currently spark.memory.storageFraction default value is 0.5, so ExecutionMemory, StorageMemory default is sharing the available memory mentioned above.
For example, if you need to load large dictionary files, you can increase the executor in StorageMemory size, so you can avoid global dictionary swapped out to reduce the GC, in this case, we use the equivalent of memory resources in exchange for the implementation of efficiency .