Spark optimization summary (three) - parameter adjustment

Spark optimization summary (three) - parameter adjustment

Foreword

• Do not expect to modify a parameter can magically magical immediately get good results! (Sometimes the effect is really great ^ _ ^) you should put arguments together as a dish of spices, can enrich the taste, but mainly rely on the quality of raw materials and cooking skills.
• When developing Spark application, it should be good to optimize your application code, think again parameter adjustment optimization (save for essential parameters).
• Parameter adjustment is a more complex subject, different environment, different code will cause the same parameter produce different results. After we recommended to determine your production environment, the optimization of the code after a good obvious problem, do the test parameter adjustment.
• The following lists some parameters used in the development, and explain, by reference

A simple example

• Spark a job submission example

  spark-submit \
  --queue test_queue --master yarn --deploy-mode cluster \
  --num-executors 10 --executor-memory 8G --executor-cores 4 \
  --driver-memory 4G --driver-cores 2 \
  --conf spark.network.timeout=300 \
  --conf spark.locality.wait=9s \
  --class com.skey.spark.app.MyApp /home/jerry/spark-demo.jar 
  

• Parameter Description

  • -queue test_queue will be submitted to the task queue YARN above test_queue
  • -master yarn task will run on top YARN
  • -deploy-mode cluster designated cluster mode
  • -num-executors 10 designated start 10 Executor process
  • -executor-memory 8G 8G memory allocated to each Executor
  • -executor-cores 4 each assigned Executor core 4, a core corresponding to a thread
  • -driver-memory 4G 4G memory allocation process for the Driver
  • -driver-cores 2 Driver is assigned 2 core, a core corresponding to a thread
  • -Conf spark.network.timeout = 300s disposed within the cluster network communication delay of 300s
  • -conf spark.locality.wait = 9s localization data set to load the waiting time, if a node does not spare resources to start Executor, will wait for the time
  • spark specified application class -class com.skey.spark.app.MyApp need to run
  • /home/jerry/spark-demo.jar specify jar package needs to run

Common parameters

• The following parameters are listed in Spark 2.4.3 configuration of the standard
• spark.memory.useLegacyMode
  • Default: false
  • Explanation: This parameter is used to control whether to use the memory management legacy (that is, before the tungsten program memory model, that is, 1.6 the previous version). The legacy mode, the memory spark.storage.memoryFraction (0.6) + spark.shuffle.memoryFraction (0.2) + default (0.2) composition.
  • Recommendation: beginning to develop applications that can not be turned on, you can direct dynamic allocation. Once the development is completed, if the application is able to determine the shuffle every time, when the amount of storage, you can try instead true, so do not dynamically allocated memory (require time-consuming). Also dynamic allocation mode execution borrowed from storage memory regardless of need do not need will not be returned to storage, it may cause some problems.
• spark.storage.memoryFraction
  • Default: 0.6
  • Explanation: When set to true effective spark.memory.useLegacyMode, for controlling the cache memory section
  • Recommendation: If the cached data is relatively large, you can turn up the parameters
• spark.shuffle.memoryFraction
  • Default: 0.2
  • Explanation: When set to true effective spark.shuffle.memoryFraction, shuffle section for controlling
  • Recommendation: If you spill (overflow to write) occurs frequently, you can turn up the parameters. Few, if your hard tune your application with a large cache of this parameter ^ _ ^
• spark.storage.unrollFraction
  • Default: 0.2
  • Explain: spark.memory.useLegacyMode is true valid, the cached data for serialization / deserialization of accounting for 20% spark.storage.memoryFraction
  • Suggestion: you can not transfer
• spark.executor.memoryOverhead
  • Default value: executorMemory * 0.10, the lowest 384M
  • Explanation: each heap memory is provided outside the Executor, mainly for its own JVM, string, NIO Buffer and other expenses. Container equal memory - Executor memory. (This parameter is not found in the official website)
  • Recommended: Spark used is based on the nio, nio direct memory area efficiency is very high, it can be an appropriate part of the memory allocated to heap outer zone.
• spark.locality.wait
  • Default value: 3s
  • Localized long waiting time for the specified data, including three sub-parameters (spark.locality.wait.node, spark.locality.wait.process, spark.locality.wait.rack) for finer adjustment: interpretation.
  • Recommendation: usually, a uniform adjustment spark.locality.wait can be. Can be appropriately extended latency, especially when a large amount of data, if we can localize, the better.
• spark.network.timeout
  • Default value: 120s
  • Explanation: spark memory communication latency network
  • Recommendation: If the spark application process is time-consuming, you can turn up the appropriate parameters (eg 300s), to prevent the error caused by delay
• spark.default.parallelism
  • Default: None
  • Explanation: specifies the number of partitions of RDD shuffle operations, e.g. reduceByKey, join the like. Call the shuffle count midnight, using a number of priority sub-partitions designated operator, otherwise spark.default.parallelism value, or no value if the value of the number of partitions parent RDD is used. Invalid sql in the shuffle.
  • Recommendation: If you want fine control, operators are willing to add parallelism value for each shuffle, then do not set ^ _ ^. Recommended to first set a value (e.g., 2-3 times the total of the core), then the code has to be adjusted to the operator sub-shuffle parameter passing, covering this default value. (Do not be misled into believing what degree of parallelism online nonsense)
• spark.sql.shuffle.partitions
  • Default: 200
  • Explanation: Specifies the number of partitions for the implementation of SparkSQL shuffle. (This parameter is not found in the official website)
  • Recommended: SQL when execution shuffle, if you default 200 less than the total number of core, will be a waste of resources. The recommended setting is 2-3 times of the total core.
• spark.jars
  • Default: None
  • Explanation: Specifies jar package (separated by a comma), which will spread Driver, Executor end
  • Recommendation: In addition to the package under the spark lib, libraries recommend other additional references are specified. (Under client mode, only the Driver will use the library, you can not pass)
• spark.jars.excludes
  • Default: None
  • Explanation: Do not rule out the packets, preventing rely conflict. Format groupId: artifactId
  • Suggest: …
• spark.executor.userClassPathFirst与spark.driver.userClassPathFirst
  • Default: false
  • Explanation: The priority is used to specify whether to load user-specified jar
  • Recommendation: Because there is a user-specified default jar and spark problem library jar package may conflict when the conflict, you can determine if your jar No problem, you can set to true. (Before encountered a bug Huawei is one such)
• spark.driver.maxResultSize
  • Default value: 1g
  • Explanation: Driver terminal data to the maximum amount, for example, collect, take the operation
  • Recommendation: sometimes might require a direct pull large amounts of data, the parameters can be adjusted according to the amount of data.
• spark.reducer.maxSizeInFlight
  • Default value: 48m
  • Explanation: Each reduce task to pull the maximum amount of data
  • Recommendation: Each output needs to create a buffer, consumption is relatively large. If the memory is large, it can increase the value, the pulling speed up.
• spark.shuffle.file.buffer
  • Default value: 32k
  • He explained: shuffle, the output data to a buffer size of the file, after filling the buffer, the data is written to disk overflow.
  • Recommendation: turn up the value, you can reduce the number of overflow to disk (reduction of I / O) to improve performance. Adequate memory, you can turn up, such as 64k.
• spark.shuffle.io.maxRetries
  • Default: 3
  • Explain: shuffle time, read from the write side end pulling data retry number
  • Recommendation: Because the GC, network latency and other issues could lead to pull fails, an appropriate increase in the number of retries to prevent accidents.
• spark.shuffle.io.retryWait
  • Default value: 5s
  • Explained: spark.shuffle.io.maxRetries each retry have to wait long
  • Suggestion: can be increased to improve stability
• spark.shuffle.manager
  • Default values: sort
  • Explanation: shuflle for managing document output to disk mode. Recommended Baidu to see the detailed flow. Version 1.2 before default HashShuffleManager, after the default is SortShuffleManager. Spark After 2 no longer has this parameter is directly SortShuffleManager, the default data will be sorted.
  • Suggestion: if you can sort by spark.shuffle.sort.bypassMergeThreshold adjustment.
• spark.shuffle.sort.bypassMergeThreshold
  • Default: 200
  • Explanation: When shffle, if read task number less than 200, will enable the bypass mechanism: not sort operation will eventually merge files created task, and create an index
  • Recommendation: do not need sorting, you can increase this parameter, you read greater than the number of task shuffle. (Have good efficiency gains, one of my applications was reduced by 20% of the time)
• spark.kryo.registrationRequired
  • Default: false
  • Explanation: whether to force kryo serialization
  • Recommendation: If false, kyro need to write the class name is not registered class for each object, can cause significant performance overhead. Recommended set to true. (Some of my friends for JVM quoted do not know how to register some classes, here recommend copying error class, use Class.forName ( "Class"))
• spark.rdd.compress
  • Default: false
  • Explanation: whether to compress the serialized data RDD
  • Recommendation: turn reduces memory, but will increase the CPU time consuming decompression
• spark.scheduler.mode
  • Default value: FIFO
  • Explanation: within the same SparkContext scheduling mechanism, including FIFO, FAIR
  • Recommendation: general use less. FIFO, FIFO, priority execution, the latter have to give free first job submitted. FAIR, equitable distribution of resources, evenly distributed computing resources for each job can be executed in parallel.
• spark.streaming.backpressure.enabled
  • Default: false
  • Explanation: whether to enable backpressure mechanisms to control the rate of the received data SparkStream.
  • Recommendation: streaming, the processing speed if slow, will lead to a backlog of data constantly. When enabled, Spark can own dynamic adjustment of the processing capability of the received data. If the backlog exists, it is enabled. There are also several parameters that control the details is not recommended adjustments.
• spark.streaming.blockInterval
  • Default value: 200ms
  • Explanation: taske Number per batch batch interval = / blockInterval
  • Recommendation: The larger the blockInterval, taske the less, it will lead to some core is not used. Depending on the amount of your core, and due to lower this parameter. (Official suggested blockInterval minimum of about 50ms)
• spark.streaming.concurrentJobs
  • Default: 1
  • Explanation: a SparkStream how many job applications can be run simultaneously. (This parameter is not found in the official website)
  • Recommendation: If you assign the core more, each batch task relatively small number (possibly handle a long time), idle core more waste, you can increase this parameter while running back job (if two of the job no association between words)
• spark.driver.extraJavaOptions与spark.executor.extraJavaOptions
  • Default: None
  • Explanation: The parameter used to specify the JVM
  • Recommendation: Look JVM parameter adjustment section

JVM parameter adjustment

• Generally not to emphasize JVM, most of the problems in development is a result of poor code quality, first look at the code, if the business logic problems, optimize, optimize, optimize again ...... ^ _ ^
• Then, determine the operating environment, again adjust JVM parameters
• Note that the default value will decrease as the environment change the system, please use the runtime java -XX:+PrintFlagsFinal -versionview the final parameters JVM command
• View GC case
  • View GC each node
    • Add -XX: PrintGCDetails, so that each node print GC logs, you need to see in each node separately. On YARN, you can click on the WebUI interface to view the log for each node.
  • Spark to see the overall application throughput?
    • Spark WebUI each application has to show, to see it directly here. If the GC for more than 10%, then state your applications require optimized!
• Select on the GC (Java 8)
  • Batch, a higher throughput? With -XX: + UseParallelGC
  • Streaming, you need to have a high data consistency? With -XX: + UseConcMarkSweepGC or G1
  • Erratic? Election ParallelGC
  • When the batch would like to use G1? You can try, but no ParallelGC good throughput (Java 8)
• Parallel garbage collector parameters (ParallelGC)
  • With throughput priority criteria
  • -XX: + UseParallelGC set up the young generation use ParallelGC, old's default SerialGC.
  • -XX: + UseParallelOldGC years old settings also use ParallelOldGC, began to support the Java1.6 (my system is turned on by default, you do not necessarily)
  • -XX: ParallelGCThreads = 8 threads disposed parallel garbage collection is 8, the number of generally the same CPU core
  • -XX: MaxGCPauseMillis = 100 disposed in the young generation ParallelGC single recovery takes up to 100 ms. If takes more than this value, JVM will automatically adjust the younger generation of memory size to accommodate the value.
  • -XX: Percentage GCTimeRatio = n representing the time disposed garbage collection program run time was 99%, i.e. 99% throughput. Formula 1 / (1 + n), n = 99 default
  • -XX: + UseAdaptiveSizePolicy enable adaptive strategy. JVM will automatically adjust the proportion of the young generation and the Eden District area the size of Survivor, in order to adapt the value of GCTimeRatio. Recommended that you turn.
• Concurrent garbage collector parameters (ConcMarkSweepGC)
  • Response time priority criteria
  • -XX: + UseConcMarkSweepGC years old settings using ConcMarkSweepGC, the young generation use ParNewGC default.
  • -XX: + UseParNewGC set up on behalf of the young ParNewGC. When using more than JDK5 CMS, the young generation will use the default ParNewGC.
  • -XX: + UseCMSCompactAtFullCollection open compression finishing (default), compress memory fragmentation problems finishing for eliminating (because of mark using CMS sweep algorithm, will not be compressed organize, so to open separately)
  • -XX: After CMSFullGCsBeforeCompaction = 2 is set twice FullGc (default 0 times) to begin memory compression finishing.
  • -XX: CMSInitiatingOccupancyFraction = 70 settings memory using 70% concurrent garbage collection began years old
• Common memory tuning parameters
  • Common ratio

          young    | old
    eden | s0 | s1 | tenured
    ------------------------
           1       |  2
    8    | 1  | 1  | 
    

  • -Xmx4G set the maximum JVM heap memory to 4G
  • -Xms4G set the initial JVM heap memory is 4G, usually set to the same large -Xmx can (avoid re-allocate memory)
  • -Xmn1G set the young generation memory size is 1G. Heap Memory = + years old the young generation, and therefore between the young generation and the old year to strike a balance based on GC circumstances. For example MinorGC more, you can turn up the young generation, MajorGC more can tune Tate's (ie, self-xiaonianqing generations).
  • -XX: NewSize = 512M to set the initial size of the young generation 512M, usually set to the same MaxNewSize be larger. (Not recommended, directly -Xmn)
  • -XX: MaxNewSize = 1024M to set the maximum size of the young generation for the 1024M. (Not recommended, directly -Xmn)
  • -XX: MetaspaceSize = 128M provided meta-information area (permanent generation) initial size of 256M, and usually set to be larger MaxMetaspaceSize same. (Java8 formerly known as -XX: PermSize)
  • -XX: MaxMetaspaceSize = 256M provided meta-information area (permanent generation) maximum size of 256M. (Java8 formerly known as -XX: MaxPermSize)
  • -XX: NewRatio = 2 to set the young generation and the old age size ratio is 1: 2 (Java8 defaults to 2)
  • -XX: SurvivorRatio = 8 disposed in the young generation area and Survivor Eden size ratio is 8: 1: 1 (enden: survivor0: survivor1)
  • -Xss256K set up private to each thread stack size to 256K (Java8 default 1M). The larger memory footprint more threads, when you need to enable ultra-multi-threaded, can lower the parameters, such as 128k.
  • -XX: MaxDirectMemory = 100M disposed outside the maximum heap memory
• Other commonly used parameters
  • -XX: PrintGCDetails print GC Details
  • -XX: + timestamp when PrintGCTimeStamps print GC
  • -Xloggc: /home/jerry/logs/gc.log set the GC log output directory
  • -XX: + HeapDumpOnOutOfMemoryError When memory overflow occurs when abnormal, dump the stack information
  • -XX:. HeapDumpPath = / my_java.hprof specified output path stack dump information
  • -XX: MaxTenuringThreshold = 8 after undergoing settings object in the young generation garbage collection is still alive eight years old to enter. Maximum value 15 (indicating that the value because the JVM 4bit), the default value of 15 (when using CMS, the default is 6)
  • -XX: + PrintFlagsFinal print JVM final parameters (e.g. java -XX: + PrintFlagsFinal -version)
  • -XX: + PrintFlagsInitial print JVM initial parameter (final value environment changes)