One, background
If at the time of RDD persistence, persistence of large amounts of data, then the garbage collector Java virtual machine may become a performance bottleneck. Because the Java virtual machine garbage collection on a regular basis, then it will keep track of all the java objects,
and the garbage collector to find those objects no longer used, and then clean up the old objects to free up memory for the new object.
Garbage collection performance overhead is with the number of objects in memory, and proportional. Therefore, the performance of garbage collection problem, the first thing is, using a more efficient data structures, such as array and String; second is when persistent RDD,
using persistent level of sequence, and the sequence of the library with Kryo such that each partition just one object - an array of bytes.
Second, monitoring garbage collection
We can monitor garbage collection, including how often garbage collection, and the time spent each time garbage collection. As long as the spark- the Submit script, you can add a configuration, --conf "spark.executor.extraJavaOptions = -verbose: gc -XX: + PrintGCDetails -XX: + PrintGCTimeStamps" . But remember, although there will print out information about garbage collection of the Java virtual machine, but it is output to a log on the worker, not the driver's log. But this is only a way, in fact, entirely possible to observe the situation of each stage of garbage collection by SparkUI (4040 port).
Third, the memory optimization executor proportion
1, illustrated
2. Description
对于垃圾回收来说，最重要的就是调节RDD缓存占用的内存空间，与算子执行时创建的对象占用的内存空间的比例。默认情况下，Spark使用每个executor 60%的内存空间来缓存RDD， 那么在task执行期间创建的对象，只有40%的内存空间来存放。 在这种情况下，很有可能因为你的内存空间的不足，task创建的对象过大，那么一旦发现40%的内存空间不够用了，就会触发Java虚拟机的垃圾回收操作。因此在极端情况下， 垃圾回收操作可能会被频繁触发。 在上述情况下，如果发现垃圾回收频繁发生。那么就需要对那个比例进行调优，使用new SparkConf().set("spark.storage.memoryFraction", "0.5")即可，可以将RDD缓存占用空间的比例降低， 从而给更多的空间让task创建的对象进行使用。 因此，对于RDD持久化，完全可以使用Kryo序列化，加上降低其executor内存占比的方式，来减少其内存消耗。给task提供更多的内存，从而避免task的执行频繁触发垃圾回收。
四、高级的垃圾回收调优
1、图解
2、说明
Java堆空间被划分成了两块空间，一个是年轻代，一个是老年代。年轻代放的是短时间存活的对象，老年代放的是长时间存活的对象。年轻代又被划分了三块空间，Eden、Survivor1、Survivor2。 首先，Eden区域和Survivor1区域用于存放对象，Survivor2区域备用。创建的对象，首先放入Eden区域和Survivor1区域，如果Eden区域满了，那么就会触发一次Minor GC， 进行年轻代的垃圾回收。Eden和Survivor1区域中存活的对象，会被移动到Survivor2区域中。然后Survivor1和Survivor2的角色调换，Survivor1变成了备用。 如果一个对象，在年轻代中，撑过了多次垃圾回收，都没有被回收掉，那么会被认为是长时间存活的，此时就会被移入老年代。此外，如果在将Eden和Survivor1中的存活对象， 尝试放入Survivor2中时，发现Survivor2放满了，那么会直接放入老年代。此时就出现了，短时间存活的对象，进入老年代的问题。 如果老年代的空间满了，那么就会触发Full GC，进行老年代的垃圾回收操作。 Spark中，垃圾回收调优的目标就是，只有真正长时间存活的对象，才能进入老年代，短时间存活的对象，只能呆在年轻代。不能因为某个Survivor区域空间不够， 在Minor GC时，就进入了老年代。从而造成短时间存活的对象，长期呆在老年代中占据了空间，而且Full GC时要回收大量的短时间存活的对象，导致Full GC速度缓慢。 如果发现，在task执行期间，大量full gc发生了，那么说明，年轻代的Eden区域，给的空间不够大。此时可以执行一些操作来优化垃圾回收行为： 1Including reducing the proportion of spark.storage.memoryFraction, giving the young generation more space to store objects short-term survival; 2, allocated more space to the Eden area, you can use -Xmn generally recommended to Eden area, 4/3 of the expected size ; 3 , if the HDFS file, then a good estimate Eden area size, if each has four Task executor, then each compressed block decompressor hdfs size is 3 times, in addition to each hdfs block size is 64M, then the expected area is the size of Eden: 4 * 3 * 64MB, then, again through -Xmn parameter, the Eden area size is set to 64 * 3 * 4 * 4/3 . ## summary In fact, ah, based on experience, for tuning garbage collection, that is as far as possible, adjust the ratio executor of memory on it. Because jvm tuning is very complicated and sensitive. Unless it is really a last resort to the place, then, for themselves and related technical jvm very understanding, then the time to adjust eden region, tuning is possible. Some advanced parameters -XX: SurvivorRatio = 4: If the value is 4, it is the ratio of the two Survivor with Eden is 2: 4, that is to say the proportion of each Survivor occupied by the young generation is 1/6 , so, you in fact, you can also try the size of the major areas of Survivor. -XX: NewRatio = 4: adjusting the proportion of the new generation and the old age