JVM调优--实例

背景：脚本任务，密集执行，需要高吞吐量

sh-4.2# jstat -gc 64
 S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT
34048.0 34048.0  0.0   34048.0 272640.0 219162.0 5950720.0  5653530.1  91580.0 88224.0 10200.0 9627.7 923832 88231.524 27146 81424.338 169655.863

ygc同样非常频繁3次/s

查看得知fgc特别频繁，大概4次/min

sh-4.2# jinfo -flags 64
Attaching to process ID 64, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.181-b13
Non-default VM flags: -XX:CICompilerCount=3 -XX:CMSInitiatingOccupancyFraction=70 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=null -XX:InitialHeapSize=6442450944 -XX:+ManagementServer -XX:MaxHeapSize=6442450944 -XX:MaxNewSize=348913664 -XX:MaxTenuringThreshold=6 -XX:MinHeapDeltaBytes=196608 -XX:NewSize=348913664 -XX:OldPLABSize=16 -XX:OldSize=6093537280 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC-XX:+UseParNewGC

排查记录

查看配置参数：

-XX:CICompilerCount=3 -XX:CMSInitiatingOccupancyFraction=70 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=null -XX:InitialHeapSize=6442450944 -XX:+ManagementServer -XX:MaxHeapSize=6442450944 -XX:MaxNewSize=348913664 -XX:MaxTenuringThreshold=6 -XX:MinHeapDeltaBytes=196608 -XX:NewSize=348913664 -XX:OldPLABSize=16 -XX:OldSize=6093537280 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC-XX:+UseParNewGC

a. 非常不可理解的是：

MaxTenuringThreshold=6

原因

查看文章:https://www.jianshu.com/p/f39cc983a31b原来这是使用CMS的默认配置，在本实例中ygc频率十分高，在此情况下MaxTenuringThreshold应该调高，充分利用年轻代复制算法的速度优势。

解决：

调整MaxTenuringThreshold=15

b. 各块的大小

   MinHeapFreeRatio         = 40
   MaxHeapFreeRatio         = 70
   MaxHeapSize              = 6442450944 (6144.0MB)
   NewSize                  = 348913664 (332.75MB)
   MaxNewSize               = 348913664 (332.75MB)
   OldSize                  = 6093537280 (5811.25MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 17592186044415 MB
   G1HeapRegionSize         = 0 (0.0MB)

suvivor 1 2: 34048 (33.25M)

eden: 272640(266.25M)

old:5950720(5811.25M)

metaspace:91580(89.43M)

old/ (eden+2*suvivor) 大概在17-18的样子

eden/suvivor = 8

原因：

同样这是CMS垃圾收集器的默认配置：https://www.jianshu.com/p/f39cc983a31b

导致年轻代的大小不走默认的堆1/3 而使用新的算法，具体看文章，总之，最终导致年轻代只分配到332.75M

sh-4.2# jstat -gc 64
 S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT
34048.0 34048.0  0.0   34048.0 272640.0 219162.0 5950720.0  5653530.1  91580.0 88224.0 10200.0 9627.7 923832 88231.524 27146 81424.338 169655.863

很显然，FGC消耗的时间远远大于YGC，既然如此，那通过减少FGC的次数，不失为一个可以尝试的方法。

解决办法：

Sun官方建议年轻代的大小为整个堆的3/8左右。

其次使用CMS最好总heap大于8G，这个可以再申请资源。

所以我们配置的年轻代远远不够，应当加上配置：-xmn=1G(保守一点)

思路总体而言：

让GC尽量发生在YGC阶段，CMS尽管很快，但是终究不如年轻代的复制算法来的快，而且CMS还会占用很多的cpu资源。

按照预期，效果应该可以达到两方面：

1. 年轻代变大->YGC次数应该下降 -- checking

2. MaxTenuringThreshold变大->对象在年轻代停留时间更长

========================================================

以上为第一阶段排查，有待验证调整以后的运行效果

========================================================

修改之前的gc数据

========================================================

sh-4.2# jinfo -flags 64
Attaching to process ID 64, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.181-b13
Non-default VM flags: -XX:CICompilerCount=3 -XX:CMSInitiatingOccupancyFraction=70 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=null -XX:InitialHeapSize=6442450944 -XX:+ManagementServer -XX:MaxHeapSize=6442450944 -XX:MaxNewSize=348913664 -XX:MaxTenuringThreshold=6 -XX:MinHeapDeltaBytes=196608 -XX:NewSize=348913664 -XX:OldPLABSize=16 -XX:OldSize=6093537280 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC-XX:+UseParNewGC
Command line:  -Xms6g -Xmx6g -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=70 -Xloggc:/data/dom.flow_job/logs/gc_20181011_214151.log -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/data/dom.flow_job/tmp/heapdump_20181011_214151.hprof -Djava.io.tmpdir=/data/dom.flow_job/tmp/ -Duser.dir=/data/dom.flow_job -DTEAM=pcd -DAPPID=dom.flow_job -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=8860

sh-4.2# jstat -gc 64 10000 10
 S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT
34048.0 34048.0 34046.8 34046.5 272640.0 272640.0 5950720.0  4691756.4  91708.0 88236.7 10328.0 9629.4 1023663 97920.980 30377 91954.513 189875.493
34048.0 34048.0  0.0   34048.0 272640.0 272640.0 5950720.0  1791237.5  91708.0 88236.7 10328.0 9629.4 1023689 97923.403 30378 91960.393 189883.796
34048.0 34048.0 34048.0  0.0   272640.0 272640.0 5950720.0  5937012.1  91708.0 88236.7 10328.0 9629.4 1023746 97929.158 30380 91960.398 189889.556
34048.0 34048.0 34048.0 34048.0 272640.0 272640.0 5950720.0  3264125.5  91708.0 88236.7 10328.0 9629.4 1023771 97931.354 30381 91967.580 189898.934
34048.0 34048.0 34048.0  0.0   272640.0 272640.0 5950720.0  5912416.4  91708.0 88236.7 10328.0 9629.4 1023807 97934.996 30382 91967.580 189902.576
34048.0 34048.0 34046.8 34047.0 272640.0 272640.0 5950720.0  5069508.9  91708.0 88236.7 10328.0 9629.4 1023861 97940.244 30383 91973.623 189913.867
34048.0 34048.0  0.0   34048.0 272640.0 30730.9  5950720.0  1866341.9  91708.0 88236.7 10328.0 9629.4 1023881 97942.222 30384 91980.416 189922.638
34048.0 34048.0 34048.0  0.0   272640.0 272640.0 5950720.0  5884673.4  91708.0 88236.7 10328.0 9629.4 1023937 97947.376 30386 91980.423 189927.800
34048.0 34048.0 34048.0 34048.0 272640.0 272640.0 5950720.0  3791197.8  91708.0 88236.7 10328.0 9629.4 1023975 97950.947 30387 91986.121 189937.068
34048.0 34048.0  0.0   34048.0 272640.0 272640.0 5950720.0  5910167.9  91708.0 88236.7 10328.0 9629.4 1024007 97954.219 30388 91986.121 189940.340

========================================================

修改之后的gc数据

========================================================

1. 调大年轻代:-xmn=1g

2. 调大 MaxTenuringThreshold=15(没想到这个最大值就是15)

FGC次数明显下降，但是FGCT反而增加了,此次修改失败告终。

2. 查看日志

2018-10-14T12:29:54.275+0800: 226083.035: [GC (Allocation Failure) 2018-10-14T12:29:54.275+0800: 226083.035: [ParNew: 306688K->34048K(306688K), 0.0981023 secs] 5540915K->5356373K(6257408K), 0.0985620 secs] [Times: user=0.36 sys=0.00, real=0.10 secs]

a. Allocation Failure

"Allocation Failure" means that no more space left in Eden to allocate object. So, it is normal cause of young GC.

"Allocation Failure" is almost only possible cause for minor GC. Another reason for minor GC to kick could be CMS remark phase (if +XX:+ScavengeBeforeRemark is enabled).

这是正常的Minor GC的原因

备注：

有意思的是306688从何而来？ = eden + suvivor；符合复制算法的意思，同时只有一个suvivor处于使用状态

b. concurrent mode failure

2018-10-14T14:09:29.938+0800: 232058.699: [GC (Allocation Failure) 2018-10-14T14:09:29.939+0800: 232058.699: [ParNew: 306688K->306688K(306688K), 0.0000487 secs]2018-10-14T14:09:29.939+0800: 232058.699: [CMS2018-10-14T14:09:32.377+0800: 232061.137: [CMS-concurrent-mark: 4.767/9.752 secs] [Times: user=51.83 sys=0.85, real=9.75 secs]
 (concurrent mode failure): 5864242K->1431263K(5950720K), 6.4683532 secs] 6170930K->1431263K(6257408K), [Metaspace: 88240K->88240K(1130496K)], 6.4689451 secs] [Times: user=6.46 sys=0.00, real=6.47 secs]

发生极其频繁，两次CMS GC就会有一次

原因：

https://blog.csdn.net/yangguosb/article/details/79857844

https://www.jianshu.com/p/ca1b0d4107c5

调整：

XX:CMSInitiatingOccupancyFraction调整，这个等第一步优化完成再进行，生成原因简单来说就是在扫描期间对象进入老年代的速度太快了！

c. promotion failed

[GC (Allocation Failure) 2018-10-14T14:25:21.674+0800: 233010.434: [ParNew (promotion failed): 306688K->306688K(306688K), 0.1197228 secs]2018-10-14T14:25:21.794+0800: 233010.554: [CMS2018-10-14T14:25:24.223+0800: 233012.983: [CMS-concurrent-mark: 5.050/10.078 secs] [Times: user=54.15 sys=0.93, real=10.08 secs]

原因：

《深入理解java虚拟机》p98说的很清楚：

担保失败了：新生代在YGC后存活的对象进入老年代，老年代没撑住。

此时会触发一次FGC，查看日志，该异常率很低，符合预期，可以不管。

解决：

尝试 -XX:CMSInitiatingOccupancyFraction=40，仍然会大量报错concurrent mode failure。这时候开始感觉到CMS作为老年代GC已经不适合这种场景了，将-XX:+UseConcMarkSweepGC替换为-XX:+UseParallelOldGC

========================================================

以上为第二阶段排查，有待验证调整以后的运行效果

========================================================

修改之后的gc数据

========================================================

sh-4.2# jstat -gc 67 10000 10
 S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT
629248.0 626176.0 543994.5  0.0   4336640.0 168778.4 11185152.0 9694337.1  92032.0 88564.8 10368.0 9684.9  46095 9889.856 1344  1315.587 11205.443
654336.0 378880.0  0.0   378432.5 4293632.0 2059764.4 11185152.0 10370661.4 92032.0 88564.8 10368.0 9684.9  46108 9891.264 1344  1315.587 11206.851
696832.0 133120.0  0.0   132843.3 4218880.0 1375817.7 11185152.0 10449455.2 92032.0 88564.8 10368.0 9684.9  46116 9891.724 1344  1315.587 11207.311
778240.0 804864.0 554211.1  0.0   3982336.0 2129352.4 11185152.0 10773913.9 92032.0 88564.8 10368.0 9684.9  46121 9892.262 1344  1315.587 11207.849
765440.0 757248.0  0.0   589650.9 4062208.0 4062208.0 11185152.0 2227491.8  92032.0 88564.8 10368.0 9684.9  46135 9894.012 1345  1316.251 11210.262
683008.0 685056.0  0.0   581235.9 4208640.0 3267721.0 11185152.0 3840590.5  92032.0 88564.8 10368.0 9684.9  46148 9895.942 1345  1316.251 11212.193
650240.0 653824.0  0.0   584854.5 4277248.0 366382.3 11185152.0 5454997.3  92032.0 88564.8 10368.0 9684.9  46162 9897.933 1345  1316.251 11214.183
650752.0 656384.0 599174.0  0.0   4279296.0 2742465.9 11185152.0 6593887.9  92032.0 88564.8 10368.0 9684.9  46175 9899.634 1345  1316.251 11215.885
643584.0 647168.0 382630.4  0.0   4297728.0 1224622.4 11185152.0 7319151.5  92032.0 88564.8 10368.0 9684.9  46189 9901.072 1345  1316.251 11217.323
709632.0 734208.0 47833.5  0.0   4123648.0 411668.7 11185152.0 7368448.7  92032.0 88564.8 10368.0 9684.9  46195 9901.346 1345  1316.251 11217.597

不愧为高吞吐量优先的GC算法，比CMS更适合多线程脚本执行的这种场景。

3. 继续优化--待补充

Parallel算法的配置参数属性

查看代码，dump，是否合理

其他：

1. 64位操作系统最大支持的内存是多少呢？ 2^64 B 特别特别大

2. https://blog.csdn.net/fanwu72/article/details/8936746

init约等于xms的值，max约等于xmx的值。used是已经被使用的内存大小，committed是当前可使用的内存大小（包括已使用的），committed >= used。committed不足时jvm向系统申请，若超过max则发生OutOfMemoryError错误。

猜你喜欢