JVM tuning - the CMS GC log analysis

A recent study JVM and GC tuning, today summarizes some characteristics and points at CMS, let us briefly look at the whole heap of the young generation and the old generation garbage collection is a combination of (the perfect fit java8 support, other versions may be slightly there are different), which marked the red line is we have to focus on to talk about today:

Garbage collector may be used in combination:

ParNew and CMS

"Concurrent Mark and Sweep" CMS is the full name, the official name given is: "Mostly Concurrent Mark and Sweep Garbage Collector";

The young generation: using at The-world-STOP Mark-Copy algorithm;

Old generation: Concurrent use Mostly Mark-Sweep Algorithm;

Design goals: the old generation collection time to avoid long pauses;

This goal can be reached mainly because of two reasons:

1 It will not take time to sort out the old generation compression, but maintains a data structure called a free-lists, the data structure used to manage those recycling memory space;

2 mark-sweep is divided into several stages, of which a large part of the stage and GC work was performed while working Application threads (of course, time gc thread and user threads competing for CPU's), the default GC worker threads for you 1/4 of the number of physical CPU core server;

Added: When your server is multicore same time your goal is to low delay, and that the GC mix is your best choice.

Journal

GC logs first experience

First, there is a general awareness of the entire GC logs

2016-08-23T02:23:07.219-0200: 64.322: [GC (Allocation Failure) 64.322: [ParNew: 613404K->68068K(613440K), 0.1020465 secs] 10885349K->10880154K(12514816K), 0.1021309 secs] [Times: user=0.78 sys=0.01, real=0.11 secs]2016-08-23T02:23:07.321-0200: 64.425: [GC (CMS Initial Mark) [1 CMS-initial-mark: 10812086K(11901376K)] 10887844K(12514816K), 0.0001997 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-08-23T02:23:07.321-0200: 64.425: [CMS-concurrent-mark-start]
2016-08-23T02:23:07.357-0200: 64.460: [CMS-concurrent-mark: 0.035/0.035 secs] [Times: user=0.07 sys=0.00, real=0.03 secs]
2016-08-23T02:23:07.357-0200: 64.460: [CMS-concurrent-preclean-start]
2016-08-23T02:23:07.373-0200: 64.476: [CMS-concurrent-preclean: 0.016/0.016 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-08-23T02:23:07.373-0200: 64.476: [CMS-concurrent-abortable-preclean-start]
2016-08-23T02:23:08.446-0200: 65.550: [CMS-concurrent-abortable-preclean: 0.167/1.074 secs] [Times: user=0.20 sys=0.00, real=1.07 secs]
2016-08-23T02:23:08.447-0200: 65.550: [GC (CMS Final Remark) [YG occupancy: 387920 K (613440 K)]65.550: [Rescan (parallel) , 0.0085125 secs]65.559: [weak refs processing, 0.0000243 secs]65.559: [class unloading, 0.0013120 secs]65.560: [scrub symbol table, 0.0008345 secs]65.561: [scrub string table, 0.0001759 secs][1 CMS-remark: 10812086K(11901376K)] 11200006K(12514816K), 0.0110730 secs] [Times: user=0.06 sys=0.00, real=0.01 secs]
2016-08-23T02:23:08.458-0200: 65.561: [CMS-concurrent-sweep-start]
2016-08-23T02:23:08.485-0200: 65.588: [CMS-concurrent-sweep: 0.027/0.027 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
2016-08-23T02:23:08.485-0200: 65.589: [CMS-concurrent-reset-start]
2016-08-23T02:23:08.497-0200: 65.601: [CMS-concurrent-reset: 0.012/0.012 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]

Minor GC

2016-08-23T02:23:07.219-02001: 64.3222:[GC3(Allocation Failure4) 64.322: [ParNew5: 613404K->68068K6(613440K)7, 0.1020465 secs8] 10885349K->10880154K9(12514816K)10, 0.1021309 secs11][Times: user=0.78 sys=0.01, real=0.11 secs]12

1. 2016-08-23T02: 23: 07.219-0200 - time GC occurred;
2. 64.322 - GC start, the relative time is relatively JVM start in seconds;
3. GC - difference MinorGC and FullGC logo, this represents MinorGC;
4. Allocation Failure - MinorGC reason, in this case inside, the young generation is not satisfied due to space applications, thus triggering the MinorGC;
5. ParNew - the name of the collector, it foreshadowed the young generation to use a parallel mark-copy stop-the-world garbage collector;
6. 613404K-> 68068K - gather around the young generation usage;
7. (613440K) - entire young generation capacity;
8. 0.1020465 secs - interpretation by this explanation original flavor: Duration for the collection w / o final cleanup.
9. 10885349K-> 10880154K - before and after the use of the entire collected heap;
10. (12514816K) - capacity of the entire stack;
11. 0.1021309 secs - ParNew collector marks the young generation and replication time spent living objects (including and communication overhead years old, was promoted to the object's old time, garbage collection cycle ended spending some final clean up the object, etc.);
12. [Times: user = 0.78 sys = 0.01, real = 0.11 secs] - GC events took a different dimension, specific use English to explain more reasonable:
- - user – Total CPU time that was consumed by Garbage Collector threads during this collection
  - sys – Time spent in OS calls or waiting for system event
  - real – Clock time for which your application was stopped. With Parallel GC this number should be close to (user time + system time) divided by the number of threads used by the Garbage Collector. In this particular case 8 threads were used. Note that due to some activities not being parallelizable, it always exceeds the ratio by a certain amount.

We have to analyze objects promoted to question (calculated in the original text in question) :

Beginning: whole heap size is 10885349K , the size of the young generation is 613404K , which shows old's size = 10885349-613404 10271945K ,

After the collection is complete: the whole heap size is 10880154K , the size of the young generation is 68068K , which shows old's size = 10880154-68068 10812086K ,

Increasing the size of old age: 10812086-10271945 = 608209K , that is to say the young generation to the old generation promot the 608209K data;

Graphical analysis :

Full/Major GC

2016-08-23T11:23:07.321-0200: 64.425: [GC (CMS Initial Mark)1 [1 CMS-initial-mark: 10812086K(11901376K)] 10887844K(12514816K), 0.0001997 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-08-23T11:23:07.321-0200: 64.425: [CMS-concurrent-mark-start]
2016-08-23T11:23:07.357-0200: 64.460: [: 0.035/0.035 secs] [Times: user=0.07 sys=0.00, real=0.03 secs]
2016-08-23T11:23:07.357-0200: 64.460: [CMS-concurrent-preclean-start]
2016-08-23T11:23:07.373-0200: 64.476: [CMS-concurrent-preclean3: 0.016/0.016 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-08-23T11:23:07.373-0200: 64.476: [CMS-concurrent-abortable-preclean-start]
2016-08-23T11:23:08.446-0200: 65.550: [CMS-concurrent-abortable-preclean4: 0.167/1.074 secs] [Times: user=0.20 sys=0.00, real=1.07 secs]
2016-08-23T11:23:08.447-0200: 65.550: [GC (CMS Final Remark5)
[YG occupancy: 387920 K (613440 K)]65.550: [Rescan (parallel) , 0.0085125 secs]65.559: 
[weak refs processing, 0.0000243 secs]65.559: [class unloading, 0.0013120 secs]65.560: 
[scrub symbol table, 0.0008345 secs]65.561: [scrub string table, 0.0001759 secs][1 CMS-remark: 10812086K(11901376K)] 11200006K(12514816K), 0.0110730 secs] 
[Times: user=0.06 sys=0.00, real=0.01 secs]
2016-08-23T11:23:08.458-0200: 65.561: [CMS-concurrent-sweep-start]
2016-08-23T11:23:08.485-0200: 65.588: [CMS-concurrent-sweep6: 0.027/0.027 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
2016-08-23T11:23:08.485-0200: 65.589: [CMS-concurrent-reset-start]
2016-08-23T11:23:08.497-0200: 65.601: [CMS-concurrent-reset7: 0.012/0.012 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]

Phase 1: Initial Mark

This is twice the CMS stop-the-world event once. It has two objectives: first, to mark all the GC Roots old era; second is to mark the referenced objects in the young generation living objects.

Mark results are as follows :

analysis:

2016-08-23T11:23:07.321-0200: 64.421: [GC (CMS Initial Mark2[1 CMS-initial-mark: 10812086K3(11901376K)4] 10887844K5(12514816K)6, 0.0001997 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]7

1. 016-08-23T11: 23: 07.321-0200: 64.42 - GC event starts, including clock time and the relative time relative JVM startup time, the same below the meaning of all the stages of the time change;
2. CMS Initial Mark - collection phase, started collecting all the GC Roots and direct reference to the object;
3. 10812086K - old's current usage;
4. (11901376K) - Old's available capacity;
5. 10887844K - Current whole heap usage;
6. (12514816K) - capacity of the entire stack;
7. 0.0001997 secs] [Times: user = 0.00 sys = 0.00, real = 0.00 secs] - time measurement;

Phase 2: Concurrent Mark

This stage goes through the entire years old and marks all live objects, found from the "Initial mark" phase GC Roots begin. Concurrent mark and is characterized by the application threads to run simultaneously. All live objects are not old age is marked, labeled the same time because the application will change some object references and so on.

Mark results are as follows :

In FIG upper side, the arrow has a reference current away from the object (current obj)

Analysis :

2016-08-23T11:23:07.321-0200: 64.425: [CMS-concurrent-mark-start]
2016-08-23T11:23:07.357-0200: 64.460: [CMS-concurrent-mark1: 035/0.035 secs2] [Times: user=0.07 sys=0.00, real=0.03 secs]3

1. CMS-concurrent-mark - concurrent collection phase, which will traverse the entire old generation and mark objects alive;
2. 035 / 0.035 secs - Display the duration of this phase and clock time;
3. [Times: user=0.07 sys=0.00, real=0.03 secs] – 同上

Phase 3: Concurrent Preclean

这个阶段又是一个并发阶段，和应用线程并行运行，不会中断他们。前一个阶段在并行运行的时候，一些对象的引用已经发生了变化，当这些引用发生变化的时候，JVM会标记堆的这个区域为Dirty Card(包含被标记但是改变了的对象，被认为"dirty")，这就是 Card Marking。

在pre-clean阶段，那些能够从dirty card对象到达的对象也会被标记，这个标记做完之后，dirty card标记就会被清除了，如下：

另外，一些必要的清扫工作也会做，还会做一些final remark阶段需要的准备工作；

分析

2016-08-23T11:23:07.357-0200: 64.460: [CMS-concurrent-preclean-start]
2016-08-23T11:23:07.373-0200: 64.476: [CMS-concurrent-preclean1: 0.016/0.016 secs2] [Times: user=0.02 sys=0.00, real=0.02 secs]3

1. CMS-concurrent-preclean – 这个阶段负责前一个阶段标记了又发生改变的对象标记；
2. 0.016/0.016 secs – 展示该阶段持续的时间和时钟时间；
3. [Times: user=0.02 sys=0.00, real=0.02 secs] – 同上

Phase 4: Concurrent Abortable Preclean

又一个并发阶段不会停止应用程序线程。这个阶段尝试着去承担STW的Final Remark阶段足够多的工作。这个阶段持续的时间依赖好多的因素，由于这个阶段是重复的做相同的事情直到发生aboart的条件（比如：重复的次数、多少量的工作、持续的时间等等）之一才会停止。

2016-08-23T11:23:07.373-0200: 64.476: [CMS-concurrent-abortable-preclean-start]
2016-08-23T11:23:08.446-0200: 65.550: [CMS-concurrent-abortable-preclean1: 0.167/1.074 secs2] [Times: user=0.20 sys=0.00, real=1.07 secs]3

1. CMS-concurrent-abortable-preclean – 可终止的并发预清理；
2. 0.167/1.074 secs – 展示该阶段持续的时间和时钟时间（It is interesting to note that the user time reported is a lot smaller than clock time. Usually we have seen that real time is less than user time, meaning that some work was done in parallel and so elapsed clock time is less than used CPU time. Here we have a little amount of work – for 0.167 seconds of CPU time, and garbage collector threads were doing a lot of waiting. Essentially, they were trying to stave off for as long as possible before having to do an STW pause. By default, this phase may last for up to 5 seconds）；
3. [Times: user=0.20 sys=0.00, real=1.07 secs] – 同上

这个阶段很大程度的影响着即将来临的Final Remark的停顿，有相当一部分重要的 configuration options 和失败的模式；

Phase 5: Final Remark

这个阶段是CMS中第二个并且是最后一个STW的阶段。该阶段的任务是完成标记整个年老代的所有的存活对象。由于之前的预处理是并发的，它可能跟不上应用程序改变的速度，这个时候，STW是非常需要的来完成这个严酷考验的阶段。

通常CMS尽量运行Final Remark阶段在年轻代是足够干净的时候，目的是消除紧接着的连续的几个STW阶段。

分析：

2016-08-23T11:23:08.447-0200: 65.5501: [GC (CMS Final Remark2) [YG occupancy: 387920 K (613440 K)3]65.550: [Rescan (parallel) , 0.0085125 secs]465.559: [weak refs processing, 0.0000243 secs]65.5595: [class unloading, 0.0013120 secs]65.5606: [scrub string table, 0.0001759 secs7][1 CMS-remark: 10812086K(11901376K)8] 11200006K(12514816K) 9, 0.0110730 secs10] [[Times: user=0.06 sys=0.00, real=0.01 secs]11

1. 2016-08-23T11:23:08.447-0200: 65.550 – 同上；
2. CMS Final Remark – 收集阶段，这个阶段会标记老年代全部的存活对象，包括那些在并发标记阶段更改的或者新创建的引用对象；
3. YG occupancy: 387920 K (613440 K) – 年轻代当前占用情况和容量；
4. [Rescan (parallel) , 0.0085125 secs] – 这个阶段在应用停止的阶段完成存活对象的标记工作；
5. weak refs processing, 0.0000243 secs]65.559 – 第一个子阶段，随着这个阶段的进行处理弱引用；
6. class unloading, 0.0013120 secs]65.560 – 第二个子阶段(that is unloading the unused classes, with the duration and timestamp of the phase);
7. scrub string table, 0.0001759 secs – 最后一个子阶段（that is cleaning up symbol and string tables which hold class-level metadata and internalized string respectively）
8. 10812086K(11901376K) – 在这个阶段之后老年代占有的内存大小和老年代的容量；
9. 11200006K(12514816K) – 在这个阶段之后整个堆的内存大小和整个堆的容量；
10. 0.0110730 secs – 这个阶段的持续时间；
11. [Times: user=0.06 sys=0.00, real=0.01 secs] – 同上；

通过以上5个阶段的标记，老年代所有存活的对象已经被标记并且现在要通过Garbage Collector采用清扫的方式回收那些不能用的对象了。

Phase 6: Concurrent Sweep

和应用线程同时进行，不需要STW。这个阶段的目的就是移除那些不用的对象，回收他们占用的空间并且为将来使用。

如图：

分析：

2016-08-23T11:23:08.458-0200: 65.561: [CMS-concurrent-sweep-start] 2016-08-23T11:23:08.485-0200: 65.588: [CMS-concurrent-sweep1: 0.027/0.027 secs2] [[Times: user=0.03 sys=0.00, real=0.03 secs] 3

1. CMS-concurrent-sweep – 这个阶段主要是清除那些没有标记的对象并且回收空间；
2. 0.027/0.027 secs – 展示该阶段持续的时间和时钟时间；
3. [Times: user=0.03 sys=0.00, real=0.03 secs] – 同上

Phase 7: Concurrent Reset

这个阶段并发执行，重新设置CMS算法内部的数据结构，准备下一个CMS生命周期的使用。

2016-08-23T11:23:08.485-0200: 65.589: [CMS-concurrent-reset-start] 2016-08-23T11:23:08.497-0200: 65.601: [CMS-concurrent-reset1: 0.012/0.012 secs2] [[Times: user=0.01 sys=0.00, real=0.01 secs]3

1. 1. CMS-concurrent-reset – 这个阶段重新设置CMS算法内部的数据结构，为下一个收集阶段做准备；
  2. 0.012/0.012 secs – 展示该阶段持续的时间和时钟时间；
  3. [Times: user=0.01 sys=0.00, real=0.01 secs] – 同上

CMS GC log analysis