[Switch] JVM gc parameter setting and analysis

original:

http://hi.baidu.com/i1see1you/item/295c1dc81f91ab55bdef69e5
gc log analysis tool: http://qa.blog.163.com/blog/static/19014700220128199421589/
Java GC log diagram: http://www.chinasb.org/archives/2012/09/4921.shtml

Overview

The biggest advantage of java is automatic garbage collection, which eliminates the need for us to manually release the object space, but it also has a corresponding negative effect. GC requires time and resources, and a bad GC will seriously affect the system performance. Therefore, Good gc is the guarantee of high performance of JVM. The JVM heap is divided into the new generation, the old generation and the old generation. The available gc methods for the new generation are: serial gc (Serial Copying), parallel recycling gc (Parellel Scavenge), parallel gc (ParNew), old generation and year The available gc modes of the old generation include serial gc (Serial MSC), parallel gc (Parallel MSC), and concurrent gc (CMS).

Choice of recycling methods

JVM has two modes: client and server. The default gc modes of these two modes are different:

In client mode, the new generation selects serial gc, and the old generation selects serial gc

In server mode, the new generation selects parallel recycling gc, and the old generation selects parallel gc

Generally speaking, there are two ways for our system application selection: throughput priority and pause time priority. For throughput priority, the default parallel gc mode of server is adopted, and for pause time priority, concurrent gc (CMS) mode is selected.

CMS gc

CMS, the full name of Concurrent Low Pause Collector, is a new gc algorithm introduced in the later version of jdk1.4. It has been further improved in jdk5 and jdk6. It is mainly suitable for scenarios where the importance of response time is greater than the throughput requirement. , can bear the application of garbage collection thread and application thread sharing processor resources, and there are many objects with long life cycle in the application. CMS is used to recycle the tenured generation, that is, the old generation. The goal is to minimize the pause time of the application, reduce the probability of full gc, and use the garbage collection thread concurrent with the application thread to mark and clear the old generation. . In our application, because of the existence of cache and relatively high requirements for response time, we hope to try to use CMS to replace the parallel collector used by the default server-type JVM, so as to obtain shorter garbage collection pauses time and improve the responsiveness of the program.

CMS does not have no pauses, but replaces the long pauses of the serial mark sorting algorithm with two short pauses. Its collection cycle is as follows:

Initial mark (CMS-initial-mark) -> concurrent mark (CMS-concurrent-mark) -> remark (CMS-remark) -> concurrent clear (CMS-concurrent-sweep) -> concurrent reset state waiting for the next CMS trigger (CMS-concurrent-reset).

Among them, steps 1 and 3 need to suspend all application threads. The first pause starts from the root object to mark surviving objects, and this phase is called initial marking; the second pause is after concurrent marking, suspending all application threads, and remarking objects missed in the concurrent marking phase (at the end of the concurrent marking phase caused by an update of the object's state). The first pause will be shorter, the second pause is usually longer, and the remark stage can be marked in parallel.

The so-called concurrency in the concurrent mark, concurrent clear, and concurrent reset phases refers to one or more garbage collection threads and application threads running concurrently. The garbage collection thread will not suspend the execution of the application. If you have more than one processing If the processor is used, the concurrent collection thread will run on a different processor from the application thread. Obviously, this overhead will reduce the throughput of the application. The parallelism of the Remark phase means that after all applications are suspended, a certain number of garbage collection processes are started for parallel marking, and the application threads are suspended at this time.

full gc

Full gc is the unified recovery of the new generation, the old generation, and the persistent generation. Because it is the recovery of the entire space, it is relatively slow, and the number of full gc should be minimized in the system.

Full gc occurs in the following situations:

Insufficient space in the old generation
Insufficient Persistent Generation Space
Promotion failed and concurrent mode failure occurred during CMS GC
When statistics get the minor gc of the new generation, the average size of the promotion to the old generation is smaller than the remaining space of the old generation
Call System.gc directly, you can DisableExplicitGC to prohibit
When there is an rmi call, System.gc will be executed every minute by default. You can set a larger interval by -Dsun.rmi.dgc.server.gcInterval=3600000.

Gc log parameters

Generate gc logs by adding relevant parameters to the tomcat startup script

-verbose.gcThe switch shows what the GC is doing. Turn it on to display when the busiest and most idle collection behavior occurs, the amount of memory before and after collection, the time it takes to collect, and more.
Turn on the -xx:+printGCdetailsswitch to learn more about the changes in the GC.
Turn on the -XX:+PrintGCTimeStampsswitch to see when these garbage collections occur, measured in seconds since the JVM started.
Finally, -xx:+PrintHeapAtGCget more detailed information about the heap through the switch.
In order to understand the situation of the new domain, you can use -XX:+PrintTenuringDistributionthe switch to understand the object right to obtain the usage period.
-Xloggc:$CATALINA_BASE/logs/gc.log Path generated by gc log
-XX:+PrintGCApplicationStoppedTime Output the time the application was paused by the GC
-XX:+PrintGCDateStamps Information about the time when GC occurred

Opentsdb open Gc parameter

# tsdb.local

# http://opentsdb.net/docs/build/html/user_guide/cli/index.html

GCARGS="-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps\

-XX:+PrintTenuringDistribution -Xloggc: /tmp/tsd-gc- ` date +%s`.log"

if test -t 0; then # if stdin is a tty, don't turn on GC logging.

GCARGS=

fi

# The Sun JDK caches all name resolution results forever, which is stupid.

# This forces you to restart your application if any of the backends change

# IP. Instead tell it to cache names for only 10 minutes at most.

FIX_DNS= '-Dsun.net.inetaddr.ttl=600'

JVMARGS= "$JVMARGS $GCARGS $FIX_DNS"

Common JVM parameters

After analyzing the gc log, it is often necessary to adjust the jvm memory related parameters. The commonly used parameters are as follows

-Xms: The initial heap size, the default is 1/64 of the physical memory (<1GB); the default (MinHeapFreeRatio parameter can be adjusted) when the free heap memory is less than 40%, the JVM will increase the heap until the maximum limit of -Xmx
-Xmx: Maximum heap size, the default (MaxHeapFreeRatio parameter can be adjusted) when the free heap memory is greater than 70%, the JVM will reduce the heap until the minimum limit of -Xms
-Xmn: The memory space size of the new generation, note: the size here is (eden+ 2 survivor space). It is different from the New gen shown in jmap -heap. The entire heap size = young generation size + old generation size + permanent generation size. In the case of keeping the heap size unchanged, after increasing the new generation, the size of the old generation will be reduced. This value has a great impact on system performance, and Sun officially recommends configuring it as 3/8 of the entire heap.
-XX:SurvivorRatio: The ratio of the capacity of the Eden area to the Survivor area in the new generation. The default value is 8. The ratio of two Survivor areas to one Eden area is 2:8, and one Survivor area accounts for 1/10 of the entire young generation.
-Xss: stack size per thread. After JDK5.0, the stack size of each thread is 1M, and the stack size of each thread is 256K before. It should be adjusted appropriately according to the amount of memory required by the threads of the application. In the same physical memory, reducing this value can generate more threads. However, the operating system still has a limit on the number of threads in a process, which cannot be generated indefinitely, and the experience value is around 3000~5000. Generally, for small applications, if the stack is not very deep, 128k should be enough, and 256k is recommended for large applications. This option has a large performance impact and requires rigorous testing. Similar to the explanation of the threadstacksize option, the official documentation does not seem to explain it. There is such a sentence in the forum: "-Xss is translated in a VM flag named ThreadStackSize" Generally, this value can be set.
-XX:PermSize: Set the initial value of the permanent generation (perm gen). The default is 1/64 of physical memory.
-XX:MaxPermSize: Set the persistent generation maximum value. 1/4 of physical memory.

Example

The following parameters are analyzed below:

JAVA_OPTS="-server -Xms2000m -Xmx2000m -Xmn800m -XX:PermSize=64m -XX:MaxPermSize=256m -XX:SurvivorRatio=4
-verbose:gc -Xloggc:$CATALINA_HOME/logs/gc.log
-Djava.awt.headless=true
-XX:+PrintGCTimeStamps -XX:+PrintGCDetails
-Dsun.rmi.dgc.server.gcInterval=600000 -Dsun.rmi.dgc.client.gcInterval=600000
-XX:+UseConcMarkSweepGC -XX:MaxTenuringThreshold=15"

-Xms2000m -Xmx2000m -Xmn800m -XX:PermSize=64m -XX:MaxPermSize=256m

Xms is the initial heap size of the JVM when the jvm is started, Xmx is the maximum heap size of the jvm, xmn is the size of the new generation, permsize is the initial size of the permanent generation, and MaxPermSize is the maximum space of the permanent generation.

-XX:SurvivorRatio=4

SurvivorRatio is the ratio of the size of the Eden area in the Cenozoic space to the Survivor area in the rescue space. The default is 32, which means that the Eden area is 32 times the size of the Survivor area. It should be noted that Survivo has two areas, so Survivor actually occupies the entire area. 1/34 of young generation. Decreasing this parameter will increase the survivor area, let the objects stay in the survivor area as long as possible, and reduce the objects entering the old generation. The idea of removing the rescue space is to let most of the data that cannot be recovered immediately enter the old generation as soon as possible, speed up the recycling frequency of the old generation, and reduce the possibility of the old generation skyrocketing. This is done by setting -XX:SurvivorRatio to a relatively large value. value (such as 65536) to do so.

-verbose:gc -Xloggc:$CATALINA_HOME/logs/gc.log

Write the information of each garbage collection of the virtual machine to the log file, the file name is specified by file, the file format is a flat file, and the content is the same as the output content of -verbose:gc.

-Djava.awt.headless=true

Headless mode is a configuration mode of the system. In this mode, the system lacks a display device, keyboard or mouse.

-XX:+PrintGCTimeStamps -XX:+PrintGCDetails

Set the format of the gc log

-Dsun.rmi.dgc.server.gcInterval=600000 -Dsun.rmi.dgc.client.gcInterval=600000

Specifies the time interval for gc when rmi is called

-XX:+UseConcMarkSweepGC -XX:MaxTenuringThreshold=15

Adopt concurrent gc mode, enter the old generation after 15 minor gcs

Xms refers to the memory size occupied when the program starts. Generally speaking, if it is larger, the program will start faster, but it may also cause the machine to temporarily slow down.

Xmx refers to the maximum memory size that can be occupied during the running of the program. If the program requires more memory to run than this set value, an OutOfMemory exception will be thrown.

Xss refers to setting the stack size of each thread. This depends on your program to see how much memory a thread needs to take up, how many threads may be running at the same time, etc.

The settings of the above three parameters are all in Byte by default. You can also add [k/K] or [m/M] after the number to indicate KB or MB. Moreover, it is not possible to exceed the memory size of the machine itself, otherwise it will wait for the machine to slow down instead of the program to slow down.

-Xmsn
Specify the initial size, in bytes, of the memory allocation pool. This value must be a multiple of 1024 greater than 1MB. Append the letter k or K to indicate kilobytes, or m or M to indicate megabytes. The default value is chosen at runtime based on system configuration. For more information, see HotSpot Ergonomics
Examples:
-Xms6291456
-Xms6144k
-Xms6m

-Xmxn
Specify the maximum size, in bytes, of the memory allocation pool. This value must a multiple of 1024 greater than 2MB. Append the letter k or K to indicate kilobytes, or m or M to indicate megabytes. The default value is chosen at runtime based on system configuration. For more information, see HotSpot Ergonomics
Examples:
-Xmx83886080
-Xmx81920k
-Xmx80m

-Xssn
Set thread stack size.

Some FAQs

In order to avoid the full gc caused by the full Perm area, it is recommended to enable the CMS to recycle the Perm area option:
```
+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled
```
The default CMS is to start CMS collection when the tenured generation is full of 68%. If your old generation is not growing so fast and you want to reduce the number of CMS, you can increase this value appropriately:
```
-XX:CMSInitiatingOccupancyFraction=80
```
When encountering two kinds of failures that cause full gc: Prommotion failed and Concurrent mode failed:

The log output of Prommotion failed is probably like this:

[ParNew (promotion failed): 320138K->320138K(353920K), 0.2365970 secs]42576.951: [CMS: 1139969K->1120688K( 166784K), 9.2214860 secs] 1458785K->1120688K(2520704K), 9.4584090 secs]

This problem arises because there is not enough rescue space, so objects are transferred to the old generation, and the old generation does not have enough space to accommodate these objects, resulting in the generation of a full gc. The solution to this problem has two diametrically opposed tendencies: increasing the salvage space, increasing the old generation, or removing the salvage space.

The log of Concurrent mode failed is probably like this:

(concurrent mode failure): 1228795K->1228598K(1228800K), 7.6748280 secs] 1911483K->1681165K(1911488K), [CMS Perm : 225407K->225394K(262144K)], 7.6751800 secs]

The cause of the problem is that the CMS recycles the old generation too slowly, causing the old generation to be filled before the CMS is completed, causing full gc. To avoid this phenomenon, reduce the value of the -XX:CMSInitiatingOccupancyFraction parameter so that the The CMS fires earlier and more frequently, reducing the chance of the old generation being full.

Gc log analysis tool

GCHisto

http://java.net/projects/gchisto

Just click gchisto.jar to run it, click add to load gc.log

The total GC times, youngGC times, FullGC times, percentage of times, GC consumption time, percentage, average consumption time, minimum and maximum consumption time, etc.

Graphical representation of statistics

YoungGC, FullGC distribution diagram of different consumption times, check to display the separate distribution of youngGC or fullGC

The detailed gc situation of the whole time process can be analyzed for the whole process

GCLogViewer

http://code.google.com/p/gclogviewer/

Click run.bat to run

The trend graph of the gc situation in the whole process also shows the gc type, throughput, average gc frequency, memory change trend, etc.

Tools can also compare different gc logs:

HPjmeter

Get the address http://www.hp.com/go/java
Reference documentation http://www.javaperformancetuning.com/tools/hpjtune/index.shtml

The tool is very powerful, but it can only open the GC log generated by the following parameters, -verbose:gc -Xloggc:gc.log, and the gc.log generated by adding other parameters cannot be opened.

GCViewer

http://www.tagtraum.com/gcviewer.html

This tool is used a lot, but it can only run in versions below JDK1.5, and there is no corresponding after 1.6.

garbagecat

http://code.google.com/a/eclipselabs.org/p/garbagecat/wiki/Documentation

Other monitoring methods

Jvisualvm

Jvisualvm dynamically analyzes the jvm memory situation and gc situation, plug-in: visualGC

jvisualvm can also heapdump out the corresponding hprof file (default storage path: under the monitored server/tmp), and use related tools, such as HPjmeter, to analyze it

grep Full gc.log to roughly observe the frequency of FullGC

jstat –gcutil [pid] [intervel] [count]

jmap

jmap -histo pid can observe the number and space of objects

jmap -heap pid can observe the jvm configuration parameters and the usage of each area of heap memory

jprofiler, jmap dump out with MAT analysis

If the dump file to be analyzed is very large, it requires a lot of memory and it is easy to crash.

So at startup, we should add some parameters: Java –Xms512M –Xmx1024M –Xss8M

References:

Exploring the Java Virtual Machine - Memory Management and Garbage Collectionhttp: //sunbean.blog.51cto.com/972509/768034