In the past few days, I tested the pre-production environment and found that the TPS was unstable. Because it is a refactored system, it is said that the original system has no problems when it has high concurrency. As a result, the refactored system is all kinds of unstable when it is pressured by dozens of concurrency. Although the test colleague didn't say anything, he felt like he was slapped in the face. . .

Therefore, the first thing that comes to mind for various investigations is the JVM parameters, so I optimized it, hoping to produce a good result. Although it was later proved that the reason for the instability was that the stress test server where LoadRunner was installed was unstable, and it was not related to my system, but it was also a record. One is to make a backup, and the other is to be a reference for others.

Words written in the front

Because the default values of parameters provided by Hotspot JDK are constantly changing between minor versions, the parameters will also affect each other. Moreover, different server configurations may affect the final result. So don't be superstitious about the parameter configuration in a certain article on the Internet (including this one). All configurations need to be tested by yourself before they can be used. In view of the constant change of JVM parameter default values, you can use to -XX:+PrintFlagsFinalprint the current environment JVM parameter default values, such as:, java -XX:PrintFlagsFinal -versionor you can use to java [生产环境参数] -XX:+PrintFlagsFinal –version | grep [待查证的参数]view specific parameter data.

Here are the parameters of an 8G server, the JDK version information is as follows:

java version "1.8.0_73"
Java(TM) SE Runtime Environment (build 1.8.0_73-b02)
Java HotSpot(TM) 64-Bit Server VM (build 25.73-b02, mixed mode)

Heap settings

The heap memory setting should be regarded as the basic quality of a Java programmer, and at least the three parameters of Xms, Xmx, and Xmn have to be modified. But how much memory might a JVM with a 2G heap size occupy in total?

Heap memory + number of threads * thread stack + permanent generation + binary code + off-heap memory

2G + 1000 * 1M + 256M + 48/240M + (～2G) = 5.5G (3.5G)-Heap
memory: Store Java objects, the default is 1/64 of physical memory
-Thread stack: Store local variables (atomic type, Reference) and others, the default is 1M
-permanent generation: storage class definition and constant pool, pay attention to the difference between JDK7/8
-binary code: JDK7 and 8, the default value when opening multi-layer compilation is different, from 48 to 240M
-heap External memory: Used by Netty, off-heap cache, etc. The default maximum is about the size of heap memory

In other words, if the heap memory is set to 2G, the JVM with 1000 threads may need to occupy 5.5G. Considering various conditions such as system occupation, IO occupation, etc., an 8G server will also start a service. Of course, if the number of threads is small, the concurrency is not high, and the pressure is not high, you can still start more than one, and you can also reduce the heap memory.

-Xms2g and -Xmx2g: heap memory size, the first is the minimum heap memory, the second is the maximum heap memory, the more suitable value is 2-4g, and the larger the GC time
-Xmn1g or (-XX:NewSize=1g and -XX:MaxNewSize=1g) or -XX:NewRatio=1: Set the Cenozoic size, the JDK defaults that the Cenozoic occupies 1/3 of the heap memory size, that is -XX:NewRatio=2. Here is the 1g set, that is -XX:NewRatio=1. You can set it according to your needs.
-XX:MetaspaceSize=128m and -XX:MaxMetaspaceSize=512m, the immortal generation of JDK8 can almost use all the memory of the machine. In order to protect the server from being unable to connect due to excessive memory usage, it is necessary to set an initial value of 128M and a maximum of 512M Value protection.
-XX:SurvivorRatio: The size of each survival area in the Cenozoic, the default is 8, which is 1/10 of the Cenozoic, 1/(SurvivorRatio+2). Some people like to set small points for the Cenozoic, but avoid too much The small size makes it impossible to put temporary objects in the survival area and want to be promoted to the old generation. It is still from the GC Log to see the actual situation.
-Xss256k: Outside the heap, threads occupy stack memory, and each thread is 1M by default. Store the stack of method calls, local variables, local variables after scalar replacement, etc. Some people like to set small points to save memory and open more threads. But anyway, if the memory is enough, it is not necessary to set a small size. Some people like to set a larger size, especially when there are recursive calls such as JSON parsing.
-XX:MaxDirectMemorySize: The size of the off-heap memory/direct memory, the default is the size of the heap memory minus a Survivor area.
-XX:ReservedCodeCacheSize: The storage area of the binary code after JIT compilation. When it is full, it will not be compiled. By default, multi-layer compilation is 240M. You can check the size of CodeCache in JMX.

GC settings

At present, the more mainstream GCs are CMS and G1, and some great gods suggest to use 8G as the boundary. (It is said that JDK 9 defaults to G1). Because the memory set by the application is relatively small, the CMS collector is selected. The following parameters are also for the CMS collector. If necessary, add the parameters of the G1 collector after waiting.

CMS settings

-XX:+UseConcMarkSweepGC: Enable the CMS garbage collector
-XX:CMSInitiatingOccupancyFraction=80 and -XX:+UseCMSInitiatingOccupancyOnly: The two parameters need to be used together, otherwise the 75 of the first parameter is just a reference value, and the JVM will recalculate the GC time.
-XX:MaxTenuringThreshold=15: How many times the object has survived the Young GC in the Survivor area and then promoted to the old generation. The default is 15. Young GC is the biggest source of application pauses, and the number of surviving objects after GC in the new generation directly affects the pause time, so if you know the execution frequency of Young GC and the longest life cycle of most temporary objects in the application, you can use it Make it shorter, so that long-term objects of the new generation who are not temporary objects can be promoted to the old generation quickly.
-XX:-DisableExplicitGC: Allow to use System.gc() to actively call GC. It needs to be explained here that some JVM optimization suggestions are to set -XX:-DisableExplicitGC and turn off the manual call to System.gc(). This is because System.gc() triggers Full GC, and frequent Full GC will seriously affect performance. However, many NIO frameworks, such as Netty, use off-heap memory. If there is no Full GC, off-heap memory cannot be reclaimed. If you don't actively call System.gc(), you need to wait until the JVM itself triggers Full GC. At this time, it may cause a long pause (STW), and the machine load will also increase. Therefore, System.gc() cannot be completely prohibited, and the time of Full GC must be shortened. Then use the -XX:+ExplicitGCInvokesConcurrentor -XX:+ExplicitGCInvokesConcurrentAndUnloadsClassesoption and use the CMS collector to trigger Full GC. These two options need to be -XX:+UseConcMarkSweepGCused together.
-XX:+ExplicitGCInvokesConcurrent: Use System.gc() to trigger CMS GC instead of Full GC. It is not enabled by default. This option can only be enabled when the -XX:+UseConcMarkSweepGC option is used.
-XX:+ExplicitGCInvokesConcurrentAndUnloadsClasses: When using System.gc(), the permanent generation is also included in the scope of CMS. This option can only be turned on when the -XX:+UseConcMarkSweepGC option is used.
-XX:+ParallelRefProcEnabled: The default is false. Reference objects such as WeakReference are processed in parallel. Unless a log with a long Reference processing time appears in the GC log, the effect will not be obvious.
-XX:+ScavengeBeforeFullGC: Perform Young GC once before Full GC is executed.
-XX:+UseGCOverheadLimit: Limit the running time of GC. If GC takes too long, throw OOM.
-XX:+UseParallelGC: Set the parallel garbage collector
-XX:+UseParallelOldGC: Set the old generation to use the parallel garbage collector
-XX:-UseSerialGC: Turn off the serial garbage collector
-XX:+CMSParallelInitialMarkEnabled and -XX:+CMSParallelRemarkEnabled: reduce the mark pause
-XX:+CMSScavengeBeforeRemark: The default is off. Before CMS remark, perform a minor GC to clear the new generation, so that the number of new generation objects referenced from the old generation objects will be less, and stop CMS all over the world The remark phase is shorter. If you see that the remark phase in the GC log takes too long, you can turn on this item to see if it has any effect. Otherwise, don't turn it on. There will be more YGCs in vain.
-XX:CMSWaitDuration=10000: Set the maximum interval of garbage collection, the default is 2000.
-XX:+CMSClassUnloadingEnabled: Clean up the expired Class in the permanent generation in CMS without waiting for Full GC. JDK7 is closed by default and JDK8 is opened. It depends on your own situation, such as whether you run dynamic language scripts such as Groovy to generate a large number of temporary classes. It will increase the pause time of CMS remark, so if new classes are not loaded frequently, this parameter is still not open.

GC log

The GC process can provide an optimization basis through the GC log.

-XX:+PrintGCDetails: Enable gc log printing function
-Xloggc:/path/to/gc.log: Specify the gc log location
-XX:+PrintHeapAtGC: Print detailed stack information before and after GC
-XX:+PrintGCDateStamps: print a readable date instead of a timestamp
-XX:+PrintGCApplicationStoppedTime: Print all JVM pause times, if you really find some unknown pauses, add temporarily -XX:+PrintSafepointStatistics -XX: PrintSafepointStatisticsCount=1to find the reason.
-XX:+PrintGCApplicationConcurrentTime: Print the normal running time of the JVM between two pauses, and the -XX:+PrintGCApplicationStoppedTimeeffect is better when used together.
-XX:+PrintTenuringDistribution: View the threshold of the new survival period after each minor GC
-XX:+UseGCLogFileRotation and -XX:NumberOfGCLogFiles=10 and -XX:GCLogFileSize=10M: GC log will be cleared after restarting, but if an application does not restart for a long time, the GC log will increase, so add these 3 parameters, The GC log is written to the file in a rolling manner, but if you restart, the name may be confused.
-XX:PrintFLSStatistics=1: Print statistics about memory fragmentation before and after each GC

Other parameter settings

-ea: Enable the assertion, there is nothing to say about this, you can choose to enable it, or you can choose not to enable it, there is no big difference. It is processed completely according to its own system.
-XX:+UseThreadPriorities: Enable thread priority, mainly because we can give lower priority to periodic tasks to avoid interfering with client work. In my current environment, it is enabled by default.
-XX:ThreadPriorityPolicy=42: Allow thread priority to be lowered
-XX:+HeapDumpOnOutOfMemoryError: A heap-dump is performed when a memory overflow occurs
-XX:HeapDumpPath=/path/to/java_pid.hprof: This parameter works in conjunction with the -XX:+HeapDumpOnOutOfMemoryErroroutput file when setting heap-dump.
-XX:ErrorFile=/path/to/hs_err_pid.log: Specify the location of the fatal error log. Generally, when a fatal error occurs in the JVM, a file similar to hs_err_pid.log will be output. The default is in the working directory (if you do not have permission, you will try to create it in /tmp), but it is better to specify the location yourself, which is convenient for collection and search. Avoid loss.
-XX:StringTableSize=1000003: Specify the string constant pool size, the default value is 60013. Those with a little common sense of Java should know that strings are constants and cannot be modified after they are created. The place where these constants are located is called the string constant pool. If there are many string operations in your system, and these string values are relatively fixed, you can appropriately increase the pool size if allowed.
-XX:+AlwaysPreTouch: All the memory defined by all parameters are stroked at startup. Using this parameter may make the startup slower, but it will be faster in the later memory usage process. It can ensure the continuous allocation of memory pages, and the new generation will not cause the GC to pause and lengthen due to the application of memory pages when the new generation is promoted. Usually only when the memory is greater than 32G will feel.
-XX:-UseBiasedLocking: Disable biased locks (in a highly concurrent environment where a large number of lock objects are created and highly concurrent (ie non-multithreaded and highly concurrent applications), disabling biased locks can bring certain performance optimizations)
-XX:AutoBoxCacheMax=20000: Increase the range of automatic boxing of digital objects. JDK defaults -128 to 127 int and long. Objects will be created immediately if the range is exceeded. Therefore, increasing the range can improve performance, but it also requires testing.
-XX:-OmitStackTraceInFastThrow: Do not ignore the stack of repeated exceptions. This is an optimization of the JDK. A large number of repeated JDK exceptions will no longer print their StackTrace. But if the system is a system that does not restart for a long time, and it runs N multiple exceptions in the same place, the result is ignored by the JDK. Isn't it because you can't see the specific StackTrace when you check the log? How to debug, so It's better to close it.
-XX:+PerfDisableSharedMem: Enable standard memory usage. JVM control is divided into standard or shared memory. The difference is that one is in JVM memory, the other is to generate /tmp/hsperfdata_{userid}/{pid} files, store statistics, and map to memory through mmap, and other processes can pass File access content. With this parameter, you can prohibit the JVM from writing statistical data in the file. The price is that the jps and jstat commands are no longer available, and the data can only be obtained through jmx. But in troubleshooting, small tools such as jps and jstat are very easy to use, and are much easier to use than heavy things like jmx, so you need to make your own choices. Here is an example of GC pause.
-Djava.net.preferIPv4Stack=true: This parameter is a parameter that belongs to network problems and can be set as required. In some machines with ipv6 enabled, the InetAddress.getLocalHost().getHostName()complete machine name can be obtained through , but in ipv4 machines, the machine name obtained by this method may be incomplete, and the complete machine name can be obtained through this parameter.

The example given by the great god

The example given by the Great God is posted below, you can refer to it, but it is recommended to use it after specific verification in your own environment. After all, the environment of the Great God is still different from your own environment.

Performance related

-XX:-UseBiasedLocking -XX:-UseCounterDecay -XX:AutoBoxCacheMax=20000
-XX:+PerfDisableSharedMem(可选) -XX:+AlwaysPreTouch -Djava.security.egd=file:/dev/./urandom

Memory size related (JDK7)

-Xms4096m -Xmx4096m -Xmn2048m -XX:MaxDirectMemorySize=4096m
-XX:PermSize=128m -XX:MaxPermSize=512m -XX:ReservedCodeCacheSize=240M

If you use jdk8, replace -XX:PermSize=128m -XX:MaxPermSize=512m with -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=512m. As mentioned earlier, these two sets of parameters are to ensure safety and it is recommended Still add.

CMS GC related

-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -XX:MaxTenuringThreshold=6
-XX:+ExplicitGCInvokesConcurrent -XX:+ParallelRefProcEnabled

GC log related

-Xloggc:/dev/shm/app-gc.log -XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCDateStamps -XX:+PrintGCDetails

Exception log related

-XX:-OmitStackTraceInFastThrow -XX:ErrorFile=${LOGDIR}/hs_err_%p.log
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${LOGDIR}/

JMX related

-Dcom.sun.management.jmxremote.port=${JMX_PORT} -Dcom.sun.management.jmxremote
-Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false