Interview question: How to perform jvm tuning? Do you have any experience?

This article is reproduced in: Interviewer: How to perform JVM tuning (with real cases)

foreword

Interviewer: Have you ever done JVM tuning at work? Tell me what JVM tuning you have done?

I have a project with a QPS of less than 10. The last time I was asked about cache penetration and cache avalanche, this time I was asked about JVM tuning. It was really difficult for me.

But don't panic, I am enthusiastic and found a few full-score answers for you, and you can choose the appropriate one to use.

Answer 1: Listen up, here will be my first JVM tuning.

Answer 2: I usually only tune during interviews.

Answer 3: I usually add machines and memory directly.

Answer 4: I used ZGC directly, and adjusted it with a snake skin.

text

1. Does the JVM need to be tuned?

After so many years of development and verification, the JVM is very robust overall. I personally think that in 99% of cases, JVM tuning is basically not needed.

Generally speaking, most of our JVM parameter configurations still follow the official JVM recommendations, for example:

• -XX:NewRatio=2, young generation: old generation=1:2

• -XX:SurvivorRatio=8，eden:survivor=8:1

• The heap memory is set to about 3/4 of the physical memory

• etc.

The default (recommended) values ​​of JVM parameters are relatively reasonable values ​​obtained through repeated testing by the JVM team and full verification by predecessors. Therefore, they are generally more reliable and common, and generally do not cause major problems.

Of course, more importantly, the QPS of most applications is less than 10, and the amount of data is less than tens of thousands. In this low-pressure environment, it is quite difficult to let the JVM go wrong.

What most students encounter more often is that their own code bugs lead to OOM, high CPU load, frequent GC, etc. These scenarios are basically code repairs, and usually do not need to move the JVM.

Of course, as the saying goes, there is no absolute in everything, and there are still a small number of scenarios that may require JVM tuning. The specific scenarios are described below.

It is worth mentioning that the JVM tuning we are talking about here is more about optimizing and adjusting JVM parameters according to our own business scenarios to make it more suitable for our business, rather than referring to changes to the JVM source code.

2. There is no need for JVM tuning. Using a garbage collector with better performance can solve the problem?

This is a statement I saw on the Internet, because there are many people who agree with it, and I guess many students will also have this idea, so I will talk about my views here.

1) Actual combat angle

Regardless of the factors to cope with the interview, upgrading the garbage collector will indeed be one of the most effective ways, for example: CMS upgrade to G1, or even ZGC.

This is easy to understand. The higher version of the garbage collector is equivalent to the optimization of the JVM by JVM developers. After all, they specialize in this, so generally speaking, the performance of upgrading to a higher version will improve a lot.

G1 has begun to be gradually applied. Many teams around have used G1 in JDK8. As far as I know, there are still many problems. Many students are constantly adjusting parameters, but in JDK11 How it can be optimized remains to be verified.

ZGC is still relatively seldom used at present. It looks good only from the published data. The maximum pause time does not exceed 10ms, or even 1ms. Everyone has high expectations. But judging from some information I have collected so far, ZGC is not a silver bullet. The known obvious problems are:

• Compared with G1, the throughput will decrease, and the official said that the maximum will not exceed 15%

• If ZGC encounters a very high object allocation rate (allocation rate), it will not be able to keep up. At present, the only effective "tuning" method is to increase the size of the entire GC heap to allow ZGC to have more breathing space - R large Original words after communicating with ZGC team leader

Moreover, with the subsequent application of ZGC, more problems will continue to appear in the future.

Overall, I personally feel that JVM tuning is still necessary in some scenarios. After all, there is a saying: there is no best, only the most suitable.

2) Interview angle

If you answer to directly upgrade the garbage collector, the interviewer may also agree, but this topic may end like this. The interviewer probably did not hear the answer he wanted. You will definitely not get extra points for this question, or even Points may be deducted.

So, during the interview, you can answer to upgrade the garbage collector, but you can't just answer to upgrade the garbage collector.

3. When is the JVM optimized?

Avoid premature optimization. Donald Ervin Knuth, the author of "The Art of Computer Programming", once said a classic sentence:

The real problem is that programmers have spent far too much time worrying about efficiency in the wrong places and at the wrong times; premature optimization is the root of all evil (or at least most of it) in programming.

The real problem is that programmers spend too much time worrying about efficiency in the wrong place and at the wrong time; premature optimization is the root of all (or at least most) evil in programming.

Avoiding being too early does not mean that it should be completely ignored. The more correct approach should be to provide monitoring and alarms for some important JVM indicators of core services. When indicators fluctuate or are abnormal, timely intervention can be performed.

Interviewer: What are the core indicators of the JVM? What should be the reasonable range?

There is no uniform answer to this question, because each service has different requirements for performance indicators such as AVG/TP999/TP9999, so the reasonable range is also different.

In order to prevent the interviewer from asking, for ordinary Java back-end applications, I will give a relatively reasonable range value here. The following indicators are for a single server:

• jvm.gc.time: The GC time per minute is within 1s, preferably within 500ms

• jvm.gc.meantime: Each YGC takes less than 100ms, preferably less than 50ms

• jvm.fullgc.count: FGC at most once every few hours, preferably less than once a day

• jvm.fullgc.time: Each FGC takes less than 1s, preferably less than 500ms

Generally speaking, as long as these indicators are normal, there will be no problems in other places. If there is a problem in other places, these indicators will generally be affected.

4. JVM optimization steps?

4.1. Analyze and locate the bottleneck of the current system

For the core indicators of the JVM, our focus and commonly used tools are as follows:

1) CPU indicators

• Find out which processes are using the most CPU

• View the most CPU-intensive threads

• View thread stack snapshot information

• Analyze code execution hotspots

• See which code takes the longest CPU execution time

• View the percentage of CPU time taken up by each method

Common commands:

  
  
   
   
// Display the resource usage of each process in the system
top
/ / Check the thread usage in a process
top -Hp pid
// View the thread stack information of the current Java process
jstack pid
  
  
   
   

Common tools: JProfiler, JVM Profiler, Arthas, etc.

2) JVM memory indicators

• Check whether the current JVM heap memory parameter configuration is reasonable

• View statistics for objects in the heap

• View heap storage snapshots and analyze memory usage

• Check whether the memory growth in each area of ​​the heap is normal

• See which area caused the GC

• Check whether the memory can be recovered normally after GC

Common commands:

// 显示系统各个进程的资源使用情况
top
// 查看某个进程中的线程占用情况
top -Hp pid
// 查看当前 Java 进程的线程堆栈信息
jstack pid

Common tools: Eclipse MAT, JConsole, etc.

3) JVM GC indicators

• Check whether the GC time per minute is normal

• Check whether the number of YGC per minute is normal

• Check whether the FGC times are normal

• Check whether the single FGC time is normal

• View the detailed time-consuming of each stage of a single GC, and find the stage with serious time-consuming

• Check whether the dynamic promotion age of the object is normal

The GC indicators of the JVM are generally checked from the GC logs. The default GC logs may be relatively small. We can add the following parameters to enrich our GC log output and facilitate us to locate problems.

Common JVM parameters for GC logs:

 
 
  
  
// Print the details of GC
-XX:+PrintGCDetails
// Print the timestamp of GC
-XX:+PrintGCDateStamps
// Print heap information before and after GC
-XX:+PrintHeapAtGC
// Print the distribution information of objects of each age group in the Survivor area
-XX:+PrintTenuringDistribution
// Output all parameter values ​​when the JVM starts, which is convenient to check whether the parameters are overwritten
-XX:+PrintFlagsFinal
// Print the stop time of the application during GC
-XX:+PrintGCApplicationStoppedTime
// Print the time referenced objects were processed during GC (only enabled when PrintGCDetails)
-XX:+PrintReferenceGC
 
 
  
  

The above are the common methods we use to locate the bottleneck of the system. Most of the problems can be located through the above methods, and then combined with the code to find the root cause of the problem.

4.2. Determine the optimization goal

After locating the system bottleneck, what is the goal of optimization before optimization, for example:

• Reduce FGC frequency from 1 time per hour to 1 time per day

• Reduce the GC time per minute from 3s to 500ms

• Reduce the time-consuming of each FGC from 5s to less than 1s

• ...

4.3. Formulate an optimization plan

Develop corresponding optimization schemes for the identified system bottlenecks. The common ones are:

• Code bugs: Upgrade to fix bugs. Typical ones are: infinite loop, using unbounded queue.

• Unreasonable JVM parameter configuration: optimize JVM parameter configuration. Typical examples are: the young generation memory configuration is too small, the heap memory configuration is too small, and the metaspace configuration is too small.

4.4. Compare the indicators before and after optimization, and count the optimization effect

4.5. Continuous observation and tracking of optimization effects

4.6, if necessary, repeat the above steps

5. Tuning case: Metaspace causes frequent FGC problems

The following cases come from the Internet or my real experience, all of which can be justified. After understanding and mastering, students can use them to match the interviewer.

Service environment: ParNew + CMS + JDK8

Symptom : FGC occurs frequently in the service

Reason analysis :

1) First check the GC log and find that the reason for the occurrence of FGC is that the metaspace space is not enough

Corresponding GC log:

Full GC (Metadata GC Threshold)
 
 
  
  

2) Check the logs further and find that there is memory fragmentation in the metaspace

Corresponding GC log:

Metaspace       used 35337K, capacity 56242K, committed 56320K, reserved 1099776K
 
 
  
  

Here is a brief explanation of the meaning of these parameters

• used : the amount of space used

• capacity: the size of the currently allocated and unreleased space

• committed: the size of the currently allocated space

• reserved: reserved space size

Here used is easier to understand. Reserved is not important in this example and can be ignored. The main reason is that capacity and committed are easy to confuse.

It is easier to understand by combining the following figure. The allocation of metaspace is in chunks. When a ClassLoader is garbage collected, all the space (chunk) belonging to it is released. At this time, the chunk is called Free Chunk, and the committed chunk is The sum of capacity chunk and free chunk.

The reason why memory is fragmented is based on the data of used and capacity. It is said that the allocation of metaspace is in units of chunks. Even if a ClassLoader only loads one class, it will monopolize the entire chunk, so when used When the difference between capacity and capacity is large, it means that there is memory fragmentation at this time.

The GC log demo is as follows:

 
 
  
  
{Heap before GC invocations= 0 (full 0):
par new generation total 314560K, used 141123K [ 0x00000000c0000000, 0x00000000d5550000, 0x00000000d5550000)
eden space 279616K, 50% used [ 0x00000000c0000000, 0x00000000c89d0d00, 0x00000000d1110000)
from space 34944K, 0% used [ 0x00000000d1110000, 0x00000000d1110000, 0x00000000d3330000)
to space 34944K, 0% used [ 0x00000000d3330000, 0x00000000d3330000, 0x00000000d5550000)
concurrent mark-sweep generation total 699072K, used 0K [ 0x00000000d5550000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 35337K, capacity 56242K, committed 56320K, reserved 1099776K
class space used 4734K, capacity 8172K, committed 8172K, reserved 1048576K
1.448: [Full GC (Metadata GC Threshold) 1.448: [CMS: 0K-> 10221K( 699072K), 0.0487207 secs] 141123K-> 10221K( 1013632K), [Metaspace: 35337K-> 35337K( 1099776K)], 0.0488547 secs] [Times: user= 0.09 sys= 0.00, real= 0.05 secs]
Heap after GC invocations= 1 (full 1):
par new generation total 314560K, used 0K [ 0x00000000c0000000, 0x00000000d5550000, 0x00000000d5550000)
eden space 279616K, 0% used [ 0x00000000c0000000, 0x00000000c0000000, 0x00000000d1110000)
from space 34944K, 0% used [ 0x00000000d1110000, 0x00000000d1110000, 0x00000000d3330000)
to space 34944K, 0% used [ 0x00000000d3330000, 0x00000000d3330000, 0x00000000d5550000)
concurrent mark-sweep generation total 699072K, used 10221K [ 0x00000000d5550000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 35337K, capacity 56242K, committed 56320K, reserved 1099776K
class space used 4734K, capacity 8172K, committed 8172K, reserved 1048576K
}
{Heap before GC invocations= 1 (full 1):
par new generation total 314560K, used 0K [ 0x00000000c0000000, 0x00000000d5550000, 0x00000000d5550000)
eden space 279616K, 0% used [ 0x00000000c0000000, 0x00000000c0000000, 0x00000000d1110000)
from space 34944K, 0% used [ 0x00000000d1110000, 0x00000000d1110000, 0x00000000d3330000)
to space 34944K, 0% used [ 0x00000000d3330000, 0x00000000d3330000, 0x00000000d5550000)
concurrent mark-sweep generation total 699072K, used 10221K [ 0x00000000d5550000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 35337K, capacity 56242K, committed 56320K, reserved 1099776K
class space used 4734K, capacity 8172K, committed 8172K, reserved 1048576K
1.497: [Full GC (Last ditch collection) 1.497: [CMS: 10221K-> 3565K( 699072K), 0.0139783 secs] 10221K-> 3565K( 1013632K), [Metaspace: 35337K-> 35337K( 1099776K)], 0.0193983 secs] [Times: user= 0.03 sys= 0.00, real= 0.02 secs]
Heap after GC invocations= 2 (full 2):
par new generation total 314560K, used 0K [ 0x00000000c0000000, 0x00000000d5550000, 0x00000000d5550000)
eden space 279616K, 0% used [ 0x00000000c0000000, 0x00000000c0000000, 0x00000000d1110000)
from space 34944K, 0% used [ 0x00000000d1110000, 0x00000000d1110000, 0x00000000d3330000)
to space 34944K, 0% used [ 0x00000000d3330000, 0x00000000d3330000, 0x00000000d5550000)
concurrent mark-sweep generation total 699072K, used 3565K [ 0x00000000d5550000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 17065K, capacity 22618K, committed 35840K, reserved 1079296K
class space used 1624K, capacity 2552K, committed 8172K, reserved 1048576K
}

The metaspace is mainly suitable for storing class-related information, and the existence of memory fragmentation indicates that more class loaders are likely to be created, and the usage rate is low.

Therefore, when memory fragmentation occurs in the metaspace, we will pay attention to whether a large number of class loaders are created.

3) Found a large number of DelegatingClassLoaders through dump heap storage files

Through further analysis, it was found that a large number of DelegatingClassLoaders were created due to reflection. Its core principles are as follows:

On the JVM, reflective invocation of methods was initially implemented through JNI calls, and when the JVM notices that a method is frequently accessed through reflection, it will generate bytecode to perform the same operation, known as the inflation mechanism. If the bytecode method is used, a DelegatingClassLoader will be generated for the method. If there are a large number of methods that are frequently called reflectively, a large number of DelegatingClassLoaders will be created.

How often does the reflection call need to be converted from JNI to bytecode?

The default is 15 times, which can be controlled by the parameter -Dsun.reflect.inflationThreshold. When the number is less than this number, the method will be called by JNI. If the number of calls exceeds this number, the method call will be generated by bytecode.

Analysis conclusion : Reflective calls lead to the creation of a large number of DelegatingClassLoaders, occupying a large amount of metaspace memory, and memory fragmentation, resulting in low metaspace utilization, which quickly reaches the threshold and triggers FGC.

Optimization Strategy:

1) Properly increase the size of the metaspace.

2) Optimize unreasonable reflection calls. For example, the most common property copy tool class BeanUtils.copyProperties can be replaced by mapstruct.

Summarize

When asked by the interviewer about JVM tuning, you can answer according to the context of this article:

• First of all, if you use a reasonable JVM parameter configuration, you should not need to tune in most cases - corresponding to the first question of this article

• Secondly, it shows that there may still be a small number of scenarios that need to be tuned. We can configure monitoring and alarms for some JVM core indicators, and human intervention analysis and evaluation when fluctuations occur—corresponding to Question 3 of this article

• Finally, give a practical tuning example to illustrate - corresponding to question 5 of this article

If the interviewer asks how to analyze and troubleshoot, you can use the common commands and tools in question 4 of this article to match it.

After this process, I believe most interviewers will have a good impression of you.

​

at last

I am Jionghui, a programmer who insists on sharing original technology and dry goods . If you think this article is helpful to you, remember to like and follow. See you next time.

recommended reading

Java basic high-frequency interview questions (2021 latest version)

Java collection framework high-frequency interview questions (2021 latest version)

Spring, which must be asked in the interview, do you understand?

MySQL, which must be asked in the interview, do you understand?