Metaspace Decryption of JVM Source Code Analysis

overview

Metaspace, as the name suggests, metadata space, is specially used to store metadata. It is a unique data structure in jdk8 to replace perm. This space has its own characteristics. Some time ago, the company had too many problems in this area. The main It is caused by the upgrade of the middleware. After seeing everyone discussing and discussing, it can be seen that many people are still ambiguous about metaspace and don’t know much about it. Therefore, I think it is necessary to write an article to introduce it and unravel its mystery. Veil, we will no longer feel at a loss when we encounter its related problems again.

Through this article, you will learn about

• Why there is metaspace
• The composition of metaspace
• VM parameters for metaspace
• What values ​​of metaspace should we pay attention to in jstat

Why there is metaspace

There are many folk legends about the origin of metaspace, but I only talk about my own understanding here, because I am not an oracle developer involved in this area, so I don't know much about its real origin.

We all know that before jdk8, there is a whole piece of perm memory to store klass and other information. It is also necessary to configure -XX:PermSize and -XX:MaxPermSize in our parameters to control the size of this memory. When jvm starts Sometimes a contiguous block of memory will be allocated according to these configurations, but as more and more dynamic class loading occurs, this memory becomes less controllable. How appropriate is the setting is a question for every developer to consider , if the setting is too small, memory overflow will easily occur during system operation, and if the setting is too large, it always feels wasteful, although such a large physical memory will not be allocated in essence. Based on such a possible reason, metaspace appeared. I hope that memory management will no longer be restricted, and don't pay much attention to the OOM problem of metadata, although so far, it has not solved this problem perfectly.

Perhaps some clues can also be seen from the JVM code, such as MaxMetaspaceSizethe default value is very large, and CompressedClassSpaceSizethe default value is 1G. From these parameters, we can guess that the author of metaspace does not want its related OOM problems.

The composition of metaspace

Metaspace actually consists of two parts

• Klass Metaspace
• NoKlass Metaspace

Klass Metaspace is used to store klass. Klass is the runtime data structure of the class file we are familiar with in jvm. However, it should be mentioned that the similar A.class we see actually exists in the heap, which is java.lang An object instance of .Class. This piece of memory is next to the Heap. Like our previous perm, the size of this piece of memory can be -XX:CompressedClassSpaceSizecontrolled by parameters. The parameter mentioned earlier is 1G by default, but this piece of memory can also be left out. If the compression pointer is not enabled, it will not There will be this piece of memory. In this case, klass will be stored in the NoKlass Metaspace. In addition, if we set -Xmx to be greater than 32G, there will be no such piece of memory, because such a large memory will turn off the compression pointer switch. There is only one piece of this memory at most.

NoKlass Metaspace is specially used to store other content related to klass, such as method, constantPool, etc. This memory is composed of multiple blocks of memory, so it can be considered as composed of discontinuous memory blocks. This piece of memory is necessary. Although it is called NoKlass Metaspace, it can actually store the content of klass. The corresponding scenario has been mentioned above.

Both Klass Metaspace and NoKlass Mestaspace are shared by all classloaders, so classloaders need to allocate memory, but each classloader has a SpaceManager to manage the memory blocks belonging to this classloader. If the Klass Metaspace is used up, it will be OOM, but generally not, the NoKlass Mestaspace is slowly combined by pieces of memory, and it will continue to lengthen the chain if the limit condition is not reached, so that it Can work continuously.

Several parameters of metaspace

If we want to change some behaviors of metaspace, we generally adjust some related parameters, because there are not many parameters in metaspace, so I will introduce all the parameters involved here, maybe there are many parameters that everyone has Misunderstood

• UseLargePagesInMetaspace
• InitialBootClassLoaderMetaspaceSize
• MetaspaceSize
• MaxMetaspaceSize
• CompressedClassSpaceSize
• MinMetaspaceExpansion
• MaxMetaspaceExpansion
• MinMetaspaceFreeRatio
• MaxMetaspaceFreeRatio

UseLargePagesInMetaspace

The default is false. This parameter indicates whether to use LargePage in the metaspace. Generally, we use a page size of 4KB. This parameter depends on the EnableLargePages parameter, but we generally do not enable this parameter.

InitialBootClassLoaderMetaspaceSize

The default is 4M for 64-bit, and 2200K for 32-bit. As mentioned earlier, metasapce is divided into two main blocks, Klass Metaspace and NoKlass Metaspace. NoKlass Metaspace is composed of blocks of memory. This parameter determines the NoKlass Metaspace. The size of a memory block, that is, 2*InitialBootClassLoaderMetaspaceSize, and the size of InitialBootClassLoaderMetaspaceSize is allocated for the first memory chunk of bootstrapClassLoader

MetaspaceSize

The default is about 20.8M (c2 mode is enabled under x86), mainly to control the initial threshold of metaspace GC, which is also the minimum threshold, but the threshold for triggering metaspace GC is constantly changing. The comparison mainly refers to the two committed Klass Metaspace and NoKlass Metaspace memory and .

MaxMetaspaceSize

The default is basically infinity, but I still recommend that you set this parameter, because it is very likely that the metaspace will be used endlessly (usually a memory leak) and be killed by the OS because there is no limit. This parameter will limit the committed memory size of metaspace (including Klass Metaspace and NoKlass Metaspace), and will ensure that the committed memory will not exceed this value. Once exceeded, GC will be triggered. Here, pay attention to the difference from MaxPermSize, MaxMetaspaceSize will not When jvm starts, allocate such a large memory, and MaxPermSize will allocate such a large memory.

CompressedClassSpaceSize

The default is 1G. This parameter is mainly to set the size of the Klass Metaspace. However, it does not necessarily work if this parameter is set. The premise is that the compression pointer can be enabled. If -Xmx exceeds 32G, the compression pointer cannot be enabled. If there is Klass Metaspace, then this piece of memory is connected to Heap.

MinMetaspaceExpansion

The two parameters MinMetaspaceExpansion and MaxMetaspaceExpansion may not be the same as what you know. Maybe many people think that these two parameters are the minimum size for expansion when the memory is not enough? actually not

These two parameters are not directly related to capacity expansion, that is, it is not to increase the committed memory, but to increase the threshold for triggering metaspace GC

These two parameters are mainly used for emergency use in special scenarios, such as gcLocker or should_concurrent_collectsome scenarios, because in these scenarios, a GC will be performed next, and it is believed that some metaspace memory may be released in the next GC, so first Temporarily expand the threshold of metaspace triggering GC, and some memory allocation failures are actually caused by the peak of this threshold, so you can temporarily bypass it by increasing the threshold

The default is 332.8K, increase the minimum requirement to trigger the metaspace GC threshold. If the memory we want to allocate is very small and has not reached MinMetaspaceExpansion, but we will increase the threshold for triggering metaspace GC this time to MinMetaspaceExpansion. The reason why it is larger than the size of memory to be allocated this time is mainly to prevent other threads from having similar requests Frequently trigger related operations, but if the memory to be allocated exceeds MaxMetaspaceExpansion, then MinMetaspaceExpansion will be an increment based on the size of the memory to be allocated

MaxMetaspaceExpansion

The default value is 5.2M, increasing the maximum requirement to trigger the metaspace GC threshold. If the memory we want to allocate exceeds MinMetaspaceExpansion but is lower than MaxMetaspaceExpansion, the increment is MaxMetaspaceExpansion. If it exceeds MaxMetaspaceExpansion, the increment is MinMetaspaceExpansion plus the memory size to be allocated

Note: Each allocation will only give the corresponding thread a chance to expand the threshold of triggering metaspace GC. If it is expanded but cannot be allocated, then it can only wait for GC

MinMetaspaceFreeRatio

MinMetaspaceFreeRatio and the following MaxMetaspaceFreeRatio mainly affect the threshold for triggering metaspaceGC

The default is 40, which means that after each GC, assuming that we allow the metaspace to continue to be committed memory, it will occupy MinMetaspaceFreeRatio% of the total amount of committed memory after commit, if the total committed amount is higher than the current threshold for triggering metaspaceGC Large, then we will try to expand, that is, increase the threshold for triggering metaspaceGC, but this increment will be done at least by MinMetaspaceExpansion, otherwise this threshold will not be increased

This parameter is mainly to avoid triggering the threshold of metaspaceGC and the amount of committed memory after gc is relatively close, so the threshold is expanded

Generally, after the gc is finished, if the amount of committed is still relatively large, in other words, when the threshold for triggering metaspaceGC is relatively close, this adjustment will be more obvious

Note: The amount of used after gc is not used here. The main reason is that the amount of committed may exceed the threshold for triggering metaspaceGC. Once this happens, it will be very dangerous and will continue to do gc. This should be after a certain version of jdk8 just fixed bug

MaxMetaspaceFreeRatio

The default is 70. This parameter is basically the opposite of the above parameter. It is to reduce the threshold to avoid triggering metaspaceGC from being too large. This parameter will be adjusted more obviously when the committed memory after gc is relatively small and far from the threshold triggering metaspaceGC

The metaspace field in jstat

Let's see if the GC is abnormal. In addition to analyzing the GC logs, we can also analyze the data displayed by tools such as jstat. There is an article in my official account that introduces the implementation of jstat. If you are interested, you can Go to my official account 你假笨to read this article of jstat.

We can see some indicators related to metaspace through jstat, namely , M, CCS, MC, MU, CCSC, CCSU, MCMN, MCMX,CCSMNCCSMX

They are defined as follows:

  column {
    header "^M^"    /* Metaspace - Percent Used */
    data (1-((sun.gc.metaspace.capacity - sun.gc.metaspace.used)/sun.gc.metaspace.capacity)) * 100
    align right
    width 6
    scale raw
    format "0.00"
  }
  column {
    header "^CCS^"  /* Compressed Class Space - Percent Used */
    data (1-((sun.gc.compressedclassspace.capacity - sun.gc.compressedclassspace.used)/sun.gc.compressedclassspace.capacity)) * 100
    align right
    width 6
    scale raw
    format "0.00"
  }
  
  column {
    header "^MC^"   /* Metaspace Capacity - Current */
    data sun.gc.metaspace.capacity
    align center
    width 6
    scale K
    format "0.0"
  }
  column {
    header "^MU^"   /* Metaspae Used */
    data sun.gc.metaspace.used
    align center
    width 6
    scale K
    format "0.0"
  }
   column {
    header "^CCSC^" /* Compressed Class Space Capacity - Current */
    data sun.gc.compressedclassspace.capacity
    width 8
    align right
    scale K
    format "0.0"
  }
  column {
    header "^CCSU^" /* Compressed Class Space Used */
    data sun.gc.compressedclassspace.used
    width 8
    align right
    scale K
    format "0.0"
  }
  column {
    header "^MCMN^" /* Metaspace Capacity - Minimum */
    data sun.gc.metaspace.minCapacity
    scale K
    align right
    width 8
    format "0.0"
  }
  column {
    header "^MCMX^" /* Metaspace Capacity - Maximum */
    data sun.gc.metaspace.maxCapacity
    scale K
    align right
    width 8
    format "0.0"
  }
  column {
    header "^CCSMN^"    /* Compressed Class Space Capacity - Minimum */
    data sun.gc.compressedclassspace.minCapacity
    scale K
    align right
    width 8
    format "0.0"
  }
  column {
    header "^CCSMX^"    /* Compressed Class Space Capacity - Maximum */
    data sun.gc.compressedclassspace.maxCapacity
    scale K
    align right
    width 8
    format "0.0"
  }

Here I will introduce the classification of these fields

MC & MU & CCSC & CCSU

• MC indicates the total committed memory size of Klass Metaspace and NoKlass Metaspace, and the unit is KB. Although we can see capacity from the above definition, it is actually not capacity but committed when calculating. This is something to pay attention to

• MU is understandable, it refers to the memory size used by both Klass Metaspace and NoKlass Metaspace

• CCSC represents the committed memory size of Klass Metaspace, and the unit is KB

• CCSU indicates the used memory size of Klass Metaspace

M & CCS

• M represents the total usage rate of Klass Metaspace and NoKlass Metaspace. In fact, it can be calculated according to the above four indicators, namely (CCSU+MU)/(CCSC+MC)

• CCS represents the utilization rate of NoKlass Metaspace, which is calculated by CCSU/CCSC

PS: So we sometimes see that the value of M reaches over 90%. In fact, this does not necessarily mean that the metaspace is used a lot, because the memory is committed slowly, so our denominator is gradually increasing. However, when When we commit to a certain amount, it will no longer grow

MCMN & MCMX & CCSMN & CCSMX

• The two values ​​of MCMN and CCSMN can be ignored, they are always 0

• MCMX indicates the total reserved memory size of Klass Metaspace and NoKlass Metaspace. For example, by default, Klass Metaspace reserves 1G of memory through the CompressedClassSpaceSize parameter, and the default reserved memory size of NoKlass Metaspace is 2* InitialBootClassLoaderMetaspaceSize

• CCSMX indicates the memory size of Klass Metaspace reserved

To sum up, in fact, the most important thing to look at metaspace is to look at MC, MU, CCSC, CCSUthese specific sizes to judge how much metaspace is used is more reliable

Originally, I wanted to write about metaspace memory allocation and GC, but that is a relatively big topic, because it may seem boring to everyone, so I will write again when I have a chance

Author: You are stupid
Link: https://www.jianshu.com/p/92a5fbb33764

For communication, please pay attention to the official account.