GC murder comes together from the principle of reflection

Foreword

First answer the question questioner. This is mainly due to the presence of a large number of reflections generated temporary class loader and ASM temporarily generated classes will be retained in Metaspace, once Metaspace nearly full time, will trigger FullGc, the recovery has reached the class is no longer used the purpose of the object. Please refer to the following specific questions of content, to achieve a better understanding of the principle of reflection.

text

Outline

There is a large memory system (70G or more) before the company has been using CMS GC, but because the system is very sensitive to the time, because gclocker occasionally lead to remark particularly long (although the addition of -XX: + CMSScavReengeBeforeRemark parameters, but gclocker that might cause remark the YGC delay), can not stand such a long pause had to move to the G1, after a series of tune as relatively stable, this parameter will be pushed to all the machines on the

But just last week, the machine suddenly appeared Full GC, originally designed G1 is hoped Full GC does not occur, there is generally not normal Full GC, GC log is as follows:

From the above log is not difficult to find because of the Perm trigger Full GC, and after Full GC Perm to go down, but need to mention that JDK7 normal G1 GC will not do like to uninstall, and only when it is Full GC will be uninstalled, but JDK8 is provided under the relevant parameters can be done in class unload some stage G1 GC

So to the business side of coredump do first, save the scene and then reboot the system, and then made a coredump for heap dump, but there heapdump 40G so big, you can take a look at how perm by jmap -permstat core.xxx there anything

This article is relatively long, more related to the knowledge, if it can not help but read on, you can skip to the last look at my description of the problem and then turn reading this article may give you a clearer understanding

What exactly stuffed in Perm

Since it is a Perm full, then we have to look at Perm where exactly to put what we know Perm in the main memory is the class of the raw data, such as we loaded a class, and that information of this kind will allocate memory in Perm where to store some of its data structure, so in most cases, the amount of Perm and the number of classes loaded is a great relationship, of course, in Perm will save some other data in the lower version of the time, such as String (String.intern ()Case).

In addition Experience tells us that if it is really Perm overflow, that possibility will have a place to build a dynamic class loader loads a class is large, jmap by the above command, we can count the number actually reached under sun.reflect.DelegatingClassLoader the 415 737

It can be basically locked reflective class loader Perm cause overflow, then why on earth is there so much of it reflecting the class loader, reflecting what is the class loader, then briefly say that under the principle of reflection

Principles reflected

Reflecting everyone is easy to use, because the performance is actually quite good, so with quite wide, so we usually use reflection

Method method = XXX.class.getDeclaredMethod(xx,xx);method.invoke(target,params)

But here I am not going to use a lot of code to describe its principle, but say a few key things, and then they string together

Get Method

Method to obtain the first call, acquired by Method Class logic of this class, and several key methods and properties as follows:

There are in Class in a key property called reflectionData, where the main deposit was obtained from each jvm, to some class attributes, such as methods, fields, etc., is probably a long way

This property is mainly SoftReference, which is in some cases more demanding memory is likely to be recycled, but under normal circumstances can -XX: to control the timing of the recovery of SoftRefLRUPolicyMSPerMB this parameter, when the time is up, as long as they occur GC will repossess it, and then after that the recovery means that there is a demand of the time to re-create one of these objects, but also need to take a piece of data from the JVM, and that the associated data structure Method, field and other fields are regenerated Object. If you are re-generated objects that might have any trouble? Talked back to understand

getDeclaredMethod method is very simple, is to copy a Method object from the list returned by the method privateGetDeclaredMethods in return. And this copying process is achieved by searchMethods

If reflectionData declaredMethods this property is not empty, then return to their privateGetDeclaredMethods directly on it, otherwise go fishing one out of the JVM, and assigned to reflectionData field, so next time when you can call privateGetDeclaredMethods use the cached data , do not always go get the data transferred to the JVM, because reflectionData is Softreference, so there is a risk to get any value, taking less than once before he went fishing in the JVM

searchMethods from the list of methods privateGetDeclaredMethods's find a way to return a match of the same name, and then copy a method object out of the copy of the specific implementation, in fact Method.copy method:

Thus, Method objects every time we returned by calling getDeclaredMethod methods are in fact a new object, it is not appropriate to adjust Oh, if you call frequently the best cached. But this new approach object has a root attribute points to a method reflectionData in the cache, the cache while its methodAccessor also used in that Method of methodAccessor.

Method calls

Once you have Method, it can call its invoke method, then take a look at a few key information Method

In fact, the root property has been said above, the main point to cache the Method object, that is, the current Method object is actually constructed according to root out this Method, and therefore the presence of a root Method Method of multiple derived.

methodAccessor this is key, in fact Method.invoke method is to call the invoke method methodAccessor, methodAccessor this property if the root itself has been, it would come directly methodAccessor assignment of root, otherwise it will create a

MethodAccessor implementation

MethodAccessor itself is an interface

There are three main realization

• DelegatingMethodAccessorImpl
• NativeMethodAccessorImpl
• GeneratedMethodAccessorXXX

DelegatingMethodAccessorImpl which is the final injection of methodAccessor to Method, which is a Method of all invoke method will be called to this DelegatingMethodAccessorImpl.invoke, the same as its name, is the agent, that is the true realization of the following may be two kind

If NativeMethodAccessorImpl, that name suggests, this is achieved mainly native achieved, and each needs to be reflected GeneratedMethodAccessorXXX Method invoked dynamically generated class, after a number XXX is constantly increasing and are reflected all the way go NativeMethodAccessorImpl, after the default tune of 15 times, it generates a GeneratedMethodAccessorXXX class, we will go after generating good invoke this method to generate classes from NativeMethodAccessorImpl how that transition to GeneratedMethodAccessorXXX it, take a look invoke the method NativeMethodAccessorImpl

I said above, which is 15 times that ReflectionFactory.inflationThreshold () This method returns, this 15 course, is not her unchangeable, we can be specified by -Dsun.reflect.inflationThreshold = xxx, we can also -Dsun .reflect.noInflation = true to bypass the 15 NativeMethodAccessorImpl call the above, and -Dsun.reflect.inflationThreshold = 0, the same effect and GeneratedMethodAccessorXXX are, once after it is created to generate through new MethodAccessorGenerator (). generateMethod set to DelegatingMethodAccessorImpl gone, so the next Method.invoke will be transferred to the newly created MethodAccessor Lane.

That generated GeneratedMethodAccessorXXX exactly look like it, this is probably the

In fact, the specific method of directly calling the target object, and no difference between the normal method calls

GeneratedMethodAccessorXXX class loader

What load GeneratedMethodAccessorXXX class loader is it, after generating a good byte code calls the following method to do class definition

So GeneratedMethodAccessorXXX class loader is actually a class loader DelegatingClassLoader

The reason to engage in a new class loader, for performance reasons, and in some cases can uninstall these generated classes, because unloading the class is only in the case of class loader can be recycled will be recycled, if with the original class loader that can cause these newly created class has been unable to be unloaded from the design point of view itself does not want them to have the memory, when needed there on the line exists in memory when tight can be freed memory

Concurrent garbage leads to class creation

See here do not know if you found a problem, in fact, are not above NativeMethodAccessorImpl.invoke locked What does it mean? If the concurrent high time, may also mean that there are many threads to create logical GeneratedMethodAccessorXXX entering class, in fact, although there will only be a final use, but these costs are not already in existence, if there are 1000 thread have entered into the logic created GeneratedMethodAccessorXXX's, that means creating more than 999 unwanted classes would have been occupied memory, GC can be recovered until the recovery will take place in Perm

That is exactly what methods are constantly reflecting it

Once you have reflected on the face of the principles of understanding, we know that after a certain number of times to perform reflection, in fact, will build a dynamic class, in this class corresponding to the target object method calls directly, as we've learned in the heap dump there are a lot of DelegatingClassLoader class loader loads the GeneratedMethodAccessorXXX class, and that in the end these classes are called what methods do, so we have to do one thing, and that is the memory of these classes dump down, and then do bytecode a statistical analyze

Dump class bytecode runtime

We can use the SA interface from coredump in progress or live in the corresponding class dump down, to dump down our particular class, first class we wrote a Filter

Use of SA jar ($ JAVA_HOME / lib / sa-jdi.jar) compiled after class, then we call the following command to dump the compiled class in the directory

So that we can all GeneratedMethodAccessor to dump down, and this time we'll just look at the bytecode of a class by javap -verbose GeneratedMethodAccessor9

See above key bci to the line 36, where the method is the method we call the reflection, such as the reflection method call above is org / codehaus / xfire / util / ParamReader.readCode

Reflective target specific classes and methods

After dump these byte code, we do a byte-code all of these statistics class, you find all the reflection method call, and then find some model class (package is the same) actually produces more than 20 ten thousand class, which means there are a lot of these model classes do reflection

With this clue is to look after the code exactly where there will be a logical reflection call these model methods, but unfortunately did not find, but this model is highly likely target in some cases, it is rpc deserialization time, the final party is asking business Xfire service use, and with the experience I have accumulated many years of development framework, is to determine Xfire deserialize objects by way of reflection, the specific code as follows (org.codehaus.xfire.aegis.type.basic .BeanType.writeProperty):

The javabean of PropertyDeor in the get / set methods, in fact, the packaging itself is SoftReference

See here perhaps we all understand it, has earlier said that it is possible to be recovered GC SoftReference out, the time comes it will be recovered in the next GC, if recovered, would have to re-acquire, then the equivalent of new method object invoke method calls, calls that number more than one, it will have to build a new dynamic class, and this class will always be saved to a GC until it can be recovered Perm

G1 recovered Perm

Note that under the operational system uses G1 JDK7, and JDK7 the G1 to perm actually under normal circumstances is not recycled, will only recover when Full GC in Perm, which explains after a number of G1 GC, Softreference those objects will be recovered, but the newly created class and in fact will not be recovered, so the more frequent G1 GC, that means SoftReference objects more likely to be recovered (although under normal circumstances is to time, but if not frequently gc , even if the time is up, will stay in memory of), more likely to be recovered and that the easier it is to produce a new class until the Full GC occurs

solution

• Upgrading to jdk8, you can do the class in G1 GC unloading process
• For a sequence of protocol, the method does not take the reflection, such as hessian
• SoftRefLRUPolicyMSPerMB adjust this parameter increases, but this is not a permanent cure

to sum up

The above relates to very much, if not more to read several times it may be difficult to string together, I will be here this situation occurred roughly describe the problem:

This system uses JDK7 under G1, and this version of the G1 will only do class of Perm in the Full GC unloaded at the time of the system because a large number of requests resulting in G1 GC happen very often, but the system is also set up -XX : SoftRefLRUPolicyMSPerMB = 0, that means SoftReference life cycle will not cross GC cycle, can quickly recover lost, there are a lot of RPC calls this system, walking Xfire agreement to return the results do deserialization time is to go Method.invoke logic, and associated method therefore SoftReference reference, it can easily be recycled, once recovered, it would create a new method object, then call its invoke method, after a call to a certain number of times (15 times) , to build a new bytecode class, along with the conduct of the GC, bytecode class with a method of continuously building until full Perm trigger a full GC was able to release

Recommended Reading

Once the investigation produce 100% CPU optimization practice

Talk about a Java application from a real case GC optimization