This article comes from: PerfMa Technology Community
Overview
This article is based on a problem that was recently investigated, and it took our team a lot of time to troubleshoot the problem. The phenomenon is that some class loaders are placed as keys in WeakHashMap, but after many times of full gc, they are still strong. Stored in memory, but from a code point of view, these class loaders should be recycled, because there is no strong reference to these class loaders, so we did a memory dump, analyzed the memory, and found that in addition to a WeakHashMap There is no other GC ROOT way to reach these class loaders, so after many times FULL GC can certainly be recycled, but the fact is not the case, in order to make this problem sound better understood, or as usual This Demo completely simulates this scenario.
Demo
First, we create two classes AAA and AAB, which are packaged into two different jars, such as AAA.jar and AAB.jar. There is a relationship between these two classes. There is an attribute in AAA that is of type AAB. Note Do not put these two jars in the classpath to allow appClassLoader to load into:
public class AAA {
private AAB aab;
public AAA(){
aab=new AAB();
}
public void clear(){
aab=null;
}
}
public class AAB {}
复制代码Then we create a class to load TestLoader, which contains a WeakHashMap, specifically to store TestLoader, and override the loadClass method, if it is to load the AAB class, create a new TestLoader to load this class from AAB.jar
import java.net.URL;
import java.net.URLClassLoader;
import java.util.WeakHashMap;
public class TestLoader extends URLClassLoader {
public static WeakHashMap<TestLoader,Object> map=new WeakHashMap<TestLoader,Object>();
private static int count=0;
public TestLoader(URL[] urls){
super(urls);
map.put(this, new Object());
}
@SuppressWarnings("resource")
public Class<?> loadClass(String name) throws ClassNotFoundException {
if(name.equals("AAB") && count==0){
try {
count=1;
URL[] urls = new URL[1];
urls[0] = new URL("file:///home/nijiaben/tmp/AAB.jar");
return new TestLoader(urls).loadClass("AAB");
}catch (Exception e){
e.printStackTrace();
}
}else{
return super.loadClass(name);
}
return null;
}
}
复制代码Looking at our main class TTest, some instructions are written in the class:
import java.lang.reflect.Method;
import java.net.URL;
/**
* Created by nijiaben on 4/22/16.
*/
public class TTest {
private Object aaa;
public static void main(String args[]){
try {
TTest tt = new TTest();
//将对象移到old，并置空aaa的aab属性
test(tt);
//清理掉aab对象
System.gc();
System.out.println("finished");
}catch (Exception e){
e.printStackTrace();
}
}
@SuppressWarnings("resource")
public static void test(TTest tt){
try {
//创建一个新的类加载器，从AAA.jar里加载AAA类
URL[] urls = new URL[1];
urls[0] = new URL("file:///home/nijiaben/tmp/AAA.jar");
tt.aaa=new TestLoader(urls).loadClass("AAA").newInstance();
//保证类加载器对象能进入到old里，因为ygc是不会对classLoader做清理的
for(int i=0;i<10;i++){
System.gc();
Thread.sleep(1000);
}
//将aaa里的aab属性清空掉，以便在后面gc的时候能清理掉aab对象，这样AAB的类加载器其实就没有什么地方有强引用了，在full gc的时候能被回收
Method[] methods=tt.aaa.getClass().getDeclaredMethods();
for(Method m:methods){
if(m.getName().equals("clear")){
m.invoke(tt.aaa);
break;
}
}
}catch (Exception e){
e.printStackTrace();
}
}
}
复制代码When running, please run under JDK8, make a breakpoint System.out.println("finished"), and then do a memory dump.
From the above example, we know that TTest is loaded by the class loader AppClassLoader, the object type of its attribute aaa is loaded from AAA.jar through TestLoader, and the aab attribute in aaa is from a brand new class loader TestLoader Loaded from AAB.jar, when we do System GC multiple times, these objects will be moved to old, after doing the last GC, the aab object will be removed from memory, and its class loader is no longer at this time There is a strong reference anywhere, and only one WeakHashMap refers to it. In theory, it should be recycled when doing GC, but in fact, the class loader of this AAB has not been recycled. From the analysis results, the GC ROOT path is WeakHashMap .
Metaspace in JDK8
One concept I have to mention here is the metaspace in JDK8. It is to replace perm. As for the benefits, I personally feel that it is not so obvious. It is a bit of a thankless feeling. The code has changed a lot, but the actual benefits are not obvious. It is said to be a result of the internal struggles of Oracle.
Although there is no perm in JDK8, the klass information still needs to be stored somewhere. Two pieces of memory are allocated for this in the jvm, one is next to the heap, just like perm, it is specifically used to store klass information, The -XX:CompressedClassSpaceSizesize can be set by , and the other piece is not necessarily connected to them. It mainly stores other information than non-klass, such as constant pool or something, which can -XX:InitialBootClassLoaderMetaspaceSizebe set by. At the same time, we can also -XX:MaxMetaspaceSizeset the threshold that triggers the recovery of metaspace.
Each class loader will take some metaChunk from the global metaspace to manage it. When there is a class definition, it is actually allocated from these memories. When it is not enough, it will allocate a block in the global metaspace and manage it.
This specific situation can be followed by writing an article to introduce, including memory structure, memory allocation, GC and so on.
ClassLoaderDataGraph in JDK8
Each class loader will correspond to a ClassLoaderData data structure, which will store, for example, specific class loader objects, loaded klass, memory management metaspace, etc. It is a chain structure that will be linked to the next ClassLoaderData, gc When traversing these ClassLoaderData through ClassLoaderDataGraph, the first ClassLoaderData of ClassLoaderDataGraph is bootstrapClassLoader
class ClassLoaderData : public CHeapObj<mtClass> {
...
static ClassLoaderData * _the_null_class_loader_data;
oop _class_loader; // oop used to uniquely identify a class loader
// class loader or a canonical class path
Dependencies _dependencies; // holds dependencies from this class loader
// data to others.
Metaspace * _metaspace; // Meta-space where meta-data defined by the
// classes in the class loader are allocated.
Mutex* _metaspace_lock; // Locks the metaspace for allocations and setup.
bool _unloading; // true if this class loader goes away
bool _keep_alive; // if this CLD is kept alive without a keep_alive_object().
bool _is_anonymous; // if this CLD is for an anonymous class
volatile int _claimed; // true if claimed, for example during GC traces.
// To avoid applying oop closure more than once.
// Has to be an int because we cas it.
Klass* _klasses; // The classes defined by the class loader.
JNIHandleBlock* _handles; // Handles to constant pool arrays
// These method IDs are created for the class loader and set to NULL when the
// class loader is unloaded. They are rarely freed, only for redefine classes
// and if they lose a data race in InstanceKlass.
JNIMethodBlock* _jmethod_ids;
// Metadata to be deallocated when it's safe at class unloading, when
// this class loader isn't unloaded itself.
GrowableArray<Metadata*>* _deallocate_list;
// Support for walking class loader data objects
ClassLoaderData* _next; /// Next loader_datas created
// ReadOnly and ReadWrite metaspaces (static because only on the null
// class loader for now).
static Metaspace* _ro_metaspace;
static Metaspace* _rw_metaspace;
...
}
复制代码Here are a few attributes:
-
_class_loader: Is the corresponding class loader object -
_keep_alive: If this value is true, then this class loader will be considered alive and will be used as part of GC ROOT, and will not be recycled when gc -
_unloading: Indicates whether this class loading needs to be unloaded -
_is_anonymous: Whether it is anonymous, this ClassLoaderData is mainly used in lambda expressions, I will talk about this in detail later -
_next: Point to the next ClassLoaderData, easy to traverse in gc -
_dependencies: This attribute is also the focus of this article, which will be detailed later
Let's look at the constructor again:
ClassLoaderData::ClassLoaderData(Handle h_class_loader, bool is_anonymous, Dependencies dependencies) :
_class_loader(h_class_loader()),
_is_anonymous(is_anonymous),
// An anonymous class loader data doesn't have anything to keep
// it from being unloaded during parsing of the anonymous class.
// The null-class-loader should always be kept alive.
_keep_alive(is_anonymous || h_class_loader.is_null()),
_metaspace(NULL), _unloading(false), _klasses(NULL),
_claimed(0), _jmethod_ids(NULL), _handles(NULL), _deallocate_list(NULL),
_next(NULL), _dependencies(dependencies),
_metaspace_lock(new Mutex(Monitor::leaf+1, "Metaspace allocation lock", true)) {
// empty
}
复制代码It can be seen that _keep_ailvethe value of the attribute is based on _is_anonymouswhether the current class loader is bootstrapClassLoader.
_keep_aliveWhere exactly is it used? In fact, it is during GC, to decide whether to use Closure or what Closure to use to scan the corresponding ClassLoaderData.
void ClassLoaderDataGraph::roots_cld_do(CLDClosure* strong, CLDClosure* weak) {
//从最后一个创建的classloader到bootstrapClassloader
for (ClassLoaderData* cld = _head; cld != NULL; cld = cld->_next) {
//如果是ygc，那weak和strong是一样的，对所有的类加载器都做扫描，保证它们都是活的
//如果是cms initmark阶段，如果要unload_classes了(should_unload_classes()返回true)，则weak为null，那就只遍历bootstrapclassloader以及正在做匿名类加载的类加载
CLDClosure* closure = cld->keep_alive() ? strong : weak;
if (closure != NULL) {
closure->do_cld(cld);
}
}
复制代码When is the class loader recycled
Whether the class loader needs to be recycled, in fact, it depends on whether the class loader object is alive. The so-called live is that any class loaded by the class loader or the objects of these classes are strongly reachable. The loader itself is part of the GC ROOT or there is a path reachable by the GC ROOT, then this class loader will definitely not be recycled.
From various GC situations:
-
If it is YGC, the class loader is used as GC ROOT, that is, it will not be recycled
-
If it is Full GC, it will be recycled as long as it is dead
-
If it is a CMS GC, the CMS GC process will also be marked (this is the default, but it can be changed by some parameters), but no real cleaning will be done. The real cleaning action will occur when the security point is next entered.
How to generate zombie class loader
If the class loader is truly dependent on the GC ROOT object, this class loader object is alive and well. We can analyze it through zprofiler or mat, and we can draw the link, but there are two cases. exception:
lambda anonymous class loading
Lambda anonymous class loading is unsafe's defineAnonymousClass method, this method corresponds to the following method in vm
UNSAFE_ENTRY(jclass, Unsafe_DefineAnonymousClass(JNIEnv *env, jobject unsafe, jclass host_class, jbyteArray data, jobjectArray cp_patches_jh))
{
instanceKlassHandle anon_klass;
jobject res_jh = NULL;
UnsafeWrapper("Unsafe_DefineAnonymousClass");
ResourceMark rm(THREAD);
HeapWord* temp_alloc = NULL;
anon_klass = Unsafe_DefineAnonymousClass_impl(env, host_class, data,
cp_patches_jh,
&temp_alloc, THREAD);
if (anon_klass() != NULL)
res_jh = JNIHandles::make_local(env, anon_klass->java_mirror());
// try/finally clause:
if (temp_alloc != NULL) {
FREE_C_HEAP_ARRAY(HeapWord, temp_alloc, mtInternal);
}
// The anonymous class loader data has been artificially been kept alive to
// this point. The mirror and any instances of this class have to keep
// it alive afterwards.
if (anon_klass() != NULL) {
anon_klass->class_loader_data()->set_keep_alive(false);
}
// let caller initialize it as needed...
return (jclass) res_jh;
}
UNSAFE_END
}
复制代码It can be seen that after the successful creation of the anonymous class, the corresponding ClassLoaderData _keep_aliveproperty will be set to false, does that mean that the _keep_aliveproperty is true before this? The following parse_streammethod is the method that will be transferred from the above method
Klass* SystemDictionary::parse_stream(Symbol* class_name,
Handle class_loader,
Handle protection_domain,
ClassFileStream* st,
KlassHandle host_klass,
GrowableArray<Handle>* cp_patches,
TRAPS) {
TempNewSymbol parsed_name = NULL;
Ticks class_load_start_time = Ticks::now();
ClassLoaderData* loader_data;
if (host_klass.not_null()) {
// Create a new CLD for anonymous class, that uses the same class loader
// as the host_klass
assert(EnableInvokeDynamic, "");
guarantee(host_klass->class_loader() == class_loader(), "should be the same");
guarantee(!DumpSharedSpaces, "must not create anonymous classes when dumping");
loader_data = ClassLoaderData::anonymous_class_loader_data(class_loader(), CHECK_NULL);
loader_data->record_dependency(host_klass(), CHECK_NULL);
} else {
loader_data = ClassLoaderData::class_loader_data(class_loader());
}
instanceKlassHandle k = ClassFileParser(st).parseClassFile(class_name,
loader_data,
protection_domain,
host_klass,
cp_patches,
parsed_name,
true,
THREAD);
...
}
ClassLoaderData* ClassLoaderData::anonymous_class_loader_data(oop loader, TRAPS) {
// Add a new class loader data to the graph.
return ClassLoaderDataGraph::add(loader, true, CHECK_NULL);
}
ClassLoaderData* ClassLoaderDataGraph::add(Handle loader, bool is_anonymous, TRAPS) {
// We need to allocate all the oops for the ClassLoaderData before allocating the
// actual ClassLoaderData object.
ClassLoaderData::Dependencies dependencies(CHECK_NULL);
No_Safepoint_Verifier no_safepoints; // we mustn't GC until we've installed the
// ClassLoaderData in the graph since the CLD
// contains unhandled oops
ClassLoaderData* cld = new ClassLoaderData(loader, is_anonymous, dependencies);
...
}
复制代码From the above code, as long as the unsafe method is taken, a ClassLoaderData object will be created for the current class loader and its _is_anonymous is set to true, which also means that the property of _keep_alive is true and added to ClassLoaderDataGraph .
Imagine if the created anonymous class is unsuccessful, that is, anon_klass () == null, then the _keep_alive property can never be set to false, which means that the ClassLoader object corresponding to this ClassLoaderData will always be part of the GC ROOT, Can't be recycled, this situation is a real zombie class loader, but I have not yet simulated this situation, interested students can try it, if it can be simulated, this is definitely in the JDK A bug can be submitted to the community.
Caused by class loader dependencies
The class loader dependency mentioned here does not mean the dependency relationship established by the parent in the ClassLoader. If it is this relationship, it can actually be analyzed by tools such as mat or zprofiler, but there is still a situation. , Those tools can not be analyzed, this relationship is derived from the _dependencies attribute in ClassLoaderData, for example, if the _dependencies attribute of the A class loader records the B class loader, then when the GC traverses the A class The loader will also traverse the class B loader and mark it live, even if the class B loader can actually be recycled, you can look at the following code
void ClassLoaderData::oops_do(OopClosure* f, KlassClosure* klass_closure, bool must_claim) {
if (must_claim && !claim()) {
return;
}
f->do_oop(&_class_loader);
_dependencies.oops_do(f);
_handles->oops_do(f);
if (klass_closure != NULL) {
classes_do(klass_closure);
}
}
复制代码The question arises, how is this dependency recorded? In fact, our demo above simulates this situation. You can take a closer look. I will also describe this demo. For example, after loading AAA with TestLoader, a class loader that loads AAA, and creating AAA objects, you will see a The type is an attribute of AAB. At this time, the type in the constant pool will be parsed. When we see the loadClass method of TestLoader, we make a judgment. If it is an AAB type class loading, then create a new class loader The object is loaded from AAB.jar. When the load returns, such a layer of dependency is actually recorded in the jvm. It is considered that the AAA class loader depends on the AAB class loader and is recorded, but looking at all The hotspot code does not have a place to clean up this dependency relationship. That is to say, as long as this dependency relationship is established, it will continue until the AAA class loader is recycled, and the AAB class loader will be recycled. So this is a pseudo-zombie class loader. Although it does not depend on the dependency relationship (such as the clearing action of the aab attribute of AAA in the demo), the GC will always It is believed that they have such a dependency relationship and will continue to exist for a period of time, depending on the AAA class loader.
In view of this situation, I personally think that a GC strategy similar to reference counting is needed. When certain two class loaders do not have any dependencies, they will be cleared of this dependency relationship. It is estimated that there are many places to implement such changes. It's not that simple, so the designer at the time probably didn't do it because of this. I think this is the result of lazy compromise. Of course, this is just my guess.
Let's study together :
JVM parameters of PerfMa KO series course [Memory]
Design enterprise-level truly high-availability distributed locks based on CAP model