java.lang.OutOfMemoryError abnormal Complete Guide

I've seen thousands of career out of memory anomalies are associated with the following eight situations. This paper analyzes what causes these abnormal, provides sample code at the same time to provide you with a troubleshooting guide.

        -- Nikita Salnikov-Tarnovski (Plumbr Co-Founder and VP of Engineering)

In this article from Plumbr, there are deletions and additions to the original content

 

This is perhaps the most complete Java OOM abnormalities resolved guide.

 

1、java.lang.OutOfMemoryError:Java heap space

Java application specifies the memory size needed at startup, it is divided into two distinct regions: Heap space (heap space) and PermGen (permanent generation):

 

wKioL1lusMjS9ur6AAACjSTeFSw383.png-wh_50

 

The size of the two regions by parameter -Xmx and -XX in the JVM (Java Virtual Machine) Start: MaxPermSize settings, if you do not explicitly set, the default value for a particular platform will be used.

 

When an application tries to add more data to the stack space, heap but did not have enough space to accommodate these data will trigger java.lang.OutOfMemoryError: Java heap space exception. Note that: even if there is enough physical memory is available, as long as the size limit is reached heap space is set, the exception will still be triggered.

 

Cause Analysis

Trigger java.lang.OutOfMemoryError: The most common reason is that Java heap space application heap space is needed XXL number, but the JVM is provided by S number. The solution is simple, to provide more heap space. In addition to the front of the factors there are more complex causes:

 

  • Flow / peak data volume: Each user application amount and the amount of data is limited to the initial design, a time, or the amount of data when the number of users suddenly reaches a peak, and the peak value has exceeded the expected threshold value at the beginning of the design, so before normal function will stop, and trigger java.lang.OutOfMemoryError: Java heap space exception.

  • Memory leak: specific programming errors can cause your application to stop consuming more memory, each will leave that can not be recycled objects to the use of heap space memory leak risk function, as time Over the leak object will consume all of the heap space, eventually triggering java.lang.OutOfMemoryError: Java heap space error.

 

Examples

①, simple example

First look at a very simple example, the following code attempts to create 2 x 1024 x 1024 integers array elements, when you try to compile and specify 12M heap space run (java -Xmx12m OOM) will fail and throw java. lang.OutOfMemoryError: Java heap space error, and when you specify 13M heap space, the normal operation.

 

                Computing array occupied by memory size, it is no longer within the scope of this article, the reader interested, you can calculate their own

class OOM {
    static final int SIZE=2*1024*1024;
    public static void main(String[] a) {
        int[] i = new int[SIZE];
    }
}

Run as follows:

D:\>javac OOM.java
D:\>java -Xmx12m OOM
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at OOM.main(OOM.java:4)
D:\>java -Xmx13m OOM

 

②, a memory leak example

In Java, when developers create a new object: when (for example, new Integer (5)), do not need to open up their own memory space, but give it to the JVM. In applications throughout the life cycle class, JVM is responsible for checking which objects are available, which objects are not being used. Unused object will be discarded, memory footprint will also be recovered, a process called garbage collection. Module is responsible for garbage collection JVM is called the garbage collector (GC).

 

Java's automatic memory management mechanism depends on the GC on a regular basis to find unused objects and delete them. Java is due to a memory leak in the GC can not recognize some objects no longer in use, and these unused objects have been left in the heap space, which will eventually lead to the accumulation of java.lang.OutOfMemoryError: Java heap space error.

 

We can be very easy to write Java code causes a memory leak:

public class KeylessEntry {

    static class Key {
        Integer id;

        Key(Integer id) {
            this.id = id;
        }

        @Override
        public int hashCode() {
            return id.hashCode();
        }
    }

    public static void main(String[] args) {
        Map<Key,String> m = new HashMap<Key,String>();
        while(true) {
            for(int i=0;i<10000;i++) {
                if(!m.containsKey(new Key(i))) {
                    m.put(new Key(i), "Number:" + i);
                }
            }
        }
    }
}

 

HashMap local cache tag, while the first loop, 10000 elements will be added to the cache. After the while loop, because the key already exists in the cache, the cache size will always remain at 10000. But that really true? Because Key entity does not implement equals () method, resulting in a for loop each execution m.containsKey (new Key (i)) the results are false, the result is a HashMap elements will always increase.

 

Over time, more and more Key object into the heap space and can not be recovered garbage collector (m is a local variable, GC would think that these objects are always available, so it will not recover), until all the stack space is occupied Finally, throw java.lang.OutOfMemoryError: Java heap space.

 

The above code directly for a long time to run may not throw an exception, you can use when you start -Xmx parameter, set the heap memory size, or the size of the print HashMap after the for loop, after the execution will find HashMap of size has been re-growth.

The solution is very simple, as long as the Key to realize their equals method:

Override
public boolean equals(Object o) {
    boolean response = false;
    if (o instanceof Key) {
        response = (((Key)o).id).equals(this.id);
    }
    return response;
}

 

solution

The first solution is obvious, you should ensure that there is sufficient heap space to run your application correctly, add the following configuration in the startup configuration of the JVM:

 

-Xmx1024m

1024M allocated heap space above configuration for your application, of course you can also use other units, such as GB represented by G, K represents KB. The following examples are representing the maximum heap space for the 1GB:

 

java -Xmx1073741824 com.mycompany.MyClass

java -Xmx1048576k com.mycompany.MyClass

java -Xmx1024m com.mycompany.MyClass

java -Xmx1g com.mycompany.MyClass

Then, more often, simply to increase heap space can not solve all problems. If you program a memory leak, blindly increase heap space is only postpone java.lang.OutOfMemoryError: Time Java heap space errors only, does not address this risk. In addition, the garbage collector in the GC, the application will stop running until the GC, and GC increase heap space can lead to prolonged, thereby affecting the throughput of the program.

 

If you want to solve this problem completely, then have to enhance their programming skills, of course make good use Debuggers, profilers, heap dump analyzers and other tools that lets you program the maximum extent to avoid memory leaks.

 

2、java.lang.OutOfMemoryError:GC overhead limit exceeded

 

Java Runtime Environment (JRE) contains a built-in garbage collection process, and in many other programming languages, developers need to manually allocate and free memory.

 

Java application developers only need to allocate memory, whenever a particular memory space is no longer in use, a separate garbage collection process will clear the memory space. How the garbage collector detect certain memory space is no longer in use beyond the scope of this article, but you just need to believe you can do a good job to GC.

 

By default, when the application is spending more than 98% of the time used for GC and recovered less than 2% of heap memory, it will throw java.lang.OutOfMemoryError: GC overhead limit exceeded error. Specific performance of your application is almost exhausted all available memory, and GC repeatedly failed to clean up.

 

Cause Analysis

java.lang.OutOfMemoryError: GC overhead limit exceeded error is a signal, indicating your application is spending too much time on garbage collection but with no eggs. The default more than 98% of the time used for GC but recovered will throw this error less than 2% of the memory. If this limit is that nothing will happen? GC process will be restarted, 100% of the CPU will be used for GC, but no CPU resources for other work properly. If a job had only a few milliseconds to complete, now takes a few minutes to complete, I think no one can not accept this result.

 

Therefore java.lang.OutOfMemoryError: GC overhead limit exceeded can also be seen in the example of a practical fail-fast (fail-fast).

 

Examples

The following code initializes a map and kept adding key-value pairs in an infinite loop, running will throw GC overhead limit exceeded error:

public class Wrapper {
    public static void main(String args[]) throws Exception {
        Map map = System.getProperties();
        Random r = new Random();
        while (true) {
            map.put(r.nextInt(), "value");
        }
    }
}

 

 

As you might expect, the program does not result normally, in fact, when we start the program using the following parameters:

java -Xmx100m -XX:+UseParallelGC Wrapper

We will soon be able to see the program throws java.lang.OutOfMemoryError: GC overhead limit exceeded error. However, if a different set of heap space at startup or use different GC algorithms, like this:

java -Xmx10m -XX:+UseParallelGC Wrapper

We will see the following error:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.util.Hashtable.rehash(Unknown Source)
    at java.util.Hashtable.addEntry(Unknown Source)
    at java.util.Hashtable.put(Unknown Source)
    at cn.moondev.Wrapper.main(Wrapper.java:12)

GC using the following algorithm: -XX: + UseConcMarkSweepGC or -XX: + UseG1GC, start command is as follows:

java -Xmx100m -XX:+UseConcMarkSweepGC Wrapper
java -Xmx100m -XX:+UseG1GC Wrapper

The result is this:

Exception: java.lang.OutOfMemoryError thrown from 
the UncaughtExceptionHandler in thread "main"

Error has been captured default exception handler, without any error stack information output.

 

These changes can be explained, in the context of limited resources, you can not can not predict how your application hang, when it will hang, so in development, you can not only ensure their own application-specific environment under normal operation.

 

solution

The first is a disingenuous solution, if you just want to see java.lang.OutOfMemoryError: GC overhead limit exceeded error message, you can add the following JVM argument when the application starts:

-XX:-UseGCOverheadLimit

It is strongly recommended not to use this option, because it does not solve any problems, but delayed the error, the error message has become more familiar java.lang.OutOfMemoryError: Java heap space only.

 

Another solution, if your application does insufficient memory, increasing the heap memory will solve the problem GC overhead limit, as follows, to your application 1G of heap memory:

java -Xmx1024m com.yourcompany.YourClass

But if you want to make sure that you have solved the underlying problem rather than hide java.lang.OutOfMemoryError: GC overhead limit exceeded error, then you should not just stop there. You have to remember there are profilers and memory dump analyzers these tools, you need to spend more time and effort to find the problem. Another point to note, these tools have significant overhead in the Java runtime, and is not recommended for use in a production environment.

 

3、java.lang.OutOfMemoryError:Permgen space

In Java JVM heap space is managed by the largest piece of memory space, you can specify the size of the JVM heap space at the start, the pile is divided into two distinct areas: the new generation (Young) and the old year (Tenured), and the new generation It is divided into three areas: Eden, From Survivor, to Survivor, as shown in FIG.

wKioL1lus2yRQX90AABm6ATLvkU235.jpg-wh_50

 

java.lang.OutOfMemoryError: PermGen space error to show that permanent generation memory in your area have been exhausted.

 

Cause Analysis

To understand java.lang.OutOfMemoryError: reason PermGen space appears, you first need to understand what the Permanent Generation Space usefulness Yes. The main permanent generation information of each class is stored, for example: class loader reference runtime constant pool (all constants, field references, method references, property), the field (Field) data, the method (Method) data, the method code A method bytecode like. We can infer, PermGen depend on the size of the class and the number of loaded classes.

 

Therefore, we can conclude that appear java.lang.OutOfMemoryError: PermGen space reasons is wrong: too much of a class or classes are loaded into the permanent generation (permanent generation).

 

Examples

①, the simplest example

As previously described, the PermGen use and loaded into the JVM is closely related to the number of classes, the following is a simple example:

import javassist.ClassPool;
public class MicroGenerator {
    public static void main(String[] args) throws Exception {
        for (int i = 0; i < 100_000_000; i++) {
            generate("cn.moondev.User" + i);
        }
    }

    public static Class generate(String name) throws Exception {
        ClassPool pool = ClassPool.getDefault();
        return pool.makeClass(name).toClass();
    }
}

Set JVM runtime parameters: -XX: MaxPermSize = 5m, smaller value is better. Note that JDK8 permanent generation space has been completely removed and replaced by a space element (Metaspace), it is best to run JDK1.7 example 1.6 or lower.

 

Keep the code at runtime to generate the class and loaded into a lasting generations, until the full support permanent generation memory space, and finally thrown java.lang.OutOfMemoryError: Permgen space. Generated code uses javassist class library.

 

②、Redeploy-time

(Press ctrl + F5 when the process of re-deployment, you can imagine when you develop, click a button or use the eclipse reploy idea) and a more complex example is the actual Redeploy. When you uninstall an application from the server, as well as the current classloader class load in the absence of reference to an instance, the permanent generation memory space will be cleaned and recycled GC. If there are instances of a class of classloader reference to the current application, then the class Permgen area will not be unloaded, resulting in memory Permgen area has been increased until Permgen space error.

 

Unfortunately, many third-party libraries and poor resource handling (for example: threads, JDBC drivers, file system handle) makes class loader unloading previously used becomes an impossibility. In turn, means that each re-deployment process, the application of all the classes of the previous version will still reside in the Permgen zone, every time you deploy will generate dozens or even hundreds of M's garbage.

 

Take the thread and JDBC drivers said. Many people will use threads to handle it periodically or time-consuming task, this time we must pay attention to the thread of life cycle issues, you need to ensure that thread can not live longer than your application. Otherwise, if the application has been uninstalled, the thread continues to run, this thread will usually maintain a reference to the application's classloader, the result will not say any more. Say one more thing, the developer has the responsibility to deal with this problem, especially if you are a provider of third-party libraries, be sure to close threads provide an interface to handle the cleanup.

 

Let us imagine a use JDBC driver to connect to a relational database sample application. When the application is deployed on the server to: create a classloader server application instance to load all classes (including corresponding JDBC driver). According to the JDBC specification, JDBC driver (for example: com.mysql.jdbc.Driver) upon initialization registers itself to java.sql.DriverManager in. One example of drivers will store the registration process in the DriverManager static field, reference code:

// com.mysql.jdbc.Driver源码
package com.mysql.jdbc;

public class Driver extends NonRegisteringDriver implements java.sql.Driver {
    public Driver() throws SQLException {
    }
    static {
        try {
            DriverManager.registerDriver(new Driver());
        } catch (SQLException var1) {
            throw new RuntimeException("Can\'t register driver!");
        }
    }
}
// // // // // // // // // //
// 再看下DriverManager对应代码
private final static CopyOnWriteArrayList<DriverInfo> registeredDrivers = new CopyOnWriteArrayList<>();

public static synchronized void registerDriver(java.sql.Driver driver,DriverAction da) throws SQLException {
    if(driver != null) {
        registeredDrivers.addIfAbsent(new DriverInfo(driver, da));
    } else {
        throw new NullPointerException();
    }
}

Now, when you uninstall the applications from the server, java.sql.DriverManager will hold a reference to the driver, and then hold a reference classloader used to load an application instance. The classloader still references of all classes of the application. If this needs to be loaded when the program starts 2000 class, taking up about 10MB of permanent-generation (PermGen) memory, it only takes 5 to 10 times redeployment, they will be the default size of the permanent generation of (PermGen) stuffed, then it will trigger java .lang.OutOfMemoryError: PermGen space errors and crashes.

 

solution

① resolve OutOfMemoryError during initialization

When the trigger OutOfMemoryError due PermGen depletion caused during application startup, the solution is simple. Application needs more space to load all of the classes to PermGen area, so we just need to increase its size. To do this, change the application launch configuration, and add (or increase if there is) -XX: MaxPermSize parameters, like the following example:

java -XX:MaxPermSize=512m com.yourcompany.YourClass

 

② resolve OutOfMemoryError when Redeploy

Analysis of the dump file: First, find out where the reference is held; secondly, to your web application to add a closed hook, or remove the reference after the application is uninstalled. You can use the following command to export dump file:

jmap -dump:format=b,file=dump.hprof <process-id>

If the problem is your own code, please modify, if a third-party library, try searching to see if there is "close" port, if not submitted a bug or issue it to developers.

 

OutOfMemoryError running ③ solve

The first thing you need to check whether to allow GC PermGen from unloading the class, JVM's standard configuration quite conservative, as long as create a class, even if they have no instance reference, which will remain in memory, applications need to dynamically create a large number of classes especially when but it is not a long life cycle, allowing the JVM to uninstall the application class very helpful, you can be achieved by adding the following configuration parameters in the startup script:

-XX:+CMSClassUnloadingEnabled

By default, this configuration is not enabled, if you enable it, GC will scan and clean up the area PermGen class is no longer used. Note, however, this configuration takes effect only in the case of UseConcMarkSweepGC, if you use a different GC algorithms, such as: ParallelGC or Serial GC time, this configuration is invalid. Therefore, when using the above configuration, with:

-XX:+UseConcMarkSweepGC

If you have made sure JVM can uninstall class, but still have memory overflow problem, then you should continue to analyze the dump file, the dump file is generated using the following command:

jmap -dump:file=dump.hprof,format=b <process-id>

When you get the generated heap dump file and use a tool like Eclipse Memory Analyzer Toolkit should uninstall failed to find the unloaded class loader, then the class loader loads the class investigation, found a suspicious object, analysis of use or generate code for these classes, to find the root cause of the problem and solve it.

 

4、java.lang.OutOfMemoryError:Metaspace

As already mentioned, the PermGen storage area for class names and fields, for class bytecode method, constant pool, the JIT optimization, but starting Java8, Java memory model in a significant change: the introduction of Metaspace called the new memory area, and deleted PermGen area. Please note: It is not simply the content stored directly PermGen area moved Metaspace area, some parts of PermGen area, has moved to the ordinary heap inside.

wKiom1lutI6Qcg_PAABXbp-tEmQ246.png-wh_50

 

Cause Analysis

Java8 reason to make such a change include, but are not limited to:

 

  • Applications require PermGen difficult to predict the size of the area set too small will trigger PermGen OutOfMemoryError error, setting excessive waste of resources.

  • GC enhance performance, each of the garbage collector in the HotSpot requires special code to handle the metadata information is stored in the class of PermGen. PermGen separate class from the metadata information to Metaspace, since the distribution Metaspace Java Heap and having the same address space, so Metaspace Java Heap and seamless management, but also simplifies the process of FullGC, parallel to and in the future metadata information garbage collection, without GC pauses.

  • Support further optimization, such as: G1 concurrent classes unloaded, are also considered to prepare for the future now

 

As you can see, the yuan space requirements depend on the number and size of such class declaration loaded class. So it is easy to see java.lang.OutOfMemoryError: Metaspace main reasons: too much of a class or classes loaded into the yuan space.

 

Examples

As explained above, the use of the space element number is loaded into the JVM in the class of closely related. The following code is the most simple example:

public class Metaspace {
    static javassist.ClassPool cp = javassist.ClassPool.getDefault();

    public static void main(String[] args) throws Exception{
        for (int i = 0; ; i++) { 
            Class c = cp.makeClass("eu.plumbr.demo.Generated" + i).toClass();
            System.out.println(i);
        }
    }
}

The program is running constantly generate new classes, all definitions of these classes will be loaded into Metaspace area until the space is fully occupied and throw java.lang.OutOfMemoryError: Metaspace. It will crash when starting MaxMetaspaceSize = 32m, a plurality of load approximately 30,000 classes: When using -XX.

31023
31024
Exception in thread "main" javassist.CannotCompileException: by java.lang.OutOfMemoryError: Metaspace
    at javassist.ClassPool.toClass(ClassPool.java:1170)
    at javassist.ClassPool.toClass(ClassPool.java:1113)
    at javassist.ClassPool.toClass(ClassPool.java:1071)
    at javassist.CtClass.toClass(CtClass.java:1275)
    at cn.moondev.book.Metaspace.main(Metaspace.java:12)
    .....

 

solution

The first solution is obvious, since the application will run out of memory space Metaspace area, it should increase its size, change the startup configuration to add the following parameters:

# 告诉JVM:Metaspace允许增长到512,然后才能抛出异常
-XX:MaxMetaspaceSize = 512m

Another method is to remove this parameter to completely lift restrictions on Metaspace size (the default is no limit). By default, the server 64 for the JVM, MetaspaceSize default size is 21M (initial limit value), once the value reaches this limit, that will trigger type FullGC unloaded, and this limit will be reset, the new limit Metaspace depends on the remaining capacity. If there is not enough space is freed, this limit will rise, and vice versa. Technically Metaspace size can grow to swap space, but this time the local memory allocation will fail (more specific analysis, refer to: Java PermGen Where to go?).

 

You can modify various startup parameters to "quick fix" these memory overflow error, but you need to correctly distinguish between whether you just put off or hide the symptoms of java.lang.OutOfMemoryError. If your application does have a memory leak or already loaded with some of the unreasonable category, then all these configurations are only postpone the problem only, and will not actually improve anything.

 

5、java.lang.OutOfMemoryError:Unable to create new native thread

A method of thinking thread is a thread saw that it was mission worker, if you have only one worker, then he can only perform one task, but if you have more than a dozen workers, you can complete several tasks simultaneously. Like these workers are in the physical world, JVM threads in the completion of their work is also needed some space when there are enough threads is not so much space will be like this:

wKiom1lutQ_AMchIAAAnxGuPTXs902.png-wh_50

 

java.lang.OutOfMemoryError appear: Unable to create new native thread means that Java applications can be started has reached its limit the number of threads.

 

Cause Analysis

It will throw Unable to create new native thread error when JVM OS request to create a new thread, and the OS can not create new native thread. A number of threads that can be created depends on the server configuration and the physical platform, it is recommended to run the sample code below to test to find out these limits. Overall, throw this error will go through the following stages:

 

  • Applications running within the JVM request to create a new thread

  • JVM OS request to create a new native thread

  • OS try to create a new native thread, then the need to allocate memory for new thread

  • OS refused to allocate memory to the thread, because 32-bit Java process has run out of memory address space (2-4GB memory address has been hit) or OS virtual memory has been completely exhausted

  • Unable to create new native thread error will be thrown

 

Examples

The following example can not create and start a new thread. When the code is running, and soon reached the limit number of threads of the OS, and throw Unable to create new native thread error.

while(true){
    new Thread(new Runnable(){
        public void run() {
            try {
                Thread.sleep(10000000);
            } catch(InterruptedException e) { }        
        }    
    }).start();
}

 

solution

Sometimes, you can limit by increasing the number of threads to bypass this error in the OS level. If you limit the number of threads in the JVM that can be created in user space, then you can check and increase this limit:

# macOS 10.12上执行
$ ulimit -u
709

When your application generates thousands of threads, and this exception is thrown, that your program there have been very serious programming error, I do not think we should solve this problem by modifying the parameters, regardless of OS-level parameters or JVM startup parameters. More desirable approach is to analyze whether your application really need to create so many threads to complete the task? Whether the number or you can use the thread pool thread pool is appropriate? Can more reasonable to split the business to achieve .....

 

6、java.lang.OutOfMemoryError:Out of swap space?

Java applications will need to specify the size of memory at startup, can be specified by -Xmx and other similar startup parameters. In the case where the total memory request JVM greater than the available physical memory, the operating system will be in a data memory is swapped to disk up.

wKioL1lutkrh2APcAAALQgUMoes375.png-wh_50

 

Out of swap space? Indicates a swap space will be exhausted, and due to the lack of physical memory and swap space, again attempts to allocate memory would fail.

 

Cause Analysis

When an application requests to allocate memory failed JVM native heap and native heap is also running out, JVM will throw an Out of swap space error. The error message contains the failed allocation size (in bytes) and the reason for the request failed.

 

Native Heap Memory Memory internal JVM is used, this part of the Memory can JNI way through the JDK to visit this part of Memory efficiency is high, but the management needs to do their own, not sure if it is best not to use, to prevent memory leaks. Native Heap Memory JVM uses to optimize the code is loaded (the JTI code generation), temporary object space applications, as well as some internal operation JVM.

When this problem often occurs in the case of Java process has begun exchange, modern GC algorithms have done good enough, then when faced with problems due to the delay caused by exchange, GC pause time tend to make most applications can not be tolerated.

 

java.lang.OutOfMemoryError:? Out of swap space is often a problem caused by the operating system level, such as:

 

  • Insufficient swap space operating system configuration.

  • Another process consumes all memory on the system resources.

 

There may be local memory leak causes the application to fail, such as: application calls a native code continuous allocate memory, but has not been released.

 

solution

There are several ways to solve this problem is usually the easiest way is to increase the swap space, way different platform will be different, for example, can be achieved through the following command in Linux:

# 原作者使用,由于我手里并没有Linux环境,所以并未测试
# 创建并附加一个大小为640MB的新交换文件
swapoff -a 
dd if=/dev/zero of=swapfile bs=1024 count=655360
mkswap swapfile
swapon swapfile

Java GC scans the data in memory, if it is running on swap space garbage collection algorithm GC pause time will increase several orders of magnitude, so you should seriously consider ways to increase the use of swap space above.

 

If your application is deployed on the JVM fierce competition with other processes need access to resources of the physical machine, it is recommended to isolate the service to a separate virtual machine

 

But in many cases, you only really viable alternative is:

 

  • Upgrade the machine to include more memory

  • Optimize the application in order to reduce its memory footprint

 

When you turn the optimal path, using a memory dump analysis program to detect large memory allocation is a good start.

 

7、java.lang.OutOfMemoryError:Requested array size exceeds VM limit

The maximum array size Java application can allocate is limited. Different platforms limit varies, but is usually between 1-2100000000 elements.

wKioL1lut3bT9r3JAAAOi6T-zDE446.png-wh_50

当你遇到Requested array size exceeds VM limit错误时,意味着你的应用程序试图分配大于Java虚拟机可以支持的数组。

 

原因分析

该错误由JVM中的native code抛出。 JVM在为数组分配内存之前,会执行特定于平台的检查:分配的数据结构是否在此平台中是可寻址的。

 

你很少见到这个错误是因为Java数组的索引是int类型。 Java中的最大正整数为2 ^ 31 - 1 = 2,147,483,647。 并且平台特定的限制可以非常接近这个数字,例如:我的环境上(64位macOS,运行Jdk1.8)可以初始化数组的长度高达2,147,483,645(Integer.MAX_VALUE-2)。如果再将数组的长度增加1到Integer.MAX_VALUE-1会导致熟悉的OutOfMemoryError:

Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit

但是,在使用OpenJDK 6的32位Linux上,在分配具有大约11亿个元素的数组时,您将遇到Requested array size exceeded VM limit的错误。 要理解你的特定环境的限制,运行下文中描述的小测试程序。

 

示例

for (int i = 3; i >= 0; i--) {
    try {
        int[] arr = new int[Integer.MAX_VALUE-i];
        System.out.format("Successfully initialized an array with %,d elements.\n", Integer.MAX_VALUE-i);
    } catch (Throwable t) {
        t.printStackTrace();
    }
}

该示例重复四次,并在每个回合中初始化一个长原语数组。 该程序尝试初始化的数组的大小在每次迭代时增加1,最终达到Integer.MAX_VALUE。 现在,当使用Hotspot 7在64位Mac OS X上启动代码片段时,应该得到类似于以下内容的输出:

java.lang.OutOfMemoryError: Java heap space
    at eu.plumbr.demo.ArraySize.main(ArraySize.java:8)
java.lang.OutOfMemoryError: Java heap space
    at eu.plumbr.demo.ArraySize.main(ArraySize.java:8)
java.lang.OutOfMemoryError: Requested array size exceeds VM limit
    at eu.plumbr.demo.ArraySize.main(ArraySize.java:8)
java.lang.OutOfMemoryError: Requested array size exceeds VM limit
    at eu.plumbr.demo.ArraySize.main(ArraySize.java:8)

注意,在出现Requested array size exceeded VM limit之前,出现了更熟悉的java.lang.OutOfMemoryError: Java heap space。 这是因为初始化2 ^ 31-1个元素的数组需要腾出8G的内存空间,大于JVM使用的默认值。

 

解决方案

java.lang.OutOfMemoryError:Requested array size exceeds VM limit可能会在以下任一情况下出现:

  • 数组增长太大,最终大小在平台限制和Integer.MAX_INT之间

  • 你有意分配大于2 ^ 31-1个元素的数组

 

在第一种情况下,检查你的代码库,看看你是否真的需要这么大的数组。也许你可以减少数组的大小,或者将数组分成更小的数据块,然后分批处理数据。

 

在第二种情况下,记住Java数组是由int索引的。因此,当在平台中使用标准数据结构时,数组不能超过2 ^ 31-1个元素。事实上,在编译时就会出错:error:integer number too large。

 

8、Out of memory:Kill process or sacrifice child

为了理解这个错误,我们需要补充一点操作系统的基础知识。操作系统是建立在进程的概念之上,这些进程在内核中作业,其中有一个非常特殊的进程,名叫“内存杀手(Out of memory killer)”。当内核检测到系统内存不足时,OOM killer被激活,然后选择一个进程杀掉。哪一个进程这么倒霉呢?选择的算法和想法都很朴实:谁占用内存最多,谁就被干掉。如果你对OOM Killer感兴趣的话,建议你阅读参考资料2中的文章。

wKiom1luuFOgWVxPAAAKcpUETe0020.png-wh_50

当可用虚拟虚拟内存(包括交换空间)消耗到让整个操作系统面临风险时,就会产生Out of memory:Kill process or sacrifice child错误。在这种情况下,OOM Killer会选择“流氓进程”并杀死它。

 

原因分析

默认情况下,Linux内核允许进程请求比系统中可用内存更多的内存,但大多数进程实际上并没有使用完他们所分配的内存。这就跟现实生活中的宽带运营商类似,他们向所有消费者出售一个100M的带宽,远远超过用户实际使用的带宽,一个10G的链路可以非常轻松的服务100个(10G/100M)用户,但实际上宽带运行商往往会把10G链路用于服务150人或者更多,以便让链路的利用率更高,毕竟空闲在那儿也没什么意义。

 

Linux内核采用的机制跟宽带运营商差不多,一般情况下都没有问题,但当大多数应用程序都消耗完自己的内存时,麻烦就来了,因为这些应用程序的内存需求加起来超出了物理内存(包括 swap)的容量,内核(OOM killer)必须杀掉一些进程才能腾出空间保障系统正常运行。就如同上面的例子中,如果150人都占用100M的带宽,那么总的带宽肯定超过了10G这条链路能承受的范围。

 

示例

当你在Linux上运行如下代码:

public static void main(String[] args){
    List<int[]> l = new java.util.ArrayList();
    for (int i = 10000; i < 100000; i++) {
        try {
            l.add(new int[100000000]);
        } catch (Throwable t) {
            t.printStackTrace();
        }
    }
}

在Linux的系统日志中/var/log/kern.log会出现以下日志:

Jun  4 07:41:59 plumbr kernel: [70667120.897649] Out of memory: Kill process 29957 (java) score 366 or sacrifice child
Jun  4 07:41:59 plumbr kernel: [70667120.897701] Killed process 29957 (java) total-vm:2532680kB, anon-rss:1416508kB, file-rss:0kB

注意:你可能需要调整交换文件和堆大小,否则你将很快见到熟悉的Java heap space异常。在原作者的测试用例中,使用-Xmx2g指定的2g堆,并具有以下交换配置:

# 注意:原作者使用,由于我手里并没有Linux环境,所以并未测试
swapoff -a 
dd if=/dev/zero of=swapfile bs=1024 count=655360
mkswap swapfile
swapon swapfile

解决方案

解决这个问题最有效也是最直接的方法就是升级内存,其他方法诸如:调整OOM Killer配置、水平扩展应用,将内存的负载分摊到若干小实例上..... 我们不建议的做法是增加交换空间,具体原因已经在前文说过。参考资料②中详细的介绍了怎样微调OOM Killer配置以及OOM Killer选择进程算法的实现,建议你参考阅读。

 

参考资料:

① 想要了解更多PermGen与Metaspace的内容推荐你阅读:

Java 8会解决PermGen OutOfMemoryError问题吗?

Java PermGen 去哪里了?

② 如果你对OOM Killer感兴趣的话,强烈建议你阅读这篇文章:

理解和配置 Linux 下的 OOM Killer

 

原文:https://blog.51cto.com/luecsc/1948800

发布了740 篇原创文章 · 获赞 65 · 访问量 10万+

Guess you like

Origin blog.csdn.net/qq_41723615/article/details/104348672