How to solve the problem of Out Of Memory

background

When using Excel Importer to import data, when the amount of data exceeds 1w rows, an OutOfMemory error often occurs. (When exporting data with Excel Exporter, there will be similar problems).

The usual symptoms are as follows, that is, after importing several lines successfully,  java.lang.OutOfMemoryError: Java heap space an error occurs.

The following is the specific information of such an error for a customer:

1. The log problem is as follows (only the key part is intercepted):

   

2. This problem only occurs in the customer's test environment and does not appear in the local development environment.
3. The test environment is deployed in a private cloud and deployed with docker.
4. When the amount of imported data is more than 2W rows, this error always occurs around 1.2w after the import is successful. If the amount of data is less than 1w rows, this problem does not occur.

In this article, let's take a look at how to fix  java.lang.OutOfMemoryError: Java heap space the error.

What is Java Heap Size?

ava Heap Size is a configuration parameter of the Java Virtual Machine (JVM), which is used to specify the heap memory size when the JVM is running. Heap memory is a memory area used to store object instances and arrays in Java programs. During the execution of a Java program, all dynamically created objects are stored in the heap memory.

Java Heap Size is very important for the performance and stability of Java programs. If the heap memory is too small, it may cause an Out of Memory Error (Out of Memory Error), because the JVM cannot meet the memory required by the program to run. On the other hand, if the heap memory is too large, it will cause garbage collection to run frequently, affecting the response time and performance of the program.

By adjusting the Java Heap Size, memory usage can be optimized according to the needs of the application. Typically, a larger heap memory is beneficial for memory-intensive applications such as large-scale data processing, while smaller applications such as lightweight services may require a smaller heap memory.

In Java, the Java Heap Size can be set by the following parameters:

1. Xms: Specifies the initial heap memory size of the JVM.
2. Xmx: Specifies the maximum heap memory size of the JVM.

Example:

java -Xms256m -Xmx1024m MyApp 

The above command will start a Java application named "MyApp" and set the initial heap memory size to 256MB and the maximum heap memory size to 1024MB.

What are the usual reasons for this error?

Usually there are two reasons:

1. Java Heap Size is indeed set too small.
2. The application has a memory leak, resulting in a large number of objects that cannot be GC in a short period of time.

In this scenario, we first troubleshoot the first cause because:

1. First of all, this problem occurs inside Excel Importer. This module has been downloaded tens of thousands of times and has been maintained by the official for so many years. The probability of problems is very small. I'm not saying that the official components will definitely not have problems, but from the perspective of probabilistic troubleshooting, we first check the more fragile parts rather than the more stable parts, which is the best choice when other information is lacking.
2. Second, this problem occurs inside Excel Importer and we don't have the source code. Even if it is confirmed that it is a problem with Excel Importer, our first choice should be to avoid it from the outside. Only in this way can we ensure that the project is completed on time.

authentication problem

In order to verify whether the reason is that the Java Heap Size is set too small, I made two attempts:

1. Reduce the Java Heap Size in the local development environment Mendix Studio Pro to see if the error can be reproduced. The setting method is as follows:
2. In the local development environment, import an excel file with 10w rows to see if the error can be reproduced.

After the above experiments, I found that when I set the Java Heap Size to 512MB and import data to 2w pieces, the same error will pop up.

This phenomenon proves that the Java Heap Size is indeed too small.

How to fix the problem?

The repair idea is very simple, that is, to change the Java Heap Size.

In  the documentation of docker-mendix-buildpack  , we know that the environment variables of the container can be changed through subordinate methods:

We open the link in the above picture and enter   the document of cf- mendix-buildpack (docker-mendix-buildpack will call cf-mendix-buildpack at runtime), we know that Java Heap Size can be set as follows:

So, we start the container with the following command:

docker run -it \

-e HEAP_SIZE=4096M

...

However, we found that  HEAP_SIZE after only setting this environment variable, the problem is still not resolved.

Later, reading the code of cf-mendix-buildpack, I found:

1. If not set  HEAP_SIZE, is can be  limit automatically set according to.
2. And if it  HEAP_SIZE exceeds  limit, it is not acceptable.

So, I found  limit a way to set it, which is to set  MEMORY_LIMIT this environment variable.

So, we start the container with the following command:

docker run -it \ 
-e MEMORY_LIMIT=4096M 
... 

Finally solved the problem successfully!

Summarize

1. When analyzing the cause of a problem, examining the weaker parts first rather than the more robust ones is optimal when no other information is available.
2. By doing a small amount of experiments, you can be more certain about the root cause of the problem instead of relying on blind guesses.
3. Open source tools, look at the source code more, you will have a different harvest.

About Mendix

In a digital-first world, where customers expect their every need to be met, employees expect better tools to get their jobs done, and businesses realize they can only survive and succeed through total digital transformation. Mendix Corporation, a Siemens business is rapidly becoming an enabler of digital transformation for enterprises. Its industry-leading low-code platform and comprehensive ecosystem integrate the most advanced technologies to help enterprises create solutions that improve interactivity, simplify operations and overcome IT bottlenecks. With the four pillars of abstraction, automation, cloud and collaboration, Mendix has greatly improved the productivity of developers, and relied on its own engineering collaboration capabilities and intuitive visual interface to help a large number of "citizen" developers who are not familiar with technology in their places. Create applications in the field of expertise. Mendix is ​​a leader and visionary in the eyes of authoritative industry analysts, as well as a cloud-native, open, scalable, agile and proven platform. From artificial intelligence and augmented reality, to intelligent automation and native mobility, Mendix has become the backbone of digital-first businesses. Mendix's enterprise low-code platform has been adopted by more than 4,000 leading companies around the world.