1. Background
Due to work requirements, the developer (me) needs to perform high concurrency stress tests on some interfaces developed by myself. And solve the performance problems according to the stress test.
There are many problems that fail the pressure test, and there are many optimization points. This article only discusses the points that the JVM can optimize.
This article mainly records the ideas for solving the problem and the methods used. The solutions given cannot be used as a reference for any other problems.
2. Pressure measurement indicators
Stress testing with JMeter. The pressure measurement indicators are:
- Number of concurrent threads: 400
- Think time: 0 seconds
- Step length: 5 seconds
- Concurrent time: 60min
How to use JMeter's concept of stress testing and stress testing indicators, you can read another article I wrote, or Baidu by yourself.
3. Preliminary knowledge preparation
Eclipse MemoryAnalyzer
This is a Java memory analyzer under Eclipse. You can Baidu on the Internet and download it yourself.
Grab Dump file
The concept of Dump file is as shown in the figure below.
How to grab the memory image? You can use the following command
jmap -dump:format=b,file=文件名
When to grab the dump file?
The jmap command grabs the current memory snapshot. For example, after a stress test, directly press the corresponding Docker container to OOM. At this time, do not stop the stress test, and then immediately execute the jmap command to capture the memory snapshot at this moment. The accident scene can be maintained.
Fourth, locate the problem and solve it
First start the high-intensity stress test in 二、压测指标this section. During the stress test, we first used the jpscommand in the Docker container to view the Java process:
Then use the jstat -gcutil 117 1000command, 117 is the process number, 1000 is ms, which is printed every second.
The jstat command is a real-time command-line monitoring of the resources and performance of Java applications, including monitoring of Heap size and garbage collection status.
The results of real-time monitoring using jstat are as follows (the unit in the figure is a percentage):
In the statistical chart in the above figure, we only need to care about the O area (Old old age, the unit is percentage), and FGC (Full GC, the unit is the number of times).
The result shown in the above figure is: after constant stress testing, the memory of the old generation has been high, and full GC is frequent, and each GC does not release the memory in the O area. This indicates that the reference is not released, possibly caused by a memory leak problem.
Learn about memory leaks and memory overflows:
Further analysis with MemoryAnalyzer
Now it seems that the container has been crushed by us. At this time, do not stop the pressure measurement, but grab the Dump file. Import the dump file into the MemoryAnalyzer tool for analysis.
After the import is complete, use the leak suspects function of MemoryAnalyzer to analyze the charts that may be leaked:
Click details to view details:
See all its references:
Get the following image:
Here we can see two metrics: shallow heap and Retained Heap
shallow heap 和 Retained Heap
An object's Shallow heap is its own size in memory .
Retained heap refers to the amount of memory that will be freed when a particular object is garbage collected .
Therefore, according to the above figure, if the garbage collection releases the queue3 reference, it will be able to release a lot of space in the O area. And the analysis tool has been precise to the class. Just follow the map to find this class and this reference.
Solve the problem
According to experience, ArrayBlockingQueue: In java multi-threaded operation, BlockingQueue, especially some multi-threads inside jdk, use blockingQueue to do it. The description is that there is a problem with a thread pool.
Look at queue3 on the reference chain, and locate ExecutorServiceUtilthis class. Then we go to the code to find this class, and the queue3 object:
It means that queue3 is called a lot, the thread pool thread is full, and the existing thread has not been executed, resulting in the memory being occupied all the time. Since the thread has not been executed, it cannot be released, so the memory in the old age accumulates more and more. Finally OOM.
The solution is imminent: increase the number of connections in the thread pool, or directly expand the Docker instance. Both methods can be solved.
postscript
The performance problems caused by this high concurrency can be analyzed with the help of MemoryAnalyzer, but there are still some problems that cannot be analyzed with the help of JVM. It is possible that the network call timed out, or there may be front-end issues. These are all possible. The topic of performance optimization is relatively large, and I only know a little bit of fur, so I will remember this.
Later, I will update the trace command of the Arthas tool to view the execution time, trace layer by layer, locate and solve performance problems under high concurrency, as a work order and record.