Project Introduction
Code introduction
You can take a look at this program. First, start a thread pool. The actual capacity and maximum capacity of this thread pool are set dead, which means that it can hold up to 50 threads to execute tasks at the same time. And the unbounded queue used by the queue.
Run on Linux service
java -cp JVMOTHERS-1.0-SNAPSHOT.jar -XX:+PrintGC -Xms200M -Xmx200M JVM调优/FullGCProblem
(Set the heap space small and throw out OOM in advance).
Excessive CPU usage troubleshooting actual combat
1. First find the process id that consumes high CPU through the top command.
The top command is one of our most commonly used commands under Linux. It can display information such as CPU usage, memory usage, and system load of the executing process in real time. The upper part shows the statistics of the system, and the lower part shows the usage statistics of the process. Some people may ask, why not use jps to view the process directly? It is because we first need to judge whether our program is causing the computer to slow down and become stuck.
Good guy, it turned out that this buddy almost couldn't keep the panties that my CPU grabbed.
2. Execute top -p 15110 to monitor the process separately
3. Enter H on the basis of step 2 (or directly enter the command top -Hp 2732) to obtain all thread information under the process
4. Through observation, we found that threads with thread numbers 15112, 15113, 15114, and 15115 consume very much CPU. (It needs to run for a while)
We need to recall at this time that in our basic JDK command-line tuning tool, jstack can take a thread snapshot for us. In this snapshot, we may be able to analyze some information about the thread.
5. Execute jstatck 15110 , and save the thread information in this snapshot at this time
These nids are actually the thread numbers of 115110 under the process displayed by jstack. ox stands for hexadecimal. Therefore, you can find a conversion tool on Baidu to convert the thread number we got before to hexadecimal. Take thread 15112 as an example:
6. Positioning problem
The value in hexadecimal notation is 3b08. Let's check in the jstack log.
This is all clear. All GC threads grab my CPU resources. So we use the GC monitoring command jstat to count and view the GC situation.
7. Use jstat to monitor GC
It is a command line tool for monitoring various operating status information of virtual machines. It can display operating data such as class loading, memory, garbage collection, JIT compilation, etc. in the local or remote virtual machine process. On a server that does not have a GUI graphical interface and only provides a plain text console environment, it will be a virtual location during runtime. The tool of choice for machine performance problems.
Assuming that the garbage collection status of the process 15110 needs to be queried every 2.5 s, a total of 10 queries, the command should be: jstat-gc 15110 2500 10
We found that 7 or 8 times can be performed within 2.5 seconds.
OOM error also appeared in our program running interface.
S0C: The size of the first surviving area
S1C: The size of the second survival area
S0U: The used size of the first surviving area
S1U: The used size of the second survivor area
EC: The size of the Eden Park
EU: Use size of Eden Park
OC: Old age size
OU: size used in the old age
MC: Method area size
MU: Method area used size
CCSC: Compressed class space size
CCSU: Compressed class space usage size
YGC: The number of young generation garbage collections
YGCT: Young generation garbage collection consumes time
FGC: Number of garbage collections in the old age
FGCT: Garbage collection in the old age consumes time
GCT: Total time consumed by garbage collection
How to do? OOM is out.
We can see, what caused the high CPU usage in this one?
Is it a business thread? No, this is caused by the high occupancy of GC threads. The JVM is frantically performing garbage collection, and then reviewing the previous knowledge, the default garbage collector in the JVM is multi-threaded (review the previous knowledge), so multi-threading is recycling frantically, resulting in excessive CPU usage.
Solutions to high memory usage
Used to generate heap dump snapshots (generally called heapdump or dump files). The role of jmap is not only to obtain dump files, it can also query the finalize execution queue, Java heap and permanent generation detailed information, such as space usage, which collector is currently used, etc. Like the jinfo command, jump has many functions that are restricted under the Windows platform, except for the -dump option for generating dump files and viewing instances of each class,
The -histo option for space usage statistics is available on all operating systems
Print the first 20 objects
jmap –histo 15110 | head -20
You can see that there are more than 800,000 instances of objects corresponding to these classes in the figure. This is our problem
Analyze the reason from the code
Generally speaking, by seeing the first few lines, you can see which objects are occupying the memory.
Can these objects be recycled?
It can be seen that the recycling is very frequent and basically no effect. So why can't these objects be recovered after FullGC? Let's go back to our code.
We can see that the unbounded queue is used here
The thread pool can execute 50 tasks at the same time
But each time there are 100 tasks. In other words, 50 tasks will be in the thread pool, while the other 50 tasks can only be in the blocking queue. We dropped tasks to the thread pool at a rate of 100 pens/0.1 second (the first time we can still drop 50 tasks into the thread pool, and the second time all of them are directly dropped into the queue). The thread pool can only hold 50 tasks at the same time, and complete 1 task every 3s. Each task corresponds to its UserInfo object. These objects in the queue will not be recycled. Therefore, a heap memory that can grow almost indefinitely is created.
to sum up
When the JVM has performance problems (in terms of performance, the CPU is 100%, and the memory usage rate remains high)
- If the CPU is 100%, you can proceed from two perspectives. One possibility is that business threads are running frantically, such as an endless loop. There is another possibility that the GC thread is reclaiming frantically. Because the garbage collection of JVM is also multi-threaded, it is easy to cause 100% CPU
- When encountering the problem of memory overflow, we generally want to check which objects in the system occupy more. This case is a very simple code. In the actual business code, find the corresponding object, analyze the corresponding class, and find the reason why these objects cannot be recycled. From the perspective of our reachability analysis algorithm, the JVM memory area. There is also the basis of the garbage collector. Of course, if you encounter a more complicated situation, the theoretical basis to master is far more than that.
Common problem analysis
Oversized object
A lot of large objects are created in the code (for example, a database query of a super large list report), and they cannot be recycled because they are referenced. These large objects will enter the old age, causing the memory to be occupied all the time, and it is easy to cause GC or even OOM
More than expected visits
Usually, the upstream system request traffic soars, which is common in all kinds of promotion/seckill activities. You can check whether there is a sharp peak in combination with business traffic indicators. For example, if a system needs 2 G of heap space during peak memory requirements, but the heap space setting is relatively small, resulting in insufficient memory, the JVM initiates frequent GC or even OOM.
Excessive use of Finalizer (usually rarely)
Excessive use of the finalizer (Finalizer), the object is not immediately GC, the Finalizer thread will compete with our main thread, but due to its lower priority, less CPU time obtained, so it will never catch up with the main thread At the same time, the program consumes all available resources, and finally throws OutOfMemoryError
Memory leak
The large object reference is not released, and the JVM cannot automatically reclaim it
Long-lived objects hold references to short-lived objects
For example, if ArrayList is set as a static variable, the objects in the container cannot be released before the end of the program, resulting in a memory leak
Connection not closed
Such as database connection, network connection and IO connection, etc., only after the connection is closed, the garbage collector will reclaim the corresponding object.
Unreasonable variable scope
For example, 1. The scope of the definition of a variable is greater than the scope of its use, 2. If the object is not set to null in time.
Inner class holds outer class
This way of creating non-static inner classes in Java will implicitly hold a reference to an external class, and this reference is a strong reference by default. Therefore, if the life cycle of the internal class is longer than the life cycle of the external class , the program is very It is easy to cause a memory leak (the garbage collector will recycle instances of the outer class, but because the inner class holds a reference to the outer class, the garbage collector cannot work properly)
Solution: You can display a soft reference (or weak reference) of an external class inside the inner class, and pass it in through the construction method. During the use of the inner class, first judge whether the outer class is recycled ；
Hash value changes
In the collection, if the fields in the object that participate in the calculation of the hash value are modified, the current object cannot be deleted from the collection separately, causing a memory leak
Classic case of memory leak
Code issue
There is a big relationship between code problems and memory leaks. If you observe a system, each time you perform FullGC, you will find that the proportion of heap space recovery is relatively small, especially in the old age, and there are more and more objects. At this time, you can judge that memory leaks may occur. . Memory overflow is not necessarily a code problem, but leaks must be.
Memory leak
After the program has applied for memory, it cannot release the applied memory space
Analysis of Memory Leak and Memory Overflow
Memory overflow: caused by a real lack of memory space;
Memory leak: The released object is not released, which is common when the container is used to store the element.
How to avoid:
Memory overflow: Check the code and set enough space
Memory leak: there must be a problem with the code
In many cases, memory overflow is often caused by memory leaks
Our general optimization ideas have an important sequence:
- Program optimization, the effect is usually very large
- Expansion, if the cost of money is relatively small, don't have trouble with yourself;
- Parameter tuning, find a balance between cost, throughput, and delay