Java deadlock and Java process Java CPU 100% troubleshooting

Three axes: top -> top -Hp ->jstack

Find the process number that consumes the most CPU through the top command;
Use the top -Hp process number command to find the thread number that consumes the most CPU (the column name is still PID);
Output the hexadecimal number corresponding to the thread number through the printf "%x\n" thread number command;
Use the jstack process number | grep hexadecimal thread number -A 10 command to find the thread method stack that consumes the most CPU.

1. Determine the Java application process number

Use the jps or ps -ef|grep java command to determine the process number of the application you want to analyze.

2. Use a deadlock detection tool to detect deadlock
2.1 Jstack command

jstack is a stack trace tool that comes with the java virtual machine. jstack is used to print out the given java process ID or core file or the Java stack information of the remote debugging service. The Jstack tool can be used to generate a thread snapshot of the Java virtual machine at the current moment. Thread snapshot is a collection of method stacks being executed by each thread in the current java virtual machine. The main purpose of generating thread snapshots is to locate the cause of long pauses in threads, such as deadlocks between threads, endless loops, and long periods caused by external resource requests. Time to wait and wait. When a thread is stalled, you can view the call stack of each thread through jstack, and you can know what the unresponsive thread is doing in the background, or what resources it is waiting for.

2.2 JConsole tool

Jconsole is a monitoring tool that comes with the JDK and can be found in the JDK/bin directory. It is used to connect to the running local or remote JVM, monitor the resource consumption and performance of running Java applications, draw a large number of charts, and provide a powerful visual interface. Moreover, the server memory occupied by itself is very small, and it can even be said that it hardly consumes.

2.3 Java Visual VM

Start VisualVM, select the corresponding JAVA application in the application window, check the thread life cycle status in the details window> thread tab (check thread visualization), and pay attention to the red part of the thread life cycle.

(1) Green: represents the running status. Generally it is normal. If it is a multi-threaded environment, in the producer-consumer mode, the consumer has been running, indicating that the consumer has low processing performance and cannot keep up with the pace of the producer. The corresponding code needs to be optimized. If it is not processed, it may cause the consumer The phenomenon of queue blocking. Corresponding to the [RUNNABLE] state of the thread.

(2) Blue: represents thread sleep. When the thread state of the Thread.sleep() function is called in the thread, it is blue. [TIMED_WAITING] state of the corresponding thread.

(3) Yellow: stands for thread waiting. The wait() function of the calling thread will appear yellow. Corresponding to the [WAITING] state of the thread.

(4) Red: The code thread is locked. Corresponding to the [BLOCKED] state of the thread.

After detecting the deadlock, we can dump the thread log to analyze the cause of the deadlock.

3. Analyze and solve the thread lock of JAVA application

There are many reasons for thread locks. I have encountered many situations where multiple threads access the same resource at the same time, and this resource uses the synchronized keyword, causing one thread to wait for another thread to run out of resources. For example, in the absence of a connection pool, access to the database interface at the same time, this situation will lead to a significant decline in performance, the solution is to increase the connection pool, or modify the access method, or refine the resource granularity, similar to the processing in ConCurrentHashMap In this way, the resource is divided into multiple smaller-grained resources, and locks are processed on the smaller-grained resources, which can solve the problem of intense resource competition.

Second, Java CPU 100%

For applications deployed in the Linux environment, sometimes for various reasons, the cpu occupies 100%. At this time, it is necessary to quickly analyze and locate the cause of cpu occupancy. Usually, through the top command of the linux system, you can see which process takes up too much cpu resources. But if it is found that it is a java process, then it is necessary to further analyze which thread in the java process has the problem.

1. Use the top command to see which process is occupying too much cpu resources

2. Determine the Java application (it can be omitted here, just use 32220 in the top of the previous step as pid1)

Use the jps or ps -ef|grep java command to determine the process number of the application you want to analyze [pid1]

3. View the CPU share of threads in Java applications

Use the top -Hp [pid1] command to view the thread cpu occupancy ratio of the specified process, analyze which thread occupancy rate is too high, and write down its process id (pid2), where [pid2] is the process determined by the first step Number, converted to hexadecimal.

Convert pid to hexadecimal system (32230->7de6):

printf “%x\n” [pid2]

4. Use to `jstack`view thread information

Select the number (PID) of the thread that accounts for a higher proportion, and convert the PID to hexadecimal, throughjstack [pid1] |grep -A 10 0x7de6

More often we need to dump the log for analysis and export the thread stack command:

 jstack -l 32220 > jstack-32220.log

Thread status in jstack Dump log file

1: In the dump file, there are thread states worth paying attention to (the part marked in red needs to be paid attention to)

Deadlock
In execution, Runnable
Waiting for resources, Waiting on condition
Waiting for the monitor, Waiting on monitor entry
Suspended
Object waiting, Object.wait() or TIMED_WAITING
Blocked, Blocked
Stop, Parked

2: Meaning and precautions of thread status in Dump file

Deadlock: Deadlock thread, generally refers to a situation in which multiple thread calls enter into mutual resource occupation, resulting in a situation where it has been waiting and cannot be released.
Runnable: Generally refers to the thread being executed, the thread occupies resources, is processing a request, may be passing SQL to the database for execution, may be operating on a certain file, may be performing data type conversion.
Waiting on condition: This state appears when the thread is waiting for the occurrence of a certain condition. The specific reason can be analyzed in combination with stacktrace. The most common situation is that the thread is waiting for the network to read and write. For example, when the network data is not ready for reading, the thread is in this waiting state. Once the data is ready for reading, the thread will reactivate, read and process the data. Before Java introduced NewIO, for each network connection, there was a corresponding thread to handle network read and write operations. Even if there is no readable and writable data, the thread is still blocked in read and write operations, which may cause waste of resources. It also brings pressure to the thread scheduling of the operating system. A new mechanism is adopted in NewIO, and the performance and scalability of the written server program have been improved. If you find that a large number of threads are in Wait on condition, from the thread stack, they are waiting for the network to read and write, this may be a sign of a network bottleneck. The thread cannot be executed because of network congestion. One situation is that the network is very busy, almost consuming all the bandwidth, and there is still a large amount of data waiting for the network to read and write; the other situation may also be that the network is idle, but due to routing problems, packets cannot arrive normally. Therefore, it is necessary to combine some performance observation tools of the system for comprehensive analysis, such as netstat statistics of the number of sent packets per unit time, if it obviously exceeds the limit of the network bandwidth; observe the utilization of the cpu, if the CPU time in the system state, relative to The proportion of CPU time in the user mode is relatively high; if the program is running on the Solaris 10 platform, you can use the dtrace tool to see the system call situation, if you observe the number of read/write system calls or the running time is far ahead; these all point to the network Network bottlenecks caused by bandwidth limitations. Another common situation where the Wait on condition occurs is that the thread is sleeping, and it will be awakened when the waiting time for sleep is up.
Locked: Thread blocking means that the required resources have been waiting for a long time but have not been obtained during the execution of the current thread. The thread manager of the container is identified as a blocked state, which can be understood as a thread waiting for resource timeout.
Waiting for monitor entry and in Object.wait(): Monitor is the main method used to achieve mutual exclusion and cooperation between threads in Java. It can be seen as an object or Class lock. Every object has, and only one monitor.

Garbage collection statistics

Capacity (Capacity) and usage (Used), size unit (KB), time unit (s)

S0C: the size of the first surviving area
S1C: the size of the second surviving area
S0U: the used size of the first surviving area
S1U: the used size of the second surviving area
EC: the size of the
Eden park EU: the use of the Eden park Size
OC: Old generation size
OU: Old generation size
MC: Method area size
MU: Method area size
CCSC: Compressed class space size
CCSU: Compressed class space used size
YGC: Number of young generation garbage collections
YGCT: Young generation garbage collection consumption Time
FGC: Number of garbage collections in the old age
FGCT: Garbage collection consumption time in the old age
GCT: Total garbage collection time consumption