LINUX host JAVA application occupies CPU, memory is too high analysis method

Reprinted from: http://dbaplus.cn/news-21-130-1.html

1. The problem of high resource usage by the application

At present, most of the applications are developed in JAVA language. After the product is launched and used for a period of time, the CPU occupied by a JAVA program often occurs, the memory is too high, and it is almost never released, causing the system to freeze and users to slow down. , if you want to recover, you must kill the process or restart the service, and when you do this, you must cause a business interruption.

The program is mainly composed of code, and optimization needs to know which piece of code occupies resources, and an application occupies a high CPU. In addition to the fact that it is indeed a computationally intensive application, the usual reason is an infinite loop, so optimize the code to reduce It is essential to reduce the resource consumption of the application or reduce the infinite loop during the use of the application.

Below we take the character gateway server of the 4A platform as an example to conduct the corresponding analysis.

2. Problem Analysis

2.1. Analysis of high CPU

1) Use the TOP command to view the CPU and memory usage status. It can be found that the CPU usage is mainly divided into two parts, one is the percentage of CPU occupied by system kernel space, and the other is the percentage of CPU occupied by user space. The id indicated in the CPU status is the percentage of idle CPU. The lower the idle CPU percentage, the higher the CPU usage.

2) Preliminary analysis shows that the process that mainly occupies the CPU is the java subprocess jerrySsh service (the monitoring service used by users to access resources). In the case of a small number of users, the CPU consumes a lot of resources. According to the feedback from the research and development, the maximum number of visits set by the character gateway can reach 500/unit, and the current resource usage status of the character gateway cannot meet the set requirements.

● Analysis tools

At present, the analysis method for the high CPU usage of the java process under Linux is mainly to use the linux command to find out the process with high CPU usage. Before analyzing whether it is due to the process or the system, after analyzing the process that consumes too much CPU, the CPU usage is listed. High and longest occupied threads and use the jstack tool that comes with jdk to analyze CPU usage analysis:

export JAVA_HOME=/usr/apps/java/jdk1.6.0_20/

export PATH=$JAVA_HOME/bin:$PATH

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

分析过程：

根据top命令，发现PID为13033的Java进程占用CPU %id 50%以上，占用CPU过高
找到该进程后，首先显示线程列表,并按照CPU占用高的线程排序：
[root@YZ-A-ZFWG-4 ~]# ps -mp 13033 -o THREAD,tid,time | sort –rn

显示结果如下：

找到了耗时最高的线程28358，占用CPU时间达8分多钟。将需要的线程ID转换为16进制格式：

[root@YZ-A-ZFWG-4 ~]# printf "%x\n" 28358

6ec6

最后打印线程的堆栈信息：

经比对发现占用CPU高的jerrySsh服务中高消耗CPU的代码均为一些等待和读取的语句。内核时间占用最长的线程所使用的代码抓取：

2.2. 内存使用分析

目前字符网关内存使用趋于平衡，除偶尔出现close_wait连接后由于未能得到及时释放而占用了大量内存导致buffers/cache较小外，其他线条暂未出现问题，据研发反馈已经做过优化，但是从目前观察来看coles_wait连接释放时间稍长。并且由于buffers、cached释放不出来，导致系统剩余物理内存较小，可能会影响系统性能，为了彻底解决此类问题，所以我们做了以下分析：

在linux的内存分配机制中，系统优先使用物理内存，当物理内存还有空闲，表示还够用时，不会释放其占用内存，即使占用内存的程序已经被关闭了，该程序所占用的内存用来做缓存使用，对于开启过的程序、或是读取刚存取过得数据会比较快，因此查看目前进程正在实际被使用的内存（used-buffers-cache），也可以认为如果交换分区（swap）没有大量使用，物理内存（mem）还是够用的，只有物理内存（mem）被当前进程实际占用完（没有了buffers和cache），才会使用到交换分区（swap）。

但是从代码的角度，目前研发人员主要关注java.lang.OutOfMemoryError: Java heap space异常，减少不必要的对象创建，同时避免内存泄漏，所以分析代码才是我们接下来要做的主要工作；以下为字符网关分析内存占用的故障排查过程：

● 分析手段

top命令：Linux命令。可以查看实时的内存使用情况。
jmap -histo:live [pid]，然后分析具体的对象数目和占用内存大小，从而定位代码。
jmap -dump:live,format=b,file=xxx.xxx [pid]，然后利用MAT工具分析是否存在内存泄漏。

Java提供了一个很好的内存监控工具：jmap命令

jmap命令有下面几种常用的用法：

从上述打印的日志可以得知该进程调系统进程占用内存的主要程序。

使用./jmap -histo:live 14978查询当前 Java进程创建的活跃对象数目和占用内存大小。

可以日志中发现constMethodKlass、methodKlass、symbolKlass都占用了大量的内存，特别是占用了大量内存的int数组，需要仔细检查相关代码，接下来这些事就可以丢给研发了。

3. 总结分析手段

● 分析CPU占用的方法和手段：

top命令：可以查看实时的CPU使用情况。
ps -ef命令：可以查看进程以及进程中线程的当前CPU使用情况以及属于当前状态的采样数据。
jstack：Java提供的命令。可以查看某个进程的当前线程栈运行情况。根据这个命令的输出可以定位某个进程的所有线程的当前运行状态、运行代码，以及是否死锁等等。
pstack：Linux命令。可以查看某个进程的当前线程栈运行情况

● 分析内存性能的方法和技巧：

top命令：可以查看实时的内存使用情况。
jmap -histo:live [pid]，然后分析具体的对象数目和占用内存大小，从而定位代码。
jmap -dump:live,format=b,file=xxx.xxx [pid]，然后利用MAT工具分析是否存在内存泄漏等等。

LINUX host JAVA application occupies CPU, memory is too high analysis method

Guess you like