How to troubleshoot user mode CPU usage rate

• View CPU usage
  in Linux systems, the top command to view CPU usage.

  %Cpu(s):  0.3 us,  0.1 sy,  0.0 ni, 99.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

  
  us (user): means that the CPU is running in user mode percentage of time, usually expressed user mode applications high CPU busy. A typical user programs are: databases, Web servers and so on.
  sy (sys): means that the CPU is running in kernel mode percentage of the time (not including interrupts), generally the lower the better CPU kernel mode, or that there are some bottlenecks in the system.
  ni (nice): represents the CPU time with nice correction process priority user mode process execution. nice is a process priority correction value, if the process by which modify the priority, CPU overhead will count separately.
  id (idle): represents the proportion of time the CPU is in an idle state, this time, CPU performs a specific virtual process, named System Idle Process.
  wa (iowait): indicates the CPU time waiting for I / O operations takes to complete, typically the index as low as possible, otherwise indicate a bottleneck in I / O, can be further analyzed by iostat commands.
  hi (hardirq): means that the CPU hardware interrupt processing time it takes. Hardware interrupt is issued by peripheral hardware (e.g., keyboard controller, sensor hardware and the like), the need for participation interrupt controller, characterized by fast execution.
  si (softirq): represents the soft interrupt CPU processing time spent. Soft interrupt is an interrupt signal, performs delay characteristic is issued by a software program (e.g., a network transceiver, a timing schedule, etc.).
  st (steal): indicates that the CPU is occupied by other virtual machines time, it appears only in multiple virtual machines scene. If the index is too high, the host or another virtual machine can check whether an exception under.

• Troubleshoot user mode CPU usage rate
  user mode CPU usage reflects how busy the application is usually closely related to our own writing code.
  Steps:
  1), by the top command to find the most CPU-intensive process ID;
  
  2), found consuming the most CPU thread number (column name still PID) by top -Hp process ID command;
  
  3), by printf "% x \ n "number of threads the thread number command output corresponding 16 hexadecimal digits;
  
  4), PID process in the case of kernel calls. If it is a Java application by jstack process ID | grep 16 -A 10 hex thread number command to find consume the most CPU thread stack method.
  Non-Java applications can use the perf

  perf top -p 7574

  If prompted perf: command not found, use yum install perf installation.
  
  perf tool is a Linux 2.6+ kernel, the position of the tools in the kernel packages / perf.
  perf trace characteristic use of Linux, it can be used for real-time tracking, event counting statistics (perf stat); or using sampling (perf record), report (perf report | script | annotate) the use for diagnosis.