Arthas troubleshoots JAVA application problems - the road to dream building

Introduction

Arthas is an open source Java application diagnostic tool developed by Alibaba that can troubleshoot problems online, dynamically track Java code, and monitor JVM status in real time.

Official website:arthas

Quick start

1. Download

wget https://arthas.aliyun.com/arthas-boot.jar

2. run

Need to run in an environment where the java program is running

java -jar arthas-boot.jar

The program will display all running Java processes in the system. Arthas assigns a serial number to each process.

[INFO] JAVA_HOME: C:\Program Files\Java\jdk1.8.0_351\jre
[INFO] arthas-boot version: 3.7.1
[INFO] Found existing java process, please choose one and input the serial number of the process, eg : 1. Then hit ENTER.
* [1]: 9400 .\target\demo-0.0.1-SNAPSHOT.jar
  [2]: 13964 org.eclipse.equinox.launcher_1.6.500.v20230717-2134.jar
  [3]: 6796 org.springframework.ide.vscode.boot.app.BootLanguageServerBootApp

Find the Java process with the problem from this list and enter the corresponding serial number. For example, here I enter 1 and press Enter. Arthas will automatically download the complete package and Attach it to the target process, the output is as follows:

[INFO] Start download arthas from remote server: https://arthas.aliyun.com/download/3.7.1?mirror=aliyun
[INFO] Download arthas success.
[INFO] arthas home: C:\Users\aneasystone\.arthas\lib\3.7.1\arthas
[INFO] Try to attach process 9400
[INFO] Attach process 9400 success.
[INFO] arthas-client connect 127.0.0.1 3658
  ,---.  ,------. ,--------.,--.  ,--.  ,---.   ,---.  
 /  O  \ |  .--. ''--.  .--'|  '--'  | /  O  \ '   .-' 
|  .-.  ||  '--'.'   |  |   |  .--.  ||  .-.  |`.  `-. 
|  | |  ||  |\  \    |  |   |  |  |  ||  | |  |.-'    |
`--' `--'`--' '--'   `--'   `--'  `--'`--' `--'`-----' 

wiki       https://arthas.aliyun.com/doc
tutorials  https://arthas.aliyun.com/doc/arthas-tutorials.html
version    3.7.1
main_class
pid        9400                                                                 
time       2023-12-06 07:16:31

[arthas@9400]$

3. View all commands

[arthas@9400]$ help
 NAME         DESCRIPTION
 help         Display Arthas Help
 auth         Authenticates the current session
 keymap       Display all the available keymap for the specified connection.
 sc           Search all the classes loaded by JVM
 sm           Search the method of classes loaded by JVM
 classloader  Show classloader info
 jad          Decompile class
 getstatic    Show the static field of a class
 monitor      Monitor method execution statistics, e.g. total/success/failure count, average rt, fail rate, etc.
 stack        Display the stack trace for the specified class and method
 thread       Display thread info, thread stack
 trace        Trace the execution time of specified method invocation.
 watch        Display the input/output parameter, return object, and thrown exception of specified method invocation
 tt           Time Tunnel
 jvm          Display the target JVM information
 memory       Display jvm memory info.
 perfcounter  Display the perf counter information.
 ognl         Execute ognl expression.
 mc           Memory compiler, compiles java files into bytecode and class files in memory.
 redefine     Redefine classes. @see Instrumentation#redefineClasses(ClassDefinition...)
 retransform  Retransform classes. @see Instrumentation#retransformClasses(Class...)
 dashboard    Overview of target jvm's thread, memory, gc, vm, tomcat info.
 dump         Dump class byte array from JVM
 heapdump     Heap dump
 options      View and change various Arthas options
 cls          Clear the screen
 reset        Reset all the enhanced classes
 version      Display Arthas version
 session      Display current session information
 sysprop      Display and change the system properties.
 sysenv       Display the system env.
 vmoption     Display, and update the vm diagnostic options.
 logger       Print logger info, and update the logger level
 history      Display command history
 cat          Concatenate and print files
 base64       Encode and decode using Base64 representation
 echo         write arguments to the standard output
 pwd          Return working directory name
 mbean        Display the mbean information
 grep         grep command for pipes.
 tee          tee command for pipes.
 profiler     Async Profiler. https://github.com/jvm-profiling-tools/async-profiler
 vmtool       jvm tool
 stop         Stop/Shutdown Arthas server and exit the console.
  • JVM related commands

  • Commands related to class loading, classes, methods

  • Statistics and observation commands

  • Linux-like commands

  • Other commands

 These are the main types of commands

 JVM related commands

These commands are mainly related to JVM and are used to view or modify JVM related properties, view JVM thread, memory, CPU, GC and other information:

  • jvm - View current JVM information;

  • sysenv - View JVM environment variables;

  • sysprop - View JVM system properties;

  • vmoption - View or modify JVM diagnostic-related parameters;

  • memory - View JVM memory information;

  • heapdump - Exports the heap snapshot of the Java process to a file to facilitate us to analyze the heap memory;

  • thread - View information about all threads, including thread name, thread group, priority, thread status, CPU usage, stack information, etc.;

  • dashboard - View the real-time data panel of the current system, including thread, memory, GC and Runtime information; it can be viewed as thread, memory, jvm, sysenv, sysprop A combination of several commands;

  • perfcounter - View the Perf Counter information of the current JVM;

  • logger - View application log information and support dynamically updating log levels;

  • mbean - View or monitor MBean information in real time;

  • vmtool - uses the JVMTI interface to implement functions such as querying memory objects and forcing GC;

Commands related to class loading, classes, methods

 These commands are mainly related to class loading, classes or methods, such as searching for classes or class methods in the JVM, viewing static attributes of classes, compiling or decompiling, hot updating classes, etc.:

  • classloader - View all Classloader information in the JVM;

  • dump - Export the specified class into .class bytecode file;

  • jad - Decompile the specified class into Java source code;

  • mc - memory compiler, compiles Java source code into .class bytecode files;

  • redefine / retransform - Both commands can hot update loaded classes, but redefine and jad / < a i=3> /  /  /  and other commands will conflict, and after  The original class cannot be restored, so it is recommended to use the  command. For the difference between the Redefine and Retransform mechanisms in the JDK, you can refer to here;watchtracemonitorttredefineretransform

  • sc - Search Class, search for classes in the JVM;

  • sm - Search Method, search for methods of classes in the JVM;

  • getstatic - View static properties of a class;

  • ognl - executes ognl expressions; ognl is very flexible and can implement many functions, such as viewing or modifying system properties above, and viewing static properties of classes can be implemented through ognl;

Statistics and Observations

 These commands can count and monitor the execution of class methods, and are a powerful tool for troubleshooting online problems:

  • monitor - monitor a given class method and count its call times, call time and success rate, etc.;

  • stack - View the execution call stack of a method;

  • trace - monitors a given class method and outputs the time it takes to call the method. The difference from monitor is that it can also track the calling link of the first-level method and Time-consuming, helping to quickly locate performance problems;

  • watch - observe the execution data of the specified method, including the method's input parameters, return value, thrown exceptions, etc.;

  • tt - Like the watch command, tt can also observe the execution data of the specified method, but tt will Each execution is recorded, and then each call is investigated and analyzed, so it is called Time Tunnel;

  • reset - The above commands related to statistical observation are implemented through bytecode enhancement technology and will be executed in the specified class. Insert some aspect code into the method, so after the production environment diagnosis is completed, remember to execute the reset command to reset the enhanced class (or execute the stop command); a>

  • profiler - Use async-profiler to sample the application and generate a flame graph from the sampling results;

  • jfr - Dynamically turn on and off JFR recording, and the generated jfr file can be analyzed through JDK Mission Control;

Comparison of Arthas commands and JDK tools 

Arthas command JDK tools Compared
sysprop jinfo -sysprops can view the system properties of the JVM, but sysprop is better than jinfo in that it can also modify the system properties
vmoption jinfo -flag can view JVM parameters, but vmoption only displays diagnostic-related parameters, such as HeapDumpOnOutOfMemoryError, PrintGC , etc.< /span>
memory jmap -heap can view the memory information of JVM, but memory is displayed in table form to facilitate user reading
heapdump jmap -dump You can export the heap memory of the process, but it is simpler to use.
thread jstack can list all threads of the JVM, but thread is displayed in table form, which is convenient for users to read, and it also adds the CPU usage function, which allows us to quickly find out the current maximum Busy Thread
perfcounter jcmd PerfCounter.print You can view the performance statistics of the JVM process
classloader jmap -clstats can view the JVM's Classloader statistics, but the classloader command also supports viewing in the form of a tree. In addition, it also supports viewing the actual URL of each Classloader, finding resources through the Classloader, etc.
jfr jcmd JFR.start You can turn on or off JFR recording and generate the jfr file

 Linux-like commands

In addition to the above commands for problem diagnosis, Arthas also provides some Linux-like commands for us to use in the Arthas terminal, such as:

  • base64 - performs base64 encoding and decoding;

  • cat - print file contents;

  • cls - Clear the current screen area;

  • echo - print parameters;

  • grep - Search text using strings or regular expressions and output matching lines;

  • history - output history commands;

  • pwd - output the current working directory;

  • tee - reads data from stdin and outputs to stdout and file simultaneously;

  • wc - Currently only supports wc -l, counting the number of output lines;

In addition, Arthas also supports running tasks in the background. Following the related commands in Linux, we can use & to run tasks in the background and use the jobs column Exit all background tasks, use Ctrl + Z to pause the task, use bg and fg to move the suspended tasks to the background or foreground to continue running, Use kill to terminate the task. For details, please refer to Arthas background asynchronous tasks.

 Other commands

There are also some commands related to Arthas itself, such as viewing Arthas version number, configuration, session and other information:

  • version - View Arthas version number;

  • options - View or modify Arthas global configuration;

  • keymap - View all currently bound shortcut keys, you can customize the shortcut keys through the ~/.arthas/conf/inputrc file;

  • session - View current session information;

  • auth - authenticate the current session;

  • quit - Exit the current Arthas client, other Arthas clients will not be affected;

  • stop - Close the Arthas server and exit all Arthas clients; this command will reset all enhanced classes (except redefine classes);

Troubleshooting ideas

 Use watch to monitor method incoming and outgoing parameters and exceptions

Use logger to dynamically update log levels

Use ognl to view system properties and application configurations

Use jad/sc/retransform hot update code

Other usage scenarios

The use of Arthas is very flexible, and sometimes it even has some unexpected functions. In addition to the above usage scenarios, Arthas's Issues also collected some user cases, several of which impressed me deeply and were very inspiring. , for reference.

  • Use stack command to locate the calling source of System.exit/System.gc :https://github.com/alibaba/arthas/issues/20

  • Use sc 和 jad Exclusion NoSuchMethodError Problem:https://github.com/alibaba/arthas/issues/160

  • Use redefine to modify StringBuilder.toString() to locate unknown log sources:https://github.com/alibaba/arthas/issues/263

  • Use trace javax.servlet.Servlet/Filter to troubleshoot Spring Boot application 404/401 issues:https://github.com/alibaba/arthas/issues/429

  • Use tt to locate high CPU load issues in Java applications:https://github.com/alibaba/arthas/issues/1202

  • Use profiler to do complex link analysis and troubleshoot performance issues:https://github.com/alibaba/arthas/issues/1416

  • Use trace command to optimize interface performance ten times:https://github.com/alibaba/arthas/issues/1892

References

 

  • Arthas documentation:https://arthas.aliyun.com/doc/

  • OGNL reference documentation:https://commons.apache.org/proper/commons-ognl/language-guide.html

  • Arthas Online Tutorial - Killercoda:https://killercoda.com/arthas/course/arthas-tutorials-cn

  • redefine VS. retransform:https://lsieun.github.io/java-agent/s01ch03/redefine-vs-retransform.html

  • Alibaba Arthas practice - get the Spring Context and then do whatever you want:https://github.com/alibaba/arthas/issues/482

  • Arthas user case:https://github.com/alibaba/arthas/issues?q=label%3Auser-case

  • Optimize microservice performance using SkyWalking & Arthas:https://github.com/alibaba/arthas/issues/1653

Use Arthas to troubleshoot online issues 

Guess you like

Origin blog.csdn.net/qq_34777982/article/details/135020967