Diagnose Docker container application business problems online through Arthas

Arthas

https://arthas.aliyun.com/

Java application diagnostic tool

Introduction

Arthas is an online monitoring and diagnosis product. It can view the status information of application load, memory, gc, and thread in real time from a global perspective, and can diagnose business problems without modifying the application code, including checking the input and output parameters of method calls, exceptions, monitoring method execution time, class loading information, etc., greatly improving the efficiency of online troubleshooting.

background

Typically, the local development environment cannot access the production environment. If you encounter problems in a production environment, you cannot use IDE remote debugging. Worse, debugging is unacceptable in a production environment, because it will suspend all threads, causing the service to pause.

Developers can try to reproduce the problems in the production environment in the test environment or the pre-release environment. However, some issues cannot be easily reproduced in a different environment, and even disappear after a reboot.

If you're thinking of adding some logging to your code to help troubleshoot issues, you're going to have to go through the following phases: testing, staging, then production. This approach is inefficient, and worse, the problem may not be fixed since it may not be reproducible once the JVM is restarted, as described above.

Arthas aims to solve these problems. Developers can troubleshoot production issues online. No JVM restarts, no code changes. Arthas as an observer never suspends a running thread.

What can Arthas do for you?

When you encounter the following similar problems and feel helpless, Arthas can help you solve them:

1. Which jar package is this class loaded from? Why are various types of related Exception reported?
2. Why is the code I changed not executed? Could it be that I didn't commit? Branching wrong?
3. If you encounter a problem and cannot debug online, can you only republish it by adding a log?
4. I encountered a problem with the data processing of a certain user online, but it also cannot be debugged online, and cannot be reproduced offline!
5. Is there a global view into the health of the system?
6. Is there any way to monitor the real-time running status of the JVM?
7. How to quickly locate application hotspots and generate flame graphs?
8. How to find an instance of a class directly from within the JVM?

Install jar in docker container instance

Enter the application container instance

// 查看应用容器实例id
$ docker ps | grep insurance
b4126d9b72a6        hub.uban360.com/basic-x86/jar:202112201122              "sh run-java.sh"         2 hours ago         Up 2 hours                              access_insurance-content_1

// 通过命令行进入应用容器实例
$ docker exec -it b4126d9b72a6 bash

Install Arthas

Quickly install Arthas

// 1.解压arthas集合归档包
unzip ~/arthas-packaging-3.5.0-bin.zip

// 2.下载arthas-boot.jar
curl -O https://arthas.aliyun.com/arthas-boot.jar
java -jar arthas-boot.jar

List of Arthas commands

List of Arthas commands

basic command

• help - view command help information

jvm-related

• dashboard - real-time data panel of the current system
• heapdump - dump java heap, heap dump function similar to jmap command
• jvm - view information about the current JVM
• memory - view JVM memory information
• thread - view the thread stack information of the current JVM
• sysenv - View JVM environment variables
• sysprop - View and modify system properties of the JVM
• getstatic - view static properties of a class
• ognl - execute ognl expressions
• vmtool - query objects from jvm, execute forceGc

class/classloader related

• classloader - view classloader inheritance tree, urls, class loading information, use classloader to getResource
• jad - decompile the source code of the specified loaded class
• sc - View the class information loaded by the JVM
• sm - view the method information of the loaded class

monitor/watch/trace related

Note: Please note that these commands are all implemented through bytecode enhancement technology, and some aspects will be inserted into the methods of the specified class to realize data statistics and observation. Therefore, when using online or in advance, please try to clarify the classes, methods, and conditions that need to be observed, and execute stopor execute the enhanced classes after the diagnosis is completed reset.

• monitor - method execution monitor
• stack - output the call path where the current method is called
• trace - the internal call path of the method, and output the time spent on each node on the method path
• tt - The space-time tunnel of method execution data, which records the input parameters and return information of each call of the specified method, and can observe these calls at different times
• watch - method execution data observation

profiler/flame graph

• profiler - Use async-profiler to sample applications and generate flame graphs
• jfr - Dynamically enable and disable JFR recording

configuration options

• options - View or set Arthas global options

example

// 查看方法调用的出入参、异常
watch java.lang.String toString '{params, returnObj}' -x 2