Simple fault injection using Arthas

table of Contents

1. Concept

Arthas is an online monitoring and diagnosis product that can view application load, memory, gc, and thread status information in real time from a global perspective, and can diagnose business problems without modifying the application code, including viewing method calls. Parameters, exceptions, monitoring method execution time-consuming, class loading information, etc., greatly improve the efficiency of online troubleshooting.

Second, the overall logic

The overall logic of Arthas is also implemented on the Instrumentationbasis of Java . All the loaded classes will be loaded by the Agent. addTransformerAfter passing , they will be enhanced, and then the corresponding will be Advicewoven into it. For class search and method search, all are passed through SearchUtil. carried out by Instrumentthe loadAllClassmethod all the JVM loaded class match by name, the same will be returned.

Three, related documents

Quick start

Advanced use

Four, fault injection practice

On the premise of ensuring the stability of its own service, in order to simulate the abnormal situation of remote invoking service, the method of Arthas hot update online service code is used to realize fault injection.

1. Download Arthas to the server

curl -o /tmp/arthas-boot.jar https://arthas.aliyun.com/arthas-boot.jar

2. Get the process number of the service

ps -ef | grep 服务名

3. Use Arthas atach to the service

/home/work/bin/服务名/java/bin/java -jar /tmp/arthas-boot.jar

4. Enter the number before the service process number and press Enter to enter the arthas operation interface

5. In the Arthas interface, use the sc command and the -d parameter to specify the fully qualified name to find the related information of the Demo class, and get the Hash value corresponding to the ClassLoader of the loaded Demo class

sc -d com.***.Demo | grep classLoaderHash | head -n1

6. Compile the Demo.java file. You can upload the modified code in advance, use the mc command in the Arthas interface to compile the Demo class, and use the -c parameter to specify the hash value of the ClassLoader obtained in the previous step (-d parameter Output directory for compiled files)

$ mc -c hash值 /tmp/服务名/Demo.java -d /tmp/服务名

7. After getting the compiled Demo.class file (there may be an anonymous class Demo$1.class file, if the anonymous class logic is not modified, no hot update is required), you can use the retrasform command to hot update the code (the class file path is /Tmp/service name/com/***/Demo.class output in the previous step).

$ retransform /tmp/服务名/com/***/Demo.class

8. Check how many hot-loaded and updated classes there are. The same one is the id added. After deletion, it is executed according to the largest id, which is the most recent replacement.

retransform -l

9. After the practice is over, use the retransform command to undo the update

retransform --deleteAll

10. Note: After canceling, it will not be restored by default. There is still a specially added log output, and jad is still the affected version. If you want to display the old version to take effect, you need to execute the following command to fully restore it.

retransform --classPattern com.***.Demo

11. Exit arthas

如果只是退出当前的连接，可以用quit或者exit命令。Attach到目标进程上的arthas还会继续运行，端口会保持开放，下次连接时可以直接连接上。

如果想完全退出arthas，可以执行stop命令。

12. Note: When you log out and re-enter the arthas operation interface through commands, if an arthas 3658 port occupancy error is reported

[ERROR] The telnet port 3658 is used by process 2822 instead of target process 6062, you will connect to an unexpected process.
[ERROR] 1. Try to restart arthas-boot, select process 2822, shutdown it first with running the 'stop' command.
[ERROR] 2. Or try to stop the existing arthas instance: java -jar arthas-client.jar 127.0.0.1 3658 -c "stop"
[ERROR] 3. Or try to use different telnet port, for example: java -jar arthas-boot.jar --telnet-port 9998 --http-port -1

13. According to the prompt, execute the command to enter normally. If you use arthas to test multiple services at the same time on a machine, you can use different telnet port numbers:

/home/work/bin/服务名/java/bin/java -jar /tmp/arthas-boot.jar --telnet-port 9998 --http-port -1