Ali monitoring diagnostic tool Arthas Source Principle Analysis

Last month, Ali open source monitoring and diagnostic tool, " Arthas ", an online tool can be used for problem analysis, short term gains a lot of attention, even on Twitter Java Twitter official also forwarded, really praise .

GitHub is the readme:

Arthas is an online monitoring diagnostic products through a global perspective to view real-time status information about application load, memory, gc, thread, and can, without modifying the application code, diagnostic business issues, including access to view the method call ginseng, exception monitoring time-consuming method, class loading information, greatly enhance the online troubleshooting efficiency.

I usually see interesting open source tools, you will find a few functional sites of greatest interest cut, from source to understand the design and implementation principle. For some of their own understanding of the realization of ideas, and then verify the source code to see if the idea is to use the same implementation. If they want to achieve and the same, you might think, Aha, thought a few weeks. If the source is another implementation, you'll want to Cool, so you can also play. As if in dialogue as author and source of the same .

Taking advantage of the National Day holiday and read some " Arthas source", roughly summarized below.

From the source packet structure can be seen divided into several major modules:

Agent - VM loaded Custom Agent
Client - Telnet client implementation
Core - Arthas core implementation, includes a connection VM, resolve all kinds of commands, etc.
Site - Arthas help book site content

I looked at the following main features:

The connection process
Decompile class, get the source code
Query class specified load

The connection process

Connect to the specified process, follow-up monitoring and diagnosis of the foundation . Only first attach to the above process, in order to obtain information corresponding to the VM, query ClassLoader loaded classes and so on.

How to connect to the process?
Readers for similar diagnostic tool may have the impression, like JProfile, VisualVM and other tools, will let you choose a process to connect to. Then operate on the specified VM. For example, to view the partition information corresponding memory, memory garbage collection information, perform BTrace script and so on.

We should first think about it, these processes a list of available connections, is how to list it?
Generally it may be similar ps aux | grep java tool for this, or is using Java provided jps -lv can list contains the process id of the content. I wrote in a previous article, a little early jps's content ( you may not be aware of a few java gadgets ), behind its implementation, are all local Java processes will start to pid as the file name stored in the Java temporary directory. This list, traversing these files can get out.

Arthas is how to do it?
In the startup script as.sh , the code has on the process list below, also achieved by jps then exclude Jps out of their own:

# check pid
    if [ -z ${TARGET_PID} ] && [ ${BATCH_MODE} = false ]; then
        local IFS_backup=$IFS
        IFS=$'\n'
        CANDIDATES=($(${JAVA_HOME}/bin/jps -l | grep -v sun.tools.jps.Jps | awk '{print $0}'))

        if [ ${#CANDIDATES[@]} -eq 0 ]; then
            echo "Error: no available java process to attach."
            # recover IFS
            IFS=$IFS_backup
            return 1
        fi

        echo "Found existing java process, please choose one and hit RETURN."

        index=0
        suggest=1
        # auto select tomcat/pandora-boot process
        for process in "${CANDIDATES[@]}"; do
            index=$(($index+1))
            if [ $(echo ${process} | grep -c org.apache.catalina.startup.Bootstrap) -eq 1 ] \
                || [ $(echo ${process} | grep -c com.taobao.pandora.boot.loader.SarLauncher) -eq 1 ]
            then
               suggest=${index}
               break
            fi
        done

After selecting process, it is connected to a specified process. In the connecting portion attachwhere

# attach arthas to target jvm
# $1 : arthas_local_version
attach_jvm()
{
    local arthas_version=$1
    local arthas_lib_dir=${ARTHAS_LIB_DIR}/${arthas_version}/arthas

    echo "Attaching to ${TARGET_PID} using version ${1}..."

    if [ ${TARGET_IP} = ${DEFAULT_TARGET_IP} ]; then
        ${JAVA_HOME}/bin/java \
            ${ARTHAS_OPTS} ${BOOT_CLASSPATH} ${JVM_OPTS} \
            -jar ${arthas_lib_dir}/arthas-core.jar \
                -pid ${TARGET_PID} \
                -target-ip ${TARGET_IP} \
                -telnet-port ${TELNET_PORT} \
                -http-port ${HTTP_PORT} \
                -core "${arthas_lib_dir}/arthas-core.jar" \
                -agent "${arthas_lib_dir}/arthas-agent.jar"
    fi
}

The JVM attach to the inner achieved
by tools.jarthe package com.sun.tools.attach.VirtualMachineand VirtualMachine.attach(pid) in this way to achieve.

It is through the bottom JVMTI. Previous article had a simple analysis JVMTI technique ( when we talk Debug, what are we talking about (Debug implementation principle) ), before running or run-time, the custom Agent and VM loads and perform communication .
In the specific implementation of the content above arthas-core.jar the main class, we look at the specific content:

private void attachAgent(Configure configure) throws Exception {
        VirtualMachineDescriptor virtualMachineDescriptor = null;
        for (VirtualMachineDescriptor descriptor : VirtualMachine.list()) {
            String pid = descriptor.id();
            if (pid.equals(Integer.toString(configure.getJavaPid()))) {
                virtualMachineDescriptor = descriptor;
            }
        }
        VirtualMachine virtualMachine = null;
        try {
            if (null == virtualMachineDescriptor) { // 使用 attach(String pid) 这种方式
                virtualMachine = VirtualMachine.attach("" + configure.getJavaPid());
            } else {
                virtualMachine = VirtualMachine.attach(virtualMachineDescriptor);
            }

            Properties targetSystemProperties = virtualMachine.getSystemProperties();
            String targetJavaVersion = targetSystemProperties.getProperty("java.specification.version");
            String currentJavaVersion = System.getProperty("java.specification.version");
            if (targetJavaVersion != null && currentJavaVersion != null) {
                if (!targetJavaVersion.equals(currentJavaVersion)) {
                    AnsiLog.warn("Current VM java version: {} do not match target VM java version: {}, attach may fail.",
                                    currentJavaVersion, targetJavaVersion);
                    AnsiLog.warn("Target VM JAVA_HOME is {}, try to set the same JAVA_HOME.",
                                    targetSystemProperties.getProperty("java.home"));
                }
            }

            virtualMachine.loadAgent(configure.getArthasAgent(),
                            configure.getArthasCore() + ";" + configure.toString());
        } finally {
            if (null != virtualMachine) {
                virtualMachine.detach();
            }
        }
    }

By VirtualMachine, you can attach to the specified pid current, or through the VirtualMachineDescriptor realization of the specified process attach, the core is this sentence:

virtualMachine.loadAgent(configure.getArthasAgent(),
                            configure.getArthasCore() + ";" + configure.toString());

Thus, VM and specify the process to establish a connection, then you can communicate with friends.

Decompile class implementation

We question the diagnosis, there are times when you need to know the contents of the corresponding class currently loaded for easy identification loaded class are correct and so on, usually by javap only show a similar summary of the content, not intuitive. On the desktop side, we can through jd-guisuch tools, the command line is generally much optional.
Arthas then integrates this functionality.
Substantially following steps:

First to find the contents of the specified class by class name
Depending on the options, determine whether to look like Inner Class
Decompile

We look to achieve the Arthas.
For VM, specify the name of the class look, we see the following lines of code:

    public void process(CommandProcess process) {
        RowAffect affect = new RowAffect();
        Instrumentation inst = process.session().getInstrumentation();
        Set<Class<?>> matchedClasses = SearchUtils.searchClassOnly(inst, classPattern, isRegEx, code);

        try {
            if (matchedClasses == null || matchedClasses.isEmpty()) {
                processNoMatch(process);
            } else if (matchedClasses.size() > 1) {
                processMatches(process, matchedClasses);
            } else {
                Set<Class<?>> withInnerClasses = SearchUtils.searchClassOnly(inst,  classPattern + "(?!.*\\$\\$Lambda\\$).*", true, code);
                processExactMatch(process, affect, inst, matchedClasses, withInnerClasses);
    }

The key is to find content, do the package, SearchUtilswhere there is a core of the argument: Instrumentationare this guy to achieve.

    /**
     * 根据类名匹配，搜已经被JVM加载的类
     *
     * @param inst             inst
     * @param classNameMatcher 类名匹配
     * @return 匹配的类集合
     */
    public static Set<Class<?>> searchClass(Instrumentation inst, Matcher<String> classNameMatcher, int limit) {
        for (Class<?> clazz : inst.getAllLoadedClasses()) {
            if (classNameMatcher.matching(clazz.getName())) {
                matches.add(clazz);
            }
        }
        return matches;
    }

inst.getAllLoadedClasses(), It is the big players behind.
After find the Class, how to decompile it?

 private String decompileWithCFR(String classPath, Class<?> clazz, String methodName) {
        List<String> options = new ArrayList<String>();
        options.add(classPath);
//        options.add(clazz.getName());
        if (methodName != null) {
            options.add(methodName);
        }
        options.add(OUTPUTOPTION);
        options.add(DecompilePath);
        options.add(COMMENTS);
        options.add("false");
        String args[] = new String[options.size()];
        options.toArray(args);
        Main.main(args);
        String outputFilePath = DecompilePath + File.separator + Type.getInternalName(clazz) + ".java";
        File outputFile = new File(outputFilePath);
        if (outputFile.exists()) {
            try {
                return FileUtils.readFileToString(outputFile, Charset.defaultCharset());
            } catch (IOException e) {
                logger.error(null, "error read decompile result in: " + outputFilePath, e);
            }
        }

        return null;
    }

By such a method: decompileWithCFRSo we learned about decompilation is "by third-party tools CFR achieved." The above code is then passed to fight Option Main method implemented CFR, and then preserved. Interested friends can search benf cfr for specific usage.

Implement the query load classes

After reading the contents of the above decompiled class, we know that encapsulates a SearchUtilclass that will be used later in many places, but also in the query above decompile then after the class is. Query process, but also on the basis of the Instrument, plus a variety of matching rule filter, so more details will not be repeated.

We found several functions to achieve the above, there are two key things:

VirtualMachine
Instrumentation

Arthas whole logic is implemented on Instrumentation basis of Java, all loaded classes will by loading the Agent through after addTransformer, enhanced, then the corresponding Advice weaving in, look for the type of search, method, are is carried out by SearchUtil, all the JVM loaded class match by name by loadAllClass method Instrument, consistent would be returned.

Instrumentation is a good comrade! :)