background

SkyWalking Java Agent 8.10.0 adds a plugin for monitoring database connection pool and tomcat thread pool: skywalking.apache.org/events/rele…

If we want to add some thread pool monitoring for our own business, Skywalking also provides expansion points.

accomplish

Client implementation

This official document describes how the client (that is, the application) uploads data to the SkyWalking server.

skywalking.apache.org/docs/skywal…

By looking at the official implementation of the tomcat thread pool, we can see that the MeterFactory.gaugeAPI is mainly used, and we can simulate it in our own components with similar writing.

Judging from the interception source code of the tomcat thread pool, in fact, the core principle is to intercept the Java thread pool object in a suitable place, and then obtain the corresponding status by calling the getXxxSize of the thread pool object, and then report it.

Because my requirement here is to monitor the business thread pool when grpc is running, first I need to write my client plug-in to get the grpc business thread pool and call it for status monitoring. My source code is as follows:

Define the interception class. I intercept it here at ServerImpl.start, that is, when the grpc service starts, and obtain the thread pool:

public class GrpcThreadPoolInstrumentation extends ClassEnhancePluginDefine {

    // 代理类
    public static final String INTERCEPT_CLASS =
            "org.apache.skywalking.apm.plugin.grpc.v1.server.GrpcDefaultThreadExecutorInterceptor";
    // 需要拦截的方法
    public static final String ENHANCE_METHOD = "start";
    // 需要拦截的类
    public static final String ENHANCE_CLASS = "io.grpc.internal.ServerImpl";

    @Override
    public ConstructorInterceptPoint[] getConstructorsInterceptPoints() {
        return new ConstructorInterceptPoint[0];
    }

    @Override
    public InstanceMethodsInterceptPoint[] getInstanceMethodsInterceptPoints() {
        return new InstanceMethodsInterceptPoint[]{new InstanceMethodsInterceptPoint() {
            @Override
            public ElementMatcher<MethodDescription> getMethodsMatcher() {
                return named(ENHANCE_METHOD);
            }

            @Override
            public String getMethodsInterceptor() {
                return INTERCEPT_CLASS;
            }

            @Override
            public boolean isOverrideArgs() {
                return false;
            }
        }};
    }

    @Override
    public StaticMethodsInterceptPoint[] getStaticMethodsInterceptPoints() {
        return new StaticMethodsInterceptPoint[0];
    }

    @Override
    protected ClassMatch enhanceClass() {
        return byName(ENHANCE_CLASS);
    }

}
复制代码

In my interception class, I obtained the executor attribute in the ServerImpl class through reflection, which is the business thread pool, and then registered it in the MeterFactory.gauge API,

Note METER_NAMEand THREAD_POOL_NAMEthese two variables, the former represents the name you reported to Skywalking, and the latter represents the name of the tag tag. It should be handled well here. Do not have the same name as the previous default tomcat.

After defining this, pay attention to package the plugin into the plugin directory of the agent, and start the project.

public class GrpcDefaultThreadExecutorInterceptor implements InstanceMethodsAroundInterceptor {

    private static final String METER_NAME = "grpc_thread_pool";
    private static final String METRIC_POOL_NAME_TAG_NAME = "pool_name";
    private static final String THREAD_POOL_NAME = "grpc_default_executor";
    private static final String METRIC_TYPE_TAG_NAME = "metric_type";

    @Override
    public void beforeMethod(EnhancedInstance objInst, Method method, Object[] allArguments, Class<?>[] argumentsTypes,
                             MethodInterceptResult result) throws Throwable {
    }

    @Override
    public Object afterMethod(EnhancedInstance objInst, Method method, Object[] allArguments, Class<?>[] argumentsTypes,
                              Object ret) throws Throwable {
        Field field = ret.getClass().getDeclaredField("executor");
        field.setAccessible(true);
        ThreadPoolExecutor threadPoolExecutor = (ThreadPoolExecutor) field.get(ret);
        MeterFactory.gauge(METER_NAME, () -> (double) threadPoolExecutor.getCorePoolSize())
                .tag(METRIC_POOL_NAME_TAG_NAME, THREAD_POOL_NAME)
                .tag(METRIC_TYPE_TAG_NAME, "core_pool_size")
                .build();
        MeterFactory.gauge(METER_NAME, () -> (double) threadPoolExecutor.getMaximumPoolSize())
                .tag(METRIC_POOL_NAME_TAG_NAME, THREAD_POOL_NAME)
                .tag(METRIC_TYPE_TAG_NAME, "max_pool_size")
                .build();
        MeterFactory.gauge(METER_NAME, () -> (double) threadPoolExecutor.getPoolSize())
                .tag(METRIC_POOL_NAME_TAG_NAME, THREAD_POOL_NAME)
                .tag(METRIC_TYPE_TAG_NAME, "pool_size")
                .build();
        MeterFactory.gauge(METER_NAME, () -> (double) threadPoolExecutor.getQueue().size())
                .tag(METRIC_POOL_NAME_TAG_NAME, THREAD_POOL_NAME)
                .tag(METRIC_TYPE_TAG_NAME, "queue_size")
                .build();
        MeterFactory.gauge(METER_NAME, () -> (double) threadPoolExecutor.getActiveCount())
                .tag(METRIC_POOL_NAME_TAG_NAME, THREAD_POOL_NAME)
                .tag(METRIC_TYPE_TAG_NAME, "active_size")
                .build();
        return ret;
    }

    @Override
    public void handleMethodException(EnhancedInstance objInst, Method method, Object[] allArguments,
                                      Class<?>[] argumentsTypes, Throwable t) {
    }
}
复制代码