Java service call tracking the whole process simple implementation

Foreword

Some time ago, I have been assisting the project team is split into multiple services made in the reconstruction of the system, system application, some of the services made cluster deployment. With the evolution of the above framework, the introduction of natural ELK + Filebeat do log collection. However, the use Kibana view the log, due to the lack TraceID, leading to difficult for developers to filter out the relevant specified log request, it is difficult to track application process calls for downstream services, spent a lot of time. I checked several times after issue, it can not stand each take so long time, I quickly put to the competent this time of transformation.

This article is mainly on my record project TraceIDissues and specific implementation research Link Trace transformation solutions encountered while this transformation also deepened my own understanding of some of the distributed service tracking, I also wrote in inside.

The main contents:

• The initial realization
• Asynchronous thread traceId loss problem
• For Dubbo RPC link tracking
• Tracking link for the HTTP Service
• Thinking of realization SpringCloud Sleuth
• summary

First, the initial realization

The general idea is to make use of MDC functions slf4j + Spring Interceptor, which generates a traceId MDC when placed into the external request.

MDC

Here briefly MDC.

MDC (Mapped Diagnostic Context, map debugging context) is a log function and a convenient log4j provided logback recorded in multithreaded conditions. Map MDC can be seen as a bound with the current thread, to which you can add key-value pairs. MDC content contained in the code can be accessed by the same thread of execution. Sub-thread of the current thread inherits parent thread contents of the MDC. When you need to log, just from the MDC to obtain the required information. MDC content stored by the program to go at the right time. For a Web application, the data is usually stored at the beginning of the request is processed.

Briefly, the MDC is a log provided by the framework InheritableThreadLocal, the code can project into the key-value pair wherein, when printing from the ThreadLocalacquisition of the corresponding value is then printed out. Details of the principles described herein will not go into details. Log4j look inside and logback implementation class will know.

achieve

1. Customizing Spring interceptors TraceInterceptor

/**
 * @author Richard_yyf
 */
public class TraceInterceptor extends HandlerInterceptorAdapter {

    @Override
    public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
        // 清空
        MDC.clear();

        ThreadMdcUtil.setTraceIdIfAbsent();

        //后续逻辑... ...
        return true;
    }
}
复制代码

2. Registration interceptor

/**
 * @author Richard_yyf
 */
@Configuration
public class WebMvcConfig implements WebMvcConfigurer {

    @Override
    public void addInterceptors(InterceptorRegistry registry) {
        registry.addInterceptor(traceInterceptor())
                .addPathPatterns("/**")
                .order(0);
    }

    @Bean
    public TraceInterceptor traceInterceptor() {
        return new TraceInterceptor();
    }

}
复制代码

ThreadMdcUtilIt is a utility class of my own package, a package of some operation on TraceId:

public class ThreadMdcUtil {
    
    public static String createTraceId() {
        String uuid = UUID.randomUUID().toString();
        return DigestUtils.md5Hex(uuid).substring(8, 24);
    }

    public static void setTraceIdIfAbsent() {
        if (MDC.get(TRACE_ID) == null) {
            MDC.put(TRACE_ID, createTraceId());
        }
    }
    // 省略了一些方法在后面会展示出来
}
复制代码

DigestUtilsFrom third parties rely on:

<dependency>
	<groupId>commons-codec</groupId>
	<artifactId>commons-codec</artifactId>
    <version>***</version>
</dependency>
复制代码

TRACE_IDIn Constantclass ease of reference:

public class Constant {
    ...
   public static final String TRACE_ID = "traceId";
    ...
}
复制代码

3. Modify the output format in the log file configuration, increase the printing field TraceID

   Value ways:%X{traceid}

result

After passing through the above steps, your web application log after receiving a print request will tape TraceId.

Second, encounter the problem of lost the thread pool TraceID

Previous program simply achieved our most basic needs. But if you really use it, you will find asynchronous task threads are not acquired to TraceIDthe.

A mature application will certainly use a lot of thread pool. Common are @Asyncthe thread pool asynchronous call, the application of the definition of some of its own thread pool, and so on.

Slightly earlier mentioned, MDC is through InheritableThreadLocalto achieve, when you create a child thread, copies inheritableThreadLocals property of the parent thread. But in the thread pool threads are multiplexed, rather than newly created, so the content can not be delivered into the MDC.

So we need to Quxianjiuguo, since threads are multiplexed, that we take for granted can think of to do some "show" operation when the task is submitted to the thread pool, in terms of content delivery MDC down.

Reform

Here placed directly Code:

/**
 * @author Richard_yyf
 */
public class ThreadMdcUtil {
    
    public static String createTraceId() {
        String uuid = UUID.randomUUID().toString();
        return DigestUtils.md5Hex(uuid).substring(8, 24);
    }

    public static void setTraceIdIfAbsent() {
        if (MDC.get(TRACE_ID) == null) {
            MDC.put(TRACE_ID, createTraceId());
        }
    }

    public static void setTraceId() {
        MDC.put(TRACE_ID, createTraceId());
    }

    public static void setTraceId(String traceId) {
        MDC.put(TRACE_ID, traceId);
    }

    public static <T> Callable<T> wrap(final Callable<T> callable, final Map<String, String> context) {
        return () -> {
            if (context == null) {
                MDC.clear();
            } else {
                MDC.setContextMap(context);
            }
            setTraceIdIfAbsent();
            try {
                return callable.call();
            } finally {
                MDC.clear();
            }
        };
    }

    public static Runnable wrap(final Runnable runnable, final Map<String, String> context) {
        return () -> {
            if (context == null) {
                MDC.clear();
            } else {
                MDC.setContextMap(context);
            }
            setTraceIdIfAbsent();
            try {
                runnable.run();
            } finally {
                MDC.clear();
            }
        };
    }
}
复制代码

Extended own packaging ThreadPoolExecutor

/**
 * @author Richard_yyf
 */
public class ThreadPoolExecutorMdcWrapper extends ThreadPoolExecutor {

    public ThreadPoolExecutorMdcWrapper(int corePoolSize, int maximumPoolSize, long keepAliveTime, TimeUnit unit,
                                        BlockingQueue<Runnable> workQueue) {
        super(corePoolSize, maximumPoolSize, keepAliveTime, unit, workQueue);
    }

    public ThreadPoolExecutorMdcWrapper(int corePoolSize, int maximumPoolSize, long keepAliveTime, TimeUnit unit,
                                        BlockingQueue<Runnable> workQueue, ThreadFactory threadFactory) {
        super(corePoolSize, maximumPoolSize, keepAliveTime, unit, workQueue, threadFactory);
    }

    public ThreadPoolExecutorMdcWrapper(int corePoolSize, int maximumPoolSize, long keepAliveTime, TimeUnit unit,
                                        BlockingQueue<Runnable> workQueue, RejectedExecutionHandler handler) {
        super(corePoolSize, maximumPoolSize, keepAliveTime, unit, workQueue, handler);
    }

    public ThreadPoolExecutorMdcWrapper(int corePoolSize, int maximumPoolSize, long keepAliveTime, TimeUnit unit,
                                        BlockingQueue<Runnable> workQueue, ThreadFactory threadFactory,
                                        RejectedExecutionHandler handler) {
        super(corePoolSize, maximumPoolSize, keepAliveTime, unit, workQueue, threadFactory, handler);
    }

    @Override
    public void execute(Runnable task) {
        super.execute(ThreadMdcUtil.wrap(task, MDC.getCopyOfContextMap()));
    }

    @Override
    public <T> Future<T> submit(Runnable task, T result) {
        return super.submit(ThreadMdcUtil.wrap(task, MDC.getCopyOfContextMap()), result);
    }

    @Override
    public <T> Future<T> submit(Callable<T> task) {
        return super.submit(ThreadMdcUtil.wrap(task, MDC.getCopyOfContextMap()));
    }

    @Override
    public Future<?> submit(Runnable task) {
        return super.submit(ThreadMdcUtil.wrap(task, MDC.getCopyOfContextMap()));
    }
}
复制代码

use

Specific use your original is to executor = new ThreadPoolExecutor(...)be changed executor = new ThreadPoolExecutorMdcWrapper(...)to.

For example, you are using Spring @Asyncasynchronous method, the configuration of the thread pool when he declared as:

@SpringBootApplication
public class Application {

    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }

    @EnableAsync
    @Configuration
    class TaskPoolConfig {

        @Bean("taskExecutor")
        public Executor taskExecutor() {
            ThreadPoolTaskExecutor executor = new ThreadPoolExecutorMdcWrapper();
            executor.setCorePoolSize(10);
            executor.setMaxPoolSize(20);
            executor.setQueueCapacity(200);
            executor.setKeepAliveSeconds(60);
            return executor;
        }
    }

}
复制代码

result

Follow the steps above, your asynchronous tasks when the print log, will bring TraceID the original request.

Third, for Dubbo RPC link tracking

The project team is mainly used Dubbo to develop micro-services framework. We would like to call between services, upstream service delivery TraceIDto achieve the effect of link tracking.

Dubbo provide such a mechanism, by Dubbo RPC+ Dubbo Filterto set and deliver consumers TraceID.

See the official website to explain these two concepts.

Dubbo RPC
Dubbo Filter

I give here a direct extension point code and configuration.

Dubbo Filter for Consumer

End consumer applications:

/**
 * @author Richard_yyf
 */
@Activate(group = {Constants.CONSUMER})
public class ConsumerRpcTraceFilter implements Filter {

    @Override
    public Result invoke(Invoker<?> invoker, Invocation invocation) throws RpcException {
        //如果MDC上下文有追踪ID，则原样传递给provider端
        String traceId = MDC.get(TRACE_ID);
        if (StringUtils.isNotEmpty(traceId)) {
            RpcContext.getContext().setAttachment(TRACE_ID, traceId);
        }
        return invoker.invoke(invocation);
    }

}
复制代码

SPI configuration :

In the resourcesdirectory, create a /META-INF/dubbo/com.alibaba.dubbo.rpc.Filterfile.

consumerRpcTraceFilter=com.xxx.xxx.filter.ConsumerRpcTraceFilter
复制代码

Dubbo Filter for Provider

Application service provider end:

/**
 * @author Richard_yyf
 */
@Activate(group = {Constants.PROVIDER})
public class ProviderRpcTraceFilter implements Filter {
    
    @Override
    public Result invoke(Invoker<?> invoker, Invocation invocation) throws RpcException {
        // 接收消费端的traceId
        String traceId = RpcContext.getContext().getAttachment(TRACE_ID);
        if (StringUtils.isBlank(traceId)) {
            traceId = ThreadMdcUtil.createTraceId();
        }

        // 设置日志traceId
        ThreadMdcUtil.setTraceId(traceId);

        // TODO 如果这个服务还会调用下一个服务，需要再次设置下游参数
        // RpcContext.getContext().setAttachment("trace_id", traceId);

        try {
            return invoker.invoke(invocation);
        } finally {
            // 调用完成后移除MDC属性
            MDC.remove(TRACE_ID);
        }
    }

}
复制代码

SPI configuration:

providerRpcTraceFilter=com.xxx.xxx.filter.ProviderRpcTraceFilter
复制代码

Fourth, the tracking link for the HTTP Service

In addition to calls between Dubbo RPC in this way, a common micro-services have to complete the call via HTTP REST. In this scenario it is necessary in the upstream service automatically sends an HTTP call when TraceIDadded to the HTTP Header in.

The usual Spring RestTemplate example, the use interceptors to wrap the HTTP Header.

        RestTemplate restTemplate = new RestTemplate();

        List<ClientHttpRequestInterceptor> list = new ArrayList<>();
        list.add((request, body, execution) -> {
            String traceId = MDC.get(TRACE_ID);
            if (StringUtils.isNotEmpty(traceId)) {
                request.getHeaders().add(TRACE_ID, traceId);
            }
            return execution.execute(request, body);
        });

        restTemplate.setInterceptors(list);
复制代码

Since downstream services exposed by HTTP interface service interceptor is added to obtain a like.

public class TraceInterceptor extends HandlerInterceptorAdapter {

    @Override
    public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) {
        MDC.clear();
        String traceId = request.getHeader(TRACE_ID);
        if (StringUtils.isEmpty(traceId)) {
            ThreadMdcUtil.setTraceId();
        } else {
            MDC.put(TRACE_ID, traceId);
        }
        return true;
    }
}
复制代码

Fifth, on the Realization of Spring Cloud Sleuth

After several steps above, we have the equivalent of their own to form a basis of comparison service tracking solutions.

Spring Cloud as a micro-stop service development framework that provides a solution to this Spring Cloud Sleuth as a distributed system technology tracking. Here I intend to talk about it.

Sleuth is a mature technology solutions, based on Google Dapper achieve the theoretical basis, some of the terms which come from that paper. For the TraceIDissue of transfer of some of the ideas to solve the problem we are talking about a simple version of the above solutions, in fact, it is also reflected in the Sleuth.

The first is distributed track, Sleuth will SpanIDand TraceIDadded to Slf4J MDC, so the printed journal will have to bring a corresponding logo.

When they encounter problems thread pool TraceID transfer failure, we submit quite a task for the operation of the packaging, and in Slueth in, is by implementing HystrixConcurrencyStrategyto solve the interface TraceIDproblem of asynchronous transfer. Hystrix when the actual call, calls HystrixConcurrencyStrategythe wrapCallablemethod. By implementing it in wrapCallablewill be TraceIDstored together (see in particular SleuthHystrixConcurrencyStrategy).

In the face of Dubbo RPC invocation and the invocation Http Service, we, extension points, and the context of the mechanism by agreement or framework itself provides, to pass through Dubbo RpcContext + Filter and the Http Header + Interceptor way TraceID. In the Spring Cloud Sleuth tracked @Async, RestTemplate, Zuul, Feignand other component, the solution is similar ideas. For example, track RestTemplateis the same as above and borrowed Spring Client's Interceptor mechanism (@see TraceRestTemplateInterceptor).

The above is a simple comparison of our solutions and Spring Cloud Sleuth, and want to show the log to track ideas and technology to solve some of the common idea is similar.

Of course, Spring Cloud Sleuth based Dapper achieve, provides a more mature distributed architecture system call tracking, after integration ZipKin + spring-cloud-sleuth-zipkin dependent, you can build a complete with a data collection, data storage and data display function distributed service tracking system.

By Sleuth can clearly understand what a service request through the service, each service takes a long process. So that we can easily clarify the relationships among micro-calling service. In addition Sleuth can help us:

• Time-consuming analysis: you can easily understand each sampling time-consuming request by Sleuth, to analyze what the service calls time-consuming;
• Visualization Error: uncaught exception for the program, you can see through the integration Zipkin service interface;
• Link Optimization: call for more frequent service, we can implement some optimization measures for these services.

PS: spring-cloud-sleth 2.0 starts officially supported Dubbo , if the idea is through Dubbo filter extension mechanism.

summary

Tell you why not introduce Sleuth + ZipKin this solution? Because the call link our system is not complicated, generally only call one relationship, it is not desirable to increase third-party components, prefer simple solutions.

This article to end here. Implement a simple micro-service call tracking program logs and not much difficulty, the important thing to solve the problem of thinking, and by analogy, to learn excellent technical solutions already exist in some of the market.

If this article helpful to you, hoping to spot a praise, this is the biggest driving force for me.