[Full link tracking] XXL-JOB adds TraceID

1. Background

First of all, this project is a small project. Due to manpower and time constraints, middleware such as Skywalking was not introduced for call link tracking. Skywalking is out of the scope of this discussion.

Next, introduce the relevant background of the project

call path

There are two main calling paths in the project

1. Web requests go through a unified gateway entrance and call back-end services
2. XXL-JOB timing task execution scheduling

deployment environment

Kubernetes

question

Don’t worry about the request to enter the unified gateway. TraceID is added to the gateway. However, since XXL-JOB is automatically registered and the deployment environment is in K8S, XXL-JOB obtains the IP of the Pod, but the gateway does not intercept it. .
Due to the complex logic of the project, the scheduling task of XXL-JOB is one of the more important parts. It is very necessary to add TraceID for the debugging of the early development and the confirmation of the later problems.

Two, the program

First of all, confirm that when XXLJOB executes scheduled tasks, the JobHandler does not have a TraceID. If the use of middleware is not considered, there are only two solutions.

One is to modify the source code of XXL-JOB and add TraceID in the request; the other is to intercept the request of XXL-JOB in the back-end service and add TraceID to the entry.

The source code of XXL-JOB has not been studied in detail. It has only been adapted to Oracle before, and it is difficult to transform. Therefore, the final solution is to intercept requests in the back-end service and add TraceID.

I searched for relevant information on the Internet and found that it is relatively simple to implement. Generally, TraceID is added to Slf4j's MDC through spring aop.
Here is a brief introduction to MDC, and I haven't done more understanding before.

MDC (Mapped Diagnostic Context) is a function provided by log4j and logback to facilitate logging under multi-threaded conditions. Some applications use multithreading to handle requests from multiple users.

MDC can be regarded as a hash table bound to the current thread, and key-value pairs can be added to it. The content contained in the MDC can be accessed by code executed in the same thread. When logging is required, only the required information can be obtained from the MDC.

In fact, it uses ThreadLocal to store, and when a request is made to the Java backend service, Tomcat will allocate a thread until the end of the request, which will ensure that the TraceID we added at the entrance will be passed to the entire link.
But there are two problems with using MDC calls:

1. The log TraceID in the child thread is lost

2. Cross-service call log TraceID is lost

At the same time, Openfeign is used in the project, and RequestInterceptor is used to intercept at the initiator, TraceID is added, and then HandlerInterceptor is used to intercept at the receiving end.

That is, the final solution is MDC+RequestInterceptor+HandlerInterceptor

The overall calling link is as follows:

Temporarily unable to display this content outside the Feishu document

3. Demo example

1、MDC

@Aspect
@Component
@Slf4j
public class XxlJobAopConfig {
    
    

    @Before("@annotation(com.xxl.job.core.handler.annotation.XxlJob)")
    public void beforeMethod() {
    
    
        MDC.put('traceId',
                UUID.randomUUID().toString().toLowerCase());
    }
}

2、RequestInterceptor

@Configuration
public class FeignRequestInterceptor implements RequestInterceptor {
    
    
    @Override
    public void apply(RequestTemplate template) {
    
    
        template.header('traceId', MDC.get(HeaderExtraInfoConstants.traceId));
    }
}

@FeignClient(name = "test", url = "xxx", configuration = FeignRequestInterceptor.class)

3、HandlerInterceptor

@Slf4j
@Component
public class HeaderInterceptor implements HandlerInterceptor {
    
    

    @Override
    public void afterCompletion(HttpServletRequest request, HttpServletResponse response, Object handler, Exception arg3) {
    
    
        MDC.remove('traceId');
    }

    @Override
    public void postHandle(HttpServletRequest request, HttpServletResponse response, Object handler, ModelAndView arg3) {
    
    
    }

    @Override
    public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) {
    
    
        String traceId = request.getHeader(HeaderExtraInfoConstants.traceId);
        if (StringUtils.isEmpty(traceId)) {
    
    
            MDC.put('traceId', UUID.randomUUID().toString().toLowerCase());
        } else {
    
    
            MDC.put('traceId', traceId);
        }
        return true;
    }

}

@Configuration
public class InterceptorConfiguration extends WebMvcConfigurationSupport {
    
    

    @Override
    protected void addInterceptors(InterceptorRegistry registry) {
    
    
        registry.addInterceptor(new HeaderInterceptor()).addPathPatterns("/**");
    }
}

4、logback.xml

%d{
    
    yyyy-MM-dd HH:mm:ss.SSS} ---> [%X{
    
    traceId}] ---> [%thread] ---> %-5level %logger{
    
    50} - %msg%n

4. Subsequent improvement ideas

The above solution has great limitations and is only applicable to the method of calling feign between services. If there are other methods such as okhttp, an interceptor needs to be added. The problem of multi-threading has not been solved, and the common way is to solve it by rewriting the thread pool.

1. Enrich call scenarios and add interceptors

2. rewrite thread pool

3. Since it is deployed in a K8S cluster, Istio can be enabled for service governance