Recently, I was responsible for the system link tracking problem in the company. Some of the company’s services have already had a basic implementation. The general principle is that when a request reaches the front-end module of the system (usually an API gateway), it is intercepted through the AOP aspect, using uuid or snowflake technology to generate a unique id (to simplify writing, we will call it cid, or correlationId), and set cid in the header of the request, and then release the request. As each service receives a request, it will output the request information to the log file. In this way, when a service receives a request and responds to the request, the cid will be output to the log file along with the request object.
So when a service receives the request, how to carry the cid in the internal processing flow of the service? The answer is to use ThreadLocal. When a service receives a call request, take out the cid in the request header and put it in ThreadLocal. In the post-processing process, take out the cid from ThreadLocal when outputting the log and splice it into the log information. That's it.
But there is a question, if asynchronous code is involved in the internal processing flow of the service, what should we do? Because the main thread and the asynchronous sub-thread are different thread environments, the ThreadLocal we set at the beginning is bound to the main thread. If it is not processed, the cid information will be lost in the asynchronous code. One of the most basic solutions is to replace the original ThreadLocal with InheritableThreadLocal. As the name suggests, this object is an "inheritable" ThreadLocal. Its performance is that when the main thread opens the asynchronous child thread, the ThreadLocal of the main thread can be passed to the child thread. The general principle is that when a child thread is created in the main thread, the ThreadLocal of the main thread is copied, and then set to the threadLocalMap object of the child thread to complete the context transfer.
Seems to solve the problem. But because InheritableThreadLocal is set when the asynchronous child thread is initialized, and the thread will only be initialized once, the ThreadLocal in the child thread will always be the same. This will not cause any problems in ordinary application scenarios. But in current development, pooling technology is generally used to manage multiple threads, that is, thread pools are used. In the thread pool, core threads generally do not die, they will be used repeatedly. The problem with InheritableThreadLocal is that threadLocal is only passed when the child thread is initialized. When this child thread completes the first task and then processes other requests, the threadLocal in this child thread is still the first request. The threadLocal set. Will cause errors.
Simply put, InheritableThreadLocal can only be set once in thread initialization, and threads in the pool need to reset threadLocal each time a different request is processed.
Therefore, InheritableThreadLocal is not applicable in the thread pool scenario.
For the problem of InheritableThreadLocal, there are many solutions, such as Ali open source TransmittableThreadLocal.
The log facade framework, slf4j, provides an MDC mechanism. Developers can add to the configuration of the log output format in the xml configuration file of logback or log4j %X{cid}, and then call MDC.put("cid","123456")it in the code , then the log information output by the logger will be automatically 123456replaced with %X{cid}and replaced to achieve dynamic addition to the log The purpose of the associated information.
The MDC mechanism also uses ThreadLocal internally. The early MDC mechanism uses InheritableThreadLocal, which takes into account the scenario where the associated information needs to be transferred between the parent and child threads. However, in later versions, due to performance considerations, ThreadLocal was replaced.
So if we want to use the MDC mechanism of slf4j, and expect to use TransmittableThreadLocal in the MDC mechanism, it is convenient to complete the context transfer between parent and child threads. How to do it
Fortunately, on the omnipotent single dating site Github, some developers provide proxy jar packages for logback and log4j that use TransmittableThreadLocal. We can -javaagentadd a proxy to our application through virtual machine parameters , so that the MDC mechanism is implemented using TransmittableThreadLocal.
In this way, for a request, there can be a unique id to identify it in the entire link. In the follow-up, the log can be centrally managed, and the problem can be located through an id when troubleshooting.
But this is just the beginning of link tracking.
At present, the popular APM link tracking tools on the market all adopt a proxy-like approach, using probe technology to inject tracking IDs into service logs, collect the logs, analyze them, and display them through the page UI.
Recently, I am studying pinpoint and skywalking, both of which are enhanced based on bytecode without changing the original code. Compared with the more intrusive zipkin, it is more convenient to deploy.