foreword
In our increasingly large microservice group, with the continuous expansion of business, the number of microservices is also increasing. The microservice architecture system and service splitting lead to the continuous complexity of system call links , a slightly complicated front-end request may eventually need to call many back-end services to complete, and a back-end service may be implemented through multiple Feign calls; when our request fails or performance decreases, we analyze it to the end The fundamentally wrong microservices also bring great difficulties. The link tracking of distributed systems is used to challenge this difficulty. Today we introduce Sleuth+Zipkin, which is used to support SpringCloude's microservice cloud system. Solutions for location and tracking of services in .
Http tracking
In order to implement request link tracking, when a request is sent to the entrance of the distributed system, it is only necessary to create a unique tracking identifier for the request in the service tracking framework, and ensure that the identifier circulates within the distributed system until the request is returned. The identifier is traceId, through which the logs of different service calls can be connected in series. In short, in the entire Http call (FeignClient or RestTemplate), we can identify a TraceId record, and in the Http Request process, pass the TraceID to the server that provides the service, and the server that provides the service is in Reuqest In the same way, the entire TraceId is passed to its service provider, so that the entire top-down call link has the same TraceId, and the logs of the services in each link are grouped together through the TraceId And, in this way, you get a call link with different TraceId, which is the main principle of link tracking. Link tracking also has another domestic product Skywalking, which has also become a top open source project in Apache. Although it is in The specific implementation is different from Sleuth, but the principle of use is the same.
MDC(Mapped Diagnostic Contexts)
Through the above method, we have the implementation method of link tracking. MDC ((Mapped Diagnostic Contexts)) translates to the mapped diagnostic context. It means: the (mapped) request ID (requestId) in the log can be used as the keyword (context) for us to locate (diagnose) the problem . This is not a new product. The basic principle of the MDC class is actually very simple. It holds a ThreadLocal instance internally for saving context data. MDC provides several core interfaces such as put/get/clear for operating ThreadLocal Data; KV in ThreadLocal can be declared in logback.xml, that is, by declaring %X{Key} in the layout to print the value corresponding to the key saved in the MDC in the log. For specific information about MDC, you can go to Baidu’s articles about MDC separately. The Http link tracking traceId is realized through MDC.
Sleuth
Sleuth provides a tracking solution in the SpringCloud distributed system. Sleuth can record the relevant information of link tracking, and send the relevant link call information to the designated statistical terminal and display terminal, such as Zipkin
Spring Cloud Sleuth keywords ( based on Google Dapper )
- Span: The basic unit of work. Sending a remote scheduling task will generate a Span. Span is uniquely identified by a 64-bit ID. Trace is uniquely identified by another 64-bit ID. Span also has other data information, such as summary and time. Stamp event, Span ID, and progress ID.
- Trace: A tree structure composed of a series of Spans. Request the API interface of a microservice system. This API interface needs to call multiple microservices. Calling each microservice will generate a new Span. All the Spans generated by this request form this Trace.
- Annotation: Used to record an event in time, some core annotations are used to define the start and end of a request. These annotations include the following:
- cs - Client Sent - The client sends a request, this annotation describes the beginning of this Span
- sr - Server Received - The server gets the request and is ready to start processing it. If you subtract the cs timestamp from sr, you can get the time of network transmission.
- ss - Server Sent (the server sends a response) – this annotation indicates the completion of the request processing (when the request returns to the client), if the timestamp of ss minus the timestamp of sr, the time of the server request can be obtained.
- cr - Client Received (the client receives the response) - the end of the Span at this time, if the cr timestamp minus the cs timestamp, the time consumed by the entire request can be obtained.
Zipkin
Zipkin is an open source project of Twitter, implemented based on Google Dapper. It can be used to collect the tracking data of the request link on each server, and through the REST API interface it provides to assist us in querying the tracking data to realize the monitoring program of the distributed system, so as to timely discover the delay increase in the system problems and identify the source of system performance bottlenecks. In addition to the development-oriented API interface, it also provides convenient UI components to help us intuitively search for tracking information and analyze request link details, for example: you can query the processing time of each user request within a certain period of time, etc.
In short, Zipkin collects the data of sleuth on the client side, and provides a visual interface for query display.
Docker start Zipkin
docker run -d -p 9411:9411 openzipkin/zipkin
Visit Zipkin's graphics page
After clicking on a certain request, you will see detailed call chain request information:
Spring Cloud uses Sleuth
It is very convenient to use Sleuth for link tracking in the SpringCloud project. You only need to map Sleuth's Starter in the project. If the information collected by Sleuth for link tracking needs to be pushed to Zipkin, then introduce Zipkin's Starter and configure Zipkin. The service address of the Server is enough.
Introduce related dependent packages and Start
<!-- 配置服务链路追踪 -->
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-sleuth-zipkin</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-sleuth</artifactId>
</dependency>
The above pom.xml file adds Sleuth's starter and Zipkin's support. FeignClient and RestTemplate will be recorded by Sleuth and sent to Zipkin's service address in the Restful call of SpringCloud's microservices.
Configure Zipkin's service address
spring:
zipkin:
base-url: http://localhost:9411/
sender:
type: web
sleuth:
sampler:
#抽样百分比，默认10%的数据发到zipkin，1为100%
probability: 1Through the above method, in our SpringCloud microservice cloud system, the service calls of each of our microservice clouds will enter the trace range of call tracking, and then through the graphical display tool provided by Zipkin, it can be seen at a glance Knowing the link tracking of each of our service calls, through the link tracking solution, when the entire request becomes slow or unavailable, we can know that the request is caused by one or some backend services Yes, read quickly to locate service failure points and prescribe the right medicine.
Use MQ to push information
Through the above method, the call tracking of each microservice will send an Http request to the Zipkin Server. When the internal network is blocked, it may cause the main business call to be blocked, resulting in more serious consequences, so at this time we can consider using Stream Or use MQ to asynchronously process requests from Sleuth;
Our SpringCloud microservice only needs to introduce a starter that supports Stream and MQ
<!-- 配置服务链路追踪 -->
<!-- <dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-zipkin</artifactId>
</dependency> -->
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-sleuth-zipkin-stream</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-stream-rabbit</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-sleuth</artifactId>
</dependency>
The configuration of Zipkin Server is also adjusted accordingly;
Zipkin Advanced Usage Persistence
Zipkin saves Sleuth's log information in memory by default, so that every time Zipkin is restarted, the previous log information will be lost. You can modify the configuration of Zipkin to let Zipkin Server persist all logs into the database. Regarding persistence and the way Zipkin supports MQ, we will introduce it in future articles.
conclusion
The link tracking of each service call, through the link tracking solution, when the entire request becomes slow or unavailable, we can know that the request is caused by one or some backend services, quick read Locate service failure points and prescribe the right medicine. This article mainly introduces the implementation of SpringCloud's microservice cloud link tracking through Sleuth+Zipking. In the article, we mentioned the usage of Zipkin's MQ receiving sleuth log information and the persistence of log information. We will be in future articles. Introduction, about Skywalking, which can also be used as a link tracking solution in the microservice system, we will introduce Skywalking separately in a special article in the future. I also hope that everyone will continue to pay attention to each of the author's articles, pay attention to the author, and don't miss the excitement.
Thank you for your continued attention.