Distributed Link Tracking ( Distributed Tracing )

It is to restore a distributed request to a call link, log records, performance monitoring and centralized display of the call situation of a distributed request.

For example, the time consumption on each service node, which machine the request reaches, the request status of each service node, and so on.

Common link tracking technologies are as follows:

cat

A real-time application monitoring platform developed by Dianping open source and based on Java, including real-time application monitoring and business monitoring. The integration scheme is to realize monitoring by burying points in code, such as interceptors, filters, etc. The code is very intrusive and the integration cost is high. The risk is greater.

pinpoint

Pinpoint is a Korean open source call chain analysis based on bytecode injection, and an application monitoring analysis tool. The feature is that it supports a variety of plug-ins, the UI is powerful, and the access terminal has no code intrusion.

skywalking

SkyWalking is a local open source call chain analysis based on bytecode injection, and an application monitoring analysis tool. The feature is that it supports a variety of plug-ins, the UI is strong, and there is no code intrusion on the access terminal. Has joined the Apache incubator.

Sleuth (main)

The link tracking solution in the distributed system provided by SpringCloud .

zipkin (main)

The open source, open source distributed tracking system by Twitter is used to collect service timing data to solve the delay problem in the microservice architecture, including: data collection, storage, search and display.

The product combined with spring-cloud-sleuth is relatively simple to use and easy to integrate, but the function is relatively simple.

Note: The SpringCloud alibaba technology stack does not provide its own link tracking technology. We can use Sleuth + Zinkin as a link tracking solution.

Sleuth

The main function of SpringCloud Sleuth is to provide tracking solutions in distributed systems.

1.Span (basic work unit)

Represents a basic set of work units . In order to count the delay of each processing unit, when a request arrives at each service component, a unique identifier ( SpanId ) is also used to mark its beginning, specific process, and end. Through the start and end timestamps of the SpanId, we can count the call time of the span . In addition, we can also get the name of the event. Request metadata such as information.

2.Trace (Tree collection of Span)

A set Trace Id same Span series to form a tree structure . In order to achieve request tracing, when the request reaches the entry endpoint of the distributed system , only the service tracing framework needs to create a unique identifier (ie TraceId ) for the request , and at the same time in the distributed system

During the internal circulation of the system, the framework always keeps passing the unique value until the return of the entire request. Then we can use the unique identifier to concatenate all the requests to form a complete request link.

3.Annotation (event unit)

Use it to record events over a period of time, important notes for internal use:

cs ( Client Send ) The client sends a request and starts a request life cycle

sr ( Server Received ) The server receives the request and starts processing, sr - cs = network delay (time for service invocation)

ss ( Server Send ) The server is processed and ready to be sent to the client, ss-sr = request processing time on the server

cr ( Client Reveived ) The client receives the response from the server, and the request ends. cr-sr = total time requested

Move the dependency in the gateway microservice:

        <!--Sleuth:形成链路追踪信息-->
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-sleuth</artifactId>
        </dependency>

To test the interface, you can see the link information of the interface call in the console. The four attributes represent: [ microservice name , traceId, spanid, whether to output the link trace result to a third-party platform ]

Zipkin

Want to know why Zipkin is used?

Sleuth can implement link tracking and store the information in the logs of each microservice . It is not convenient to view the logs one by one, and at the same time, there are more and more log files, which is not convenient to view.

And Zipkin can aggregate logs and perform visual display and full-text retrieval.

Zipkin is an open source project of Twitter . It is based on Google Dapper. It is committed to collecting service timing data to solve the delay problem in the microservice architecture, including data collection, storage, search and display .

effect:

1. Collect the tracking data of the request link on each server , and query the tracking data through the REST API interface it provides to realize the monitoring program of the distributed system , so as to find the delay increase in the system in time and find out The root cause of system performance bottlenecks .

2. It provides UI components that can help our intuitive search and track information and analysis, please request link details , such as: can query processing time within a certain period of time each user requests and so on.

Zipkin provides pluggable data storage methods: In-Memory , MySql , Cassandra and Elasticsearch .

Zipkin 's infrastructure:

1. Collector : Collector component, which is mainly used to process tracking information sent from external systems and convert this information into the Span format processed internally by Zipkin to support subsequent storage, analysis, display and other functions.

2.Storage : storage component, which mainly deal with tracking information collected is received, by default the information is stored in memory , we can also modify storage strategy , by using other storage components to track information stored in the database .

3. RESTful API : API component, it is mainly used to provide external access interface . For example, display tracking information to the client , or access to an external system for monitoring .

4. Web UI : UI components, upper-level applications based on API components. Through UI components, users can conveniently and intuitively query and analyze tracking information .

Zipkin CS:

Zipkin is divided into two ends, one is the Zipkin server and the other is the Zipkin client . The client is the application of the microservice . The client configures the server's URL address, the time between service calls once occurred, is configured in micro-services inside Sleuth listener prison

Listen, and generate the corresponding Trace and Span information and send it to the server .

ZipKin server installation

https://dl.bintray.com/openzipkin/maven/io/zipkin/java/zipkin-server/

Download version 2.12.9 and start it with cmd command window

java -jar zipkin-server-2.12.9-exec.jar --logging.level.zipkin2=INFO

The default level of logging is INFO (so you don't need to set it)

Accessed through a browser http: // localhost: 9411 visit

Zipkin client integration

Add dependencies on each microservice ( must ensure that the services in the interface call chain have imported zipkin dependencies, otherwise the corresponding microservice will not be recognized in the final link ):

        <!--zipkin:链路追踪信息的收集、存储、查找和展现,常和Sleuth一起使用-->
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-zipkin</artifactId>
        </dependency>

Configuration file:

spring:
  zipkin:
    base-url: http://127.0.0.1:9411/ #zipkin 服务端的地址
    discoveryClientEnabled: false #让nacos把它当成一个URL，而不要当做服务名
  sleuth:
    sampler:
      probability: 1.0 #采样的百分比

Test interface:

Go to the zipkin server ( http://127.0.0.1:9411/zipkin/ ) and click the find button

Find 3 link tracking logs, choose one to see:

You can see the link tracking information:

ZipKin data persistence

Just querying the link tracking is not enough, because the data is stored in the memory at this time and will disappear once the service is restarted; therefore, the persistence of the data needs to be realized.

There are many ways to persist: In-Memory , MySql , Cassandra and Elasticsearch .

Here I mainly introduce the easiest and most convenient way I think: Elasticsearch.

Create a new module module (shop-zipkin) and import the key dependencies zipkin-autoconfigure-ui and zipkin-server :

        <dependency>
            <groupId>io.zipkin.java</groupId>
            <artifactId>zipkin-autoconfigure-ui</artifactId>
            <version>2.12.9</version>
        </dependency>
        <dependency>
            <groupId>io.zipkin.java</groupId>
            <artifactId>zipkin-server</artifactId>
            <version>2.12.9</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

Configuration file: (remember to configure nacos, otherwise an error will be reported: java.lang.IllegalArgumentException: no server available )

server:
  port: 9411
spring:
  application:
    name: service-zipkin
  cloud:
    nacos: #nacos的作用就是一个注册中心
      discovery:
        server-addr: 127.0.0.1:8848

Remember to import the spring-boot-starter-web dependency, otherwise an error will be reported:

Caused by: org.springframework.context.ApplicationContextException: Unable to start ServletWebServerApplicationContext due to missing ServletWebServerFactory bean.
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.getWebServerFactory(ServletWebServerApplicationContext.java:206) ~[spring-boot-2.1.3.RELEASE.jar:2.1.3.RELEASE]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.createWebServer(ServletWebServerApplicationContext.java:180) ~[spring-boot-2.1.3.RELEASE.jar:2.1.3.RELEASE]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:154) ~[spring-boot-2.1.3.RELEASE.jar:2.1.3.RELEASE]
	... 8 common frames omitted

Then you will see that the startup is successful: but there will be a red warning:

This is because: SLF4J is bound to multiple implemented log frameworks . Although it does not affect the startup of the project, as an obsessive-compulsive disorder, I choose to remove log4j bound to zipkin-server in the pom file. The modified dependency is:

        <!--nacos客户端,引入依赖即可自动注册微服务-->
        <dependency>
            <groupId>com.alibaba.cloud</groupId>
            <artifactId>spring-cloud-starter-alibaba-nacos-discovery
            </artifactId>
        </dependency>
        <dependency>
            <groupId>io.zipkin.java</groupId>
            <artifactId>zipkin-server</artifactId>
            <version>2.12.9</version>
            <exclusions>
                <exclusion>
                    <artifactId>log4j-slf4j-impl</artifactId>
                    <groupId>org.apache.logging.log4j</groupId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>io.zipkin.java</groupId>
            <artifactId>zipkin-autoconfigure-ui</artifactId>
            <version>2.12.9</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

Then start and report an error, as follows:

Caused by: java.lang.NullPointerException: null
	at zipkin2.autoconfigure.ui.ZipkinUiAutoConfiguration.lambda$uiServerConfigurator$0(ZipkinUiAutoConfiguration.java:179) ~[zipkin-autoconfigure-ui-2.12.9.jar:na]
	at com.linecorp.armeria.spring.ArmeriaAutoConfiguration.lambda$armeriaServer$0(ArmeriaAutoConfiguration.java:111) ~[armeria-spring-boot-autoconfigure-0.83.0.jar:na]
	at java.util.ArrayList.forEach(ArrayList.java:1257) ~[na:1.8.0_211]
	at com.linecorp.armeria.spring.ArmeriaAutoConfiguration.lambda$armeriaServer$1(ArmeriaAutoConfiguration.java:110) ~[armeria-spring-boot-autoconfigure-0.83.0.jar:na]
	at java.util.Optional.ifPresent(Optional.java:159) ~[na:1.8.0_211]
	at com.linecorp.armeria.spring.ArmeriaAutoConfiguration.armeriaServer(ArmeriaAutoConfiguration.java:109) ~[armeria-spring-boot-autoconfigure-0.83.0.jar:na]
	at com.linecorp.armeria.spring.ArmeriaAutoConfiguration$$EnhancerBySpringCGLIB$$167b7a26.CGLIB$armeriaServer$0(<generated>) ~[armeria-spring-boot-autoconfigure-0.83.0.jar:na]
	at com.linecorp.armeria.spring.ArmeriaAutoConfiguration$$EnhancerBySpringCGLIB$$167b7a26$$FastClassBySpringCGLIB$$641124e0.invoke(<generated>) ~[armeria-spring-boot-autoconfigure-0.83.0.jar:na]
	at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244) ~[spring-core-5.1.5.RELEASE.jar:5.1.5.RELEASE]
	at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:363) ~[spring-context-5.1.5.RELEASE.jar:5.1.5.RELEASE]
	at com.linecorp.armeria.spring.ArmeriaAutoConfiguration$$EnhancerBySpringCGLIB$$167b7a26.armeriaServer(<generated>) ~[armeria-spring-boot-autoconfigure-0.83.0.jar:na]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_211]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_211]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_211]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_211]
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ~[spring-beans-5.1.5.RELEASE.jar:5.1.5.RELEASE]
	... 20 common frames omitted

This is because there is a larger update since version 2.12.6 , which uses the Armeria HTTP engine for migration. Starting from this version, if you directly add dependent Spring Boot applications, there will be conflicts . So change to version 2.12.3 (other versions will have problems) :

        <!--nacos客户端-->
        <dependency>
            <groupId>com.alibaba.cloud</groupId>
            <artifactId>spring-cloud-starter-alibaba-nacos-discovery
            </artifactId>
        </dependency>
        <dependency>
            <groupId>io.zipkin.java</groupId>
            <artifactId>zipkin-server</artifactId>
            <version>2.12.3</version>
            <exclusions>
                <exclusion>
                    <artifactId>log4j-slf4j-impl</artifactId>
                    <groupId>org.apache.logging.log4j</groupId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>io.zipkin.java</groupId>
            <artifactId>zipkin-autoconfigure-ui</artifactId>
            <version>2.12.3</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

The startup is successful, but an error is reported when visiting http://127.0.0.1:9411/ :

Modify the configuration file ( added to turn off automatic detection of all requested configurations ):

server:
  port: 9411
spring:
  application:
    name: service-zipkin
  cloud:
    nacos: #nacos的作用就是一个注册中心
      discovery:
        server-addr: 127.0.0.1:8848
management:
  metrics:
    web:
      server:
        auto-time-requests: false # 关闭自动检测所有请求

Visit http://127.0.0.1:9411 and note that if it is set to other ports, it seems that only the current microservice can be detected, but other microservices cannot be detected. This should be related to the address mapping of the zipkin backend server:

Nepxion Discovery study notes 9 link tracking (Sleuth+Zipkin)