Opentracing + Jaeger whole call chain link based on the gradation

When the gateway and services in the implementation of the grayscale full link distributed publishing and routing, we need a tracking system to monitor and gateway services which go gray group, which grayscale version, which gray area, even from monitor Http header head full of gray rule pass and routing policy. The functional significance lies in:

• Not only monitor basic information call the whole link, you can also monitor additional gray scale information to help us determine whether the gray routes and publishing accurate, if there is a problem, you can quickly locate
• Traffic can be monitored when to switch to the new version, or a new area, a new machine or
• You can monitor the gray rules and routing policy is configured accurately
• Subordinate relationship can be monitored on a tree service gateway and gray
• You can monitor the whole flow link topology

I try to study a series of distributed tracking system and middleware, including Opentracing, Uber Jaeger, Twitter Zipkin, Apache Skywalking, Pinpoint, CAT, and finally decided to use Opentracing + Uber Jaeger way to achieve important because in addition to ease of use and scalability outside, Opentracing support WebMvc and WebFlux two ways, the industry tracking system can support a relatively small WebFlux

The concept [** OpenTracing **] OpenTracing has entered CNCF, are being distributed worldwide to provide a unified tracking system, specification, architecture and data standards. It has nothing to do by providing a platform, vendor-independent API, allowing developers to easily add (or replace) to track implementation of the system. For the call chain technology stack coexistence of diverse presence in, Opentracing adaptation Java, C, Go and .Net technology stack, to achieve full distributed link tracking. So far, Uber Jaeger, Twitter Zipkin and Apache Skywalking specification has been adapted to the Opentracing

Author Discovery open source framework Nepxion community (the students interested in open source framework, please visit the following link) to expand integration as an example

Source homepage, visit github.com/Nepxion/Dis...

Home Guide, please visit github.com/Nepxion/Dis...

Documentation home, please visit pan.baidu.com/s/1i57rXaNK...

Integrated renderings

basic concepts

Gradation call chain 11 includes the following parameters. Users can define their own call chain passed parameters, for example: traceId, spanId like; service call chain can also define their own parameters to be passed, for example: mobile, user, etc.

1. n-d-service-group - 服务所属组或者应用
2. n-d-service-type - 服务类型，分为“网关”和“服务”
3. n-d-service-id - 服务ID
4. n-d-service-address - 服务地址，包括Host和Port
5. n-d-service-version - 服务版本
6. n-d-service-region - 服务所属区域
7. n-d-version - 版本路由值
8. n-d-region - 区域路由值
9. n-d-address - 地址路由值
10. n-d-version-weight - 版本权重路由值
11. n-d-region-weight - 区域权重路由值复制代码

Core implementation

Opentracing Universal Module

Source reference github.com/Nepxion/Dis...

Since OpenTracing expansion needs to take into account Spring Cloud Gateway, Zuul and services, its core logic there are some may encapsulation, so I extracted a common module discovery-plugin-strategy-opentracing, comprising a configuration, operation, context modules, focuses on the operation module, other relatively simple, does not enumerate

Before going, I need to explain a configuration that the core will determine and implement display terminal interface

1. If enabled, the output gradation information into separate nodes Span, the display interface means, gradation information is displayed by a separate node GRAY Span. The advantage is clear and concise information, the disadvantage is Span node will be doubled. We can call it the mode [A]
2. If off, the gray scale information is output to the node native Span, the display interface means, and call information message native gray Span node, the protocol information and the like are mixed, the disadvantage is complex mixing information, the advantage of the number of nodes Span It will not grow. We can call it the [mode B]

# 启动和关闭调用链的灰度信息在Opentracing中以独立的Span节点输出，如果关闭，则灰度信息输出到原生的Span节点中。缺失则默认为true
spring.application.strategy.trace.opentracing.separate.span.enabled=true复制代码

Opentracing common class of operation - StrategyOpentracingOperation.java

• Tracer injection assembly of objects Opentracing
• opentracingInitialize method, providing services to the gateway and node initialization Span
  • The mode [A], tracer.buildSpan (...). ThreadLocal start () implement a new Span, and place it in the storage context StrategyOpentracingContext
  • Under [B] mode, you do not need to do any work
• opentracingHeader method, it is provided to the gateway call chain output gradation
  • [Mode] A case is first acquired from the object ThreadLocal StrategyOpentracingContext's Span, followed by the customizationMap (call chain custom parameter) Tag elements are put into the last main chain of the 11 gradation call parameters (by strategyContextHolder. getHeader (...) Gets) and more contextual information to put in Tag
  • The [B] mode, with mode [A] Similarly, the only difference is Tags.COMPONENT process, since the original has Span node with this information, it is not necessary to put the Tag
• opentracingLocal method of providing services to a call chain output gradation
  • [Mode] A case is first acquired from the object ThreadLocal StrategyOpentracingContext's Span, followed by the customizationMap (call chain custom parameter) Tag elements are put into the last main chain of the 11 gradation call parameters (by pluginAdapter. getXXX () Gets) and additional information into the context of the Tag
  • The [B] mode, with mode [A] Similarly, the only difference is Tags.COMPONENT process, since the original has Span node with this information, it is not necessary to put the Tag
• opentracingError method of providing services to a call chain abnormal output gradation
  • The mode [A], the object is first acquired from ThreadLocal StrategyOpentracingContext Span's, followed span.log (...) method to achieve error output
  • Under [B] mode, you do not need to do any work
• opentracingClear method, Span gradation reporting call chain and removals
  • Under [A] mode, first obtain Span object from the ThreadLocal StrategyOpentracingContext's, followed by span.finish () method implementation Span report, last StrategyOpentracingContext.clearCurrentContext () method implementation Span Clear
  • Under [B] mode, you do not need to do any work
• getCurrentSpan方法
  • [A] the mode, returns StrategyOpentracingContext.getCurrentContext (). Span new objects getSpan (), i.e. opentracingInitialize
  • [B] the mode, returns tracer.activeSpan (), i.e., the object native Span

public class StrategyOpentracingOperation {
    private static final Logger LOG = LoggerFactory.getLogger(StrategyOpentracingOperation.class);

    @Autowired
    protected PluginAdapter pluginAdapter;

    @Autowired
    protected StrategyContextHolder strategyContextHolder;

    @Autowired
    private Tracer tracer;

    @Value("${" + StrategyOpentracingConstant.SPRING_APPLICATION_STRATEGY_TRACE_OPENTRACING_ENABLED + ":false}")
    protected Boolean traceOpentracingEnabled;

    @Value("${" + StrategyOpentracingConstant.SPRING_APPLICATION_STRATEGY_TRACE_OPENTRACING_SEPARATE_SPAN_ENABLED + ":true}")
    protected Boolean traceOpentracingSeparateSpanEnabled;

    public void opentracingInitialize() {
        if (!traceOpentracingEnabled) {
            return;
        }

        if (!traceOpentracingSeparateSpanEnabled) {
            return;
        }

        Span span = tracer.buildSpan(DiscoveryConstant.SPAN_VALUE).start();
        StrategyOpentracingContext.getCurrentContext().setSpan(span);

        LOG.debug("Trace chain for Opentracing initialized...");
    }

    public void opentracingHeader(Map<String, String> customizationMap) {
        if (!traceOpentracingEnabled) {
            return;
        }

        Span span = getCurrentSpan();
        if (span == null) {
            LOG.error("Span not found in context to opentracing header");

            return;
        }

        if (MapUtils.isNotEmpty(customizationMap)) {
            for (Map.Entry<String, String> entry : customizationMap.entrySet()) {
                span.setTag(entry.getKey(), entry.getValue());
            }
        }

        if (traceOpentracingSeparateSpanEnabled) {
            span.setTag(Tags.COMPONENT.getKey(), DiscoveryConstant.TAG_COMPONENT_VALUE);
        }
        span.setTag(DiscoveryConstant.PLUGIN, DiscoveryConstant.PLUGIN_VALUE);
        span.setTag(DiscoveryConstant.TRACE_ID, span.context().toTraceId());
        span.setTag(DiscoveryConstant.SPAN_ID, span.context().toSpanId());
        span.setTag(DiscoveryConstant.N_D_SERVICE_GROUP, strategyContextHolder.getHeader(DiscoveryConstant.N_D_SERVICE_GROUP));
        ...

        String routeVersion = strategyContextHolder.getHeader(DiscoveryConstant.N_D_VERSION);
        if (StringUtils.isNotEmpty(routeVersion)) {
            span.setTag(DiscoveryConstant.N_D_VERSION, routeVersion);
        }
        ...

        LOG.debug("Trace chain information outputs to Opentracing...");
    }

    public void opentracingLocal(String className, String methodName, Map<String, String> customizationMap) {
        if (!traceOpentracingEnabled) {
            return;
        }

        Span span = getCurrentSpan();
        if (span == null) {
            LOG.error("Span not found in context to opentracing local");

            return;
        }

        if (MapUtils.isNotEmpty(customizationMap)) {
            for (Map.Entry<String, String> entry : customizationMap.entrySet()) {
                span.setTag(entry.getKey(), entry.getValue());
            }
        }

        if (traceOpentracingSeparateSpanEnabled) {
            span.setTag(Tags.COMPONENT.getKey(), DiscoveryConstant.TAG_COMPONENT_VALUE);
        }
        span.setTag(DiscoveryConstant.PLUGIN, DiscoveryConstant.PLUGIN_VALUE);
        span.setTag(DiscoveryConstant.CLASS, className);
        span.setTag(DiscoveryConstant.METHOD, methodName);
        span.setTag(DiscoveryConstant.TRACE_ID, span.context().toTraceId());
        span.setTag(DiscoveryConstant.SPAN_ID, span.context().toSpanId());
        span.setTag(DiscoveryConstant.N_D_SERVICE_GROUP, pluginAdapter.getGroup());
        ...

        String routeVersion = strategyContextHolder.getHeader(DiscoveryConstant.N_D_VERSION);
        if (StringUtils.isNotEmpty(routeVersion)) {
            span.setTag(DiscoveryConstant.N_D_VERSION, routeVersion);
        }
        ...

        LOG.debug("Trace chain information outputs to Opentracing...");
    }

    public void opentracingError(String className, String methodName, Throwable e) {
        if (!traceOpentracingEnabled) {
            return;
        }

        if (!traceOpentracingSeparateSpanEnabled) {
            return;
        }

        Span span = getCurrentSpan();
        if (span == null) {
            LOG.error("Span not found in context to opentracing error");

            return;
        }

        span.log(new ImmutableMap.Builder<String, Object>()
                .put(DiscoveryConstant.CLASS, className)
                .put(DiscoveryConstant.METHOD, methodName)
                .put(DiscoveryConstant.EVENT, Tags.ERROR.getKey())
                .put(DiscoveryConstant.ERROR_OBJECT, e)
                .build());

        LOG.debug("Trace chain error outputs to Opentracing...");
    }

    public void opentracingClear() {
        if (!traceOpentracingEnabled) {
            return;
        }

        if (!traceOpentracingSeparateSpanEnabled) {
            return;
        }

        Span span = getCurrentSpan();
        if (span != null) {
            span.finish();
        } else {
            LOG.error("Span not found in context to opentracing clear");
        }
        StrategyOpentracingContext.clearCurrentContext();

        LOG.debug("Trace chain context of Opentracing cleared...");
    }

    public Span getCurrentSpan() {
        return traceOpentracingSeparateSpanEnabled ? StrategyOpentracingContext.getCurrentContext().getSpan() : tracer.activeSpan();
    }

    public String getTraceId() {
        if (!traceOpentracingEnabled) {
            return null;
        }

        Span span = getCurrentSpan();
        if (span != null) {
            return span.context().toTraceId();
        }

        return null;
    }

    public String getSpanId() {
        if (!traceOpentracingEnabled) {
            return null;
        }

        Span span = getCurrentSpan();
        if (span != null) {
            return span.context().toSpanId();
        }

        return null;
    }
}复制代码

Opentracing Service Module

Source reference github.com/Nepxion/Dis...

Achieve OpenTracing extension of service, including configuration, tracer and other modules, the module focuses tracer, the other is relatively simple, does not enumerate

Opentracing tracking service class - DefaultServiceStrategyOpentracingTracer.java

• Inherit DefaultServiceStrategyTracer, and injected StrategyOpentracingOperation
• trace method in the first implementation opentracingInitialize initialization Span, which would allow the logic behind can get traceId and spanId from Span, perform grayscale output opentracingLocal implement the call chain services
• error method in the implementation of the grayscale call chain to achieve abnormal output opentracingError services
• release method in the call chain execution opentracingClear achieve gray and clear reporting of Span

public class DefaultServiceStrategyOpentracingTracer extends DefaultServiceStrategyTracer {
    @Autowired
    private StrategyOpentracingOperation strategyOpentracingOperation;

    @Override
    public void trace(ServiceStrategyTracerInterceptor interceptor, MethodInvocation invocation) {
        strategyOpentracingOperation.opentracingInitialize();

        super.trace(interceptor, invocation);

        strategyOpentracingOperation.opentracingLocal(interceptor.getMethod(invocation).getDeclaringClass().getName(), interceptor.getMethodName(invocation), getCustomizationMap());
    }

    @Override
    public void error(ServiceStrategyTracerInterceptor interceptor, MethodInvocation invocation, Throwable e) {
        super.error(interceptor, invocation, e);

        strategyOpentracingOperation.opentracingError(interceptor.getMethod(invocation).getDeclaringClass().getName(), interceptor.getMethodName(invocation), e);
    }

    @Override
    public void release(ServiceStrategyTracerInterceptor interceptor, MethodInvocation invocation) {
        super.release(interceptor, invocation);

        strategyOpentracingOperation.opentracingClear();
    }

    @Override
    public String getTraceId() {
        return strategyOpentracingOperation.getTraceId();
    }

    @Override
    public String getSpanId() {
        return strategyOpentracingOperation.getSpanId();
    }
}复制代码

Opentracing Spring Cloud Gateway模块

Source reference github.com/Nepxion/Dis...

OpenTracing achieve an extension of Spring Cloud Gateway, with the discovery-plugin-strategy-starter-service-opentracing module is similar to not enumerate

Opentracing Zuul module

Source reference github.com/Nepxion/Dis...

Achieve OpenTracing extension of Zuul, with the discovery-plugin-strategy-starter-service-opentracing module is similar to not enumerate

Instructions for use

Reference Example github.com/Nepxion/Dis...

Use

Uber Jaeger Opentracing output to an example embodiment will be described, the step is very simple

1. From pan.baidu.com/s/1i57rXaNK... get Jaeger-1.14.0.zip, after decompression run the Windows operating system jaeger.bat, Mac OS and Lunix your own research
2. After performing Postman calls, visit http: // localhost: 16686 View grayscale call chain
3. Gray call chain support WebMvc and WebFlux two ways to mark the words to identify GRAY

Switch control

Opentracing function call chain for opening and closing, the control needs to be done by the switch:

# 启动和关闭调用链。缺失则默认为false
spring.application.strategy.trace.enabled=true
# 启动和关闭调用链的Opentracing输出，支持F版或更高版本的配置，其它版本不需要该行配置。缺失则默认为false
spring.application.strategy.trace.opentracing.enabled=true
# 启动和关闭调用链的灰度信息在Opentracing中以独立的Span节点输出，如果关闭，则灰度信息输出到原生的Span节点中。缺失则默认为true
spring.application.strategy.trace.opentracing.separate.span.enabled=true复制代码

Optional features

Custom call chain context parameters to create (the class is not necessary), inherited DefaultStrategyTracerAdapter

// 自定义调用链上下文参数的创建
// 对于getTraceId和getSpanId方法，在Opentracing等调用链中间件引入的情况下，由调用链中间件决定，在这里定义不会起作用；在Opentracing等调用链中间件未引入的情况下，在这里定义才有效，下面代码中表示从Http Header中获取，并全链路传递
// 对于getCustomizationMap方法，表示输出到调用链中的定制化业务参数，可以同时输出到日志和Opentracing等调用链中间件，下面代码中表示从Http Header中获取，并全链路传递
public class MyStrategyTracerAdapter extends DefaultStrategyTracerAdapter {
    @Override
    public String getTraceId() {
        return StringUtils.isNotEmpty(strategyContextHolder.getHeader(DiscoveryConstant.TRACE_ID)) ? strategyContextHolder.getHeader(DiscoveryConstant.TRACE_ID) : StringUtils.EMPTY;
    }

    @Override
    public String getSpanId() {
        return StringUtils.isNotEmpty(strategyContextHolder.getHeader(DiscoveryConstant.SPAN_ID)) ? strategyContextHolder.getHeader(DiscoveryConstant.SPAN_ID) : StringUtils.EMPTY;
    }

    @Override
    public Map<String, String> getCustomizationMap() {
        return new ImmutableMap.Builder<String, String>()
                .put("mobile", StringUtils.isNotEmpty(strategyContextHolder.getHeader("mobile")) ? strategyContextHolder.getHeader("mobile") : StringUtils.EMPTY)
                .put("user", StringUtils.isNotEmpty(strategyContextHolder.getHeader("user")) ? strategyContextHolder.getHeader("user") : StringUtils.EMPTY)
                .build();
    }
}
复制代码

Called chain class configuration class @Bean create, built-in call chain cover frame class

@Bean
public StrategyTracerAdapter strategyTracerAdapter() {
    return new MyStrategyTracerAdapter();
}复制代码

Author

Renhao Jun, 10 years of open source experience, Github ID: @ HaojunRen, founder of the open source community Nepxion, Nacos Group Member, Spring Cloud Alibaba & Nacos & Sentinel Committer

Please contact me

Micro-letters, numbers and public documents

This article from the blog article multiple platforms OpenWrite release!