12 调用链监控Sleuth（上）

调用链监控原理剖析

前面的微服务架构基本都实现了，通过consul实现了服务发现组件和配置服务器，外面的请求通过网关打进来，微服务之间通过轻量级通信机制，包括http和rabbitmq，微服务之间有一定的认证与授权机制，从而保证安全，对于高并发的应用，通过集成Resilience4j，从而提升应用的可用性。

但是一直没有探讨，如果应用发现问题，如何快速进行定位问题呢？

定位的两点需求：

跨微服务的API调用发生异常，要求快速定位（比如5分钟以内）出问题出在哪里，该怎么办？
跨微服务的API调用发生性能瓶颈，要求迅速定位（比如5分钟以内）出系统瓶颈，该怎么办？

如果采用传统的方式是很难定位的。

利用调用链监控工具。

调用链监控工具是微服务不可或缺的组件，下面来分析一下调用链监控原理。

在课程微服务调用用户微服务的课程中，可能有3个节点会出现问题

课程微服务
网络
用户微服务

要想快速定位只需要一个数据库一张表就够了。
在这里插入图片描述
一次调用可以分为四个阶段，第一client send,表示课程微服务向用户微服务发起请求的那个瞬间，server receive代表用户微服务收到请求的那个瞬间，server sent表示用户微服务发送响应的那个瞬间，client receive表示课程微服务接收响应的那个瞬间。这个库里面有一张表表trace，通过trace表就可以简单实现调用链监控工具了。

如果一次调用只有两条数据，表示用户微服务没有调用成功。如果只有一条数据，则标识用户微服务或者网络有异常，那如何分析性能呢？通过timestamp可以获取请求和响应的时间。针对这些数据可以计算各种数据，分析出性能瓶颈在哪里。

因此调用链监控工具就是一个存储和界面。市面上实现的调用链监控工具不一定是这么实现的，但是基本原理都是相通的。

整合Sleuth

Sleuth是什么？

Sleuth是一个Spring Cloud的分布式跟踪解决方案。Sleuth集成在每个微服务上，负责产生监控数据。

Sleuth术语

Span(跨度)：Sleuth的基本工作单元，它用一个64位的id唯一标识。除ID外，span还包含其他数据，例如描述、时间戳事件、键值对的注解（标签）、spanID、span父ID等

trace(跟踪)：一组span组成的树状结构称为trace

Annotation(标注)：

CS（Client Sent 客户端发送）：客户端发起一个请求，该annotation描述了span的开始

SR（Server Received 服务器端接收）：服务器端获得请求并准备处理它

SS （Server Sent 服务器端发送）：该annotation表明完成请求处理（当响应发回客户端时）

CR （Client Received 客户端接收）：span结束的标识。客户端成功接收到服务器端的响应。

整合Sleuth

用户微服务添加依赖：

<dependency>
	<groupId>org.springframework.cloud</groupId>
	<artifactId>spring-cloud-starter-sleuth</artifactId>
</dependency>

用户微服务修改配置：

logging:
  level:
    org.springframework.web: debug

启动用户微服务：

2020-03-11 19:59:04.158  INFO [ms-user,,,] 10120 --- [  restartedMain] o.s.b.devtools.restart.ChangeableUrls    : The Class-Path manifest attribute in D:\maven\repo\org\glassfish\jaxb\jaxb-runtime\2.3.2\jaxb-runtime-2.3.2.jar referenced one or more files that do not exist: file:/D:/maven/repo/org/glassfish/jaxb/jaxb-runtime/2.3.2/jakarta.xml.bind-api-2.3.2.jar,file:/D:/maven/repo/org/glassfish/jaxb/jaxb-runtime/2.3.2/txw2-2.3.2.jar,file:/D:/maven/repo/org/glassfish/jaxb/jaxb-runtime/2.3.2/istack-commons-runtime-3.0.8.jar,file:/D:/maven/repo/org/glassfish/jaxb/jaxb-runtime/2.3.2/stax-ex-1.8.1.jar,file:/D:/maven/repo/org/glassfish/jaxb/jaxb-runtime/2.3.2/FastInfoset-1.2.16.jar,file:/D:/maven/repo/org/glassfish/jaxb/jaxb-runtime/2.3.2/jakarta.activation-api-1.2.1.jar
2020-03-11 19:59:04.161  INFO [ms-user,,,] 10120 --- [  restartedMain] .e.DevToolsPropertyDefaultsPostProcessor : Devtools property defaults active! Set 'spring.devtools.add-properties' to 'false' to disable
2020-03-11 19:59:05.039  INFO [ms-user,,,] 10120 --- [  restartedMain] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.retry.annotation.RetryConfiguration' of type [org.springframework.retry.annotation.RetryConfiguration$$EnhancerBySpringCGLIB$$287db866] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.2.2.RELEASE)

2020-03-11 19:59:06.965  INFO [ms-user,,,] 10120 --- [  restartedMain] com.cloud.msuser.MsUserApplication       : The following profiles are active: dev
2020-03-11 19:59:08.636  INFO [ms-user,,,] 10120 --- [  restartedMain] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.
2020-03-11 19:59:08.773  INFO [ms-user,,,] 10120 --- [  restartedMain] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 129ms. Found 2 JPA repository interfaces.
2020-03-11 19:59:09.105  WARN [ms-user,,,] 10120 --- [  restartedMain] o.s.boot.actuate.endpoint.EndpointId     : Endpoint ID 'service-registry' contains invalid characters, please migrate to a valid format.
2020-03-11 19:59:09.435  INFO [ms-user,,,] 10120 --- [  restartedMain] o.s.cloud.context.scope.GenericScope     : BeanFactory id=7bf32038-65f8-347f-abf0-7347ad504552
2020-03-11 19:59:09.622  INFO [ms-user,,,] 10120 --- [  restartedMain] faultConfiguringBeanFactoryPostProcessor : No bean named 'errorChannel' has been explicitly defined. Therefore, a default PublishSubscribeChannel will be created.
2020-03-11 19:59:09.637  INFO [ms-user,,,] 10120 --- [  restartedMain] faultConfiguringBeanFactoryPostProcessor : No bean named 'taskScheduler' has been explicitly defined. Therefore, a default ThreadPoolTaskScheduler will be created.
2020-03-11 19:59:09.645  INFO [ms-user,,,] 10120 --- [  restartedMain] faultConfiguringBeanFactoryPostProcessor : No bean named 'integrationHeaderChannelRegistry' has been explicitly defined. Therefore, a default DefaultHeaderChannelRegistry will be created.
2020-03-11 19:59:09.706  INFO [ms-user,,,] 10120 --- [  restartedMain] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.retry.annotation.RetryConfiguration' of type [org.springframework.retry.annotation.RetryConfiguration$$EnhancerBySpringCGLIB$$287db866] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-03-11 19:59:10.017  INFO [ms-user,,,] 10120 --- [  restartedMain] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration' of type [org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-03-11 19:59:10.063  INFO [ms-user,,,] 10120 --- [  restartedMain] trationDelegate$BeanPostProcessorChecker : Bean 'integrationChannelResolver' of type [org.springframework.integration.support.channel.BeanFactoryChannelResolver] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-03-11 19:59:10.072  INFO [ms-user,,,] 10120 --- [  restartedMain] trationDelegate$BeanPostProcessorChecker : Bean 'integrationDisposableAutoCreatedBeans' of type [org.springframework.integration.config.annotation.Disposables] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-03-11 19:59:10.150  INFO [ms-user,,,] 10120 --- [  restartedMain] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.integration.config.IntegrationManagementConfiguration' of type [org.springframework.integration.config.IntegrationManagementConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-03-11 19:59:10.190  INFO [ms-user,,,] 10120 --- [  restartedMain] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.integration.IntegrationAutoConfiguration$IntegrationJmxConfiguration' of type [org.springframework.boot.autoconfigure.integration.IntegrationAutoConfiguration$IntegrationJmxConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-03-11 19:59:10.208  INFO [ms-user,,,] 10120 --- [  restartedMain] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.jmx.JmxAutoConfiguration' of type [org.springframework.boot.autoconfigure.jmx.JmxAutoConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-03-11 19:59:10.218  INFO [ms-user,,,] 10120 --- [  restartedMain] trationDelegate$BeanPostProcessorChecker : Bean 'mbeanServer' of type [com.sun.jmx.mbeanserver.JmxMBeanServer] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-03-11 19:59:12.035  INFO [ms-user,,,] 10120 --- [  restartedMain] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8081 (http)
2020-03-11 19:59:12.122  INFO [ms-user,,,] 10120 --- [  restartedMain] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2020-03-11 19:59:12.123  INFO [ms-user,,,] 10120 --- [  restartedMain] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.29]
2020-03-11 19:59:12.458  INFO [ms-user,,,] 10120 --- [  restartedMain] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2020-03-11 19:59:12.458 DEBUG [ms-user,,,] 10120 --- [  restartedMain] o.s.web.context.ContextLoader            : Published root WebApplicationContext as ServletContext attribute with name [org.springframework.web.context.WebApplicationContext.ROOT]
2020-03-11 19:59:12.458  INFO [ms-user,,,] 10120 --- [  restartedMain] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 5468 ms
2020-03-11 19:59:12.718  WARN [ms-user,,,] 10120 --- [  restartedMain] c.n.c.sources.URLConfigurationSource     : No URLs will be polled as dynamic configuration sources.
2020-03-11 19:59:12.718  INFO [ms-user,,,] 10120 --- [  restartedMain] c.n.c.sources.URLConfigurationSource     : To enable URLs as dynamic configuration sources, define System property archaius.configurationSource.additionalUrls or make config.properties available on classpath.
2020-03-11 19:59:12.784  INFO [ms-user,,,] 10120 --- [  restartedMain] c.netflix.config.DynamicPropertyFactory  : DynamicPropertyFactory is initialized with configuration sources: com.netflix.config.ConcurrentCompositeConfiguration@2862d8b8
2020-03-11 19:59:15.591  INFO [ms-user,,,] 10120 --- [  restartedMain] o.s.s.c.ThreadPoolTaskScheduler          : Initializing ExecutorService 'taskScheduler'
2020-03-11 19:59:16.893  INFO [ms-user,,,] 10120 --- [  restartedMain] o.hibernate.jpa.internal.util.LogHelper  : HHH000204: Processing PersistenceUnitInfo [name: default]
2020-03-11 19:59:18.098  INFO [ms-user,,,] 10120 --- [  restartedMain] org.hibernate.Version                    : HHH000412: Hibernate Core {5.4.9.Final}
2020-03-11 19:59:18.998  INFO [ms-user,,,] 10120 --- [  restartedMain] o.hibernate.annotations.common.Version   : HCANN000001: Hibernate Commons Annotations {5.1.0.Final}
2020-03-11 19:59:19.441  INFO [ms-user,,,] 10120 --- [  restartedMain] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Starting...
2020-03-11 19:59:21.361  INFO [ms-user,,,] 10120 --- [  restartedMain] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Start completed.
2020-03-11 19:59:21.416  INFO [ms-user,,,] 10120 --- [  restartedMain] org.hibernate.dialect.Dialect            : HHH000400: Using dialect: org.hibernate.dialect.MySQL57Dialect
2020-03-11 19:59:22.994  INFO [ms-user,,,] 10120 --- [  restartedMain] o.h.e.t.j.p.i.JtaPlatformInitiator       : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform]
2020-03-11 19:59:23.004  INFO [ms-user,,,] 10120 --- [  restartedMain] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default'
2020-03-11 19:59:24.011  WARN [ms-user,,,] 10120 --- [  restartedMain] c.n.c.sources.URLConfigurationSource     : No URLs will be polled as dynamic configuration sources.
2020-03-11 19:59:24.012  INFO [ms-user,,,] 10120 --- [  restartedMain] c.n.c.sources.URLConfigurationSource     : To enable URLs as dynamic configuration sources, define System property archaius.configurationSource.additionalUrls or make config.properties available on classpath.
2020-03-11 19:59:24.204  WARN [ms-user,,,] 10120 --- [  restartedMain] JpaBaseConfiguration$JpaWebConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning
2020-03-11 19:59:24.495  INFO [ms-user,,,] 10120 --- [  restartedMain] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService 'applicationTaskExecutor'
2020-03-11 19:59:24.542 DEBUG [ms-user,,,] 10120 --- [  restartedMain] s.w.s.m.m.a.RequestMappingHandlerAdapter : ControllerAdvice beans: 0 @ModelAttribute, 0 @InitBinder, 1 RequestBodyAdvice, 1 ResponseBodyAdvice
2020-03-11 19:59:24.736 DEBUG [ms-user,,,] 10120 --- [  restartedMain] s.w.s.m.m.a.RequestMappingHandlerMapping : 5 mappings in 'requestMappingHandlerMapping'
2020-03-11 19:59:24.931 DEBUG [ms-user,,,] 10120 --- [  restartedMain] o.s.w.s.handler.SimpleUrlHandlerMapping  : Patterns [/webjars/**, /**] in 'resourceHandlerMapping'
2020-03-11 19:59:24.996 DEBUG [ms-user,,,] 10120 --- [  restartedMain] .m.m.a.ExceptionHandlerExceptionResolver : ControllerAdvice beans: 1 @ExceptionHandler, 1 ResponseBodyAdvice
2020-03-11 19:59:26.359  INFO [ms-user,,,] 10120 --- [  restartedMain] o.s.b.d.a.OptionalLiveReloadServer       : LiveReload server is running on port 35729
2020-03-11 19:59:27.212  WARN [ms-user,,,] 10120 --- [  restartedMain] ockingLoadBalancerClientRibbonWarnLogger : You already have RibbonLoadBalancerClient on your classpath. It will be used by default. As Spring Cloud Ribbon is in maintenance mode. We recommend switching to BlockingLoadBalancerClient instead. In order to use it, set the value of `spring.cloud.loadbalancer.ribbon.enabled` to `false` or remove spring-cloud-starter-netflix-ribbon from your project.
2020-03-11 19:59:27.437  INFO [ms-user,,,] 10120 --- [  restartedMain] o.s.s.c.ThreadPoolTaskScheduler          : Initializing ExecutorService 'configWatchTaskScheduler'
2020-03-11 19:59:27.463  INFO [ms-user,,,] 10120 --- [  restartedMain] o.s.s.c.ThreadPoolTaskScheduler          : Initializing ExecutorService 'catalogWatchTaskScheduler'
2020-03-11 19:59:27.772  INFO [ms-user,,,] 10120 --- [  restartedMain] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 21 endpoint(s) beneath base path '/actuator'
2020-03-11 19:59:28.080  INFO [ms-user,,,] 10120 --- [  restartedMain] o.s.c.s.m.DirectWithAttributesChannel    : Channel 'ms-user-1.input' has 1 subscriber(s).
2020-03-11 19:59:28.238  INFO [ms-user,,,] 10120 --- [  restartedMain] o.s.i.monitor.IntegrationMBeanExporter   : Registering MessageChannel input
2020-03-11 19:59:28.366  INFO [ms-user,,,] 10120 --- [  restartedMain] o.s.i.monitor.IntegrationMBeanExporter   : Registering MessageChannel errorChannel
2020-03-11 19:59:28.439  INFO [ms-user,,,] 10120 --- [  restartedMain] o.s.i.monitor.IntegrationMBeanExporter   : Registering MessageChannel nullChannel
2020-03-11 19:59:28.475  INFO [ms-user,,,] 10120 --- [  restartedMain] o.s.i.monitor.IntegrationMBeanExporter   : Registering MessageHandler org.springframework.cloud.stream.binding.StreamListenerMessageHandler@387a1f93
2020-03-11 19:59:28.599  INFO [ms-user,,,] 10120 --- [  restartedMain] o.s.i.monitor.IntegrationMBeanExporter   : Registering MessageHandler errorLogger
2020-03-11 19:59:28.714  INFO [ms-user,,,] 10120 --- [  restartedMain] o.s.i.endpoint.EventDrivenConsumer       : Adding {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
2020-03-11 19:59:28.714  INFO [ms-user,,,] 10120 --- [  restartedMain] o.s.i.channel.PublishSubscribeChannel    : Channel 'ms-user-1.errorChannel' has 1 subscriber(s).
2020-03-11 19:59:28.714  INFO [ms-user,,,] 10120 --- [  restartedMain] o.s.i.endpoint.EventDrivenConsumer       : started bean '_org.springframework.integration.errorLogger'
2020-03-11 19:59:29.936  INFO [ms-user,,,] 10120 --- [  restartedMain] c.s.b.r.p.RabbitExchangeQueueProvisioner : declaring queue for inbound: lesson-buy.g1, bound to: lesson-buy
2020-03-11 19:59:29.940  INFO [ms-user,,,] 10120 --- [  restartedMain] o.s.a.r.c.CachingConnectionFactory       : Attempting to connect to: [192.168.99.100:5672]
2020-03-11 19:59:50.942  INFO [ms-user,,,] 10120 --- [  restartedMain] o.s.a.r.c.CachingConnectionFactory       : Attempting to connect to: [192.168.99.100:5672]

ZipKin搭建与整合

ZipKin是什么

ZipKin是Twitter开源的分布式跟踪系统，主要用来收集系统的时序数据，从而追踪系统的调用问题

GitHub地址： https://github.com/openzipkin/zipkin

Zipkin is a distributed tracing system. It helps gather timing data needed to troubleshoot latency problems in service architectures. Features include both the collection and lookup of this data.

通俗的说，zipkin就是前面所说的界面和数据库。

搭建ZipKinServer

curl -sSL https://zipkin.io/quickstart.sh | bash -s
java -jar zipkin.jar

通过docker搭建

docker run -d -p 9411:9411 openzipkin/zipkin

通过http://your_host:9411/zipkin/即可访问
在这里插入图片描述

docker pull openzipkin/zipkin:2.19.0
docker run -d --name zipkin -p 9411:9411 openzipkin/zipkin:2.19.0

整合ZipKin

添加依赖

<dependency>
	<groupId>org.springframework.cloud</groupId>
	<artifactId>spring-cloud-starter-zipkin</artifactId>
</dependency>

在这里插入图片描述
通过分析依赖，发现zipkin中包含了sleuth的依赖，没有必要手动添加sleuth的依赖了。

添加配置：

spring :
  zipkin:
    # 指定zipkin server的地址
    base-url: http://localhost:9411
    sender:
      # 指定用什么方式上报数据给zipkin server
      # web表示用http 还可以利用activemq rabbit kafka
      # 有一个小坑
      type: web
  sleuth:
    sampler:
      # 配置数据的采样率 默认值0.1(10%)
      probability: 1

启动用户微服务

会出现如下的错误，

java.util.concurrent.TimeoutException: null
	at java.util.concurrent.FutureTask.get(FutureTask.java:205) ~[na:1.8.0_121]

这是一个bug https://github.com/spring-cloud/spring-cloud-sleuth/issues/1471

不影响使用

访问zipkin首页

通过postman发请求
通过http://localhost:8081/login获取一个token
通过http://localhost:8081/users/1调用用户微服务
在这里插入图片描述

发现此时只有两条数据因为这时候只是针对用户微服务单独发了一个请求，不涉及两个微服务之间的调用

可以点击页面右上角下载json数据：

[
  {
    "traceId": "c5560583b60f027e",
    "id": "c5560583b60f027e",
    "kind": "SERVER",
    "name": "get /users/{id}",
    "timestamp": 1583932795421177,
    "duration": 114088,
    "localEndpoint": {
      "serviceName": "ms-user",
      "ipv4": "192.168.7.201"
    },
    "remoteEndpoint": {
      "ipv6": "::1",
      "port": 51715
    },
    "tags": {
      "http.method": "GET",
      "http.path": "/users/1",
      "mvc.controller.class": "UserController",
      "mvc.controller.method": "findById"
    }
  }
]

ZipKin集成测试

为课程微服务和网关整合zipkin

拷贝依赖和配置

<dependency>
	<groupId>org.springframework.cloud</groupId>
	<artifactId>spring-cloud-starter-zipkin</artifactId>
</dependency>

  zipkin:
    # 指定zipkin server的地址
    base-url: http://localhost:9411
    sender:
      # 指定用什么方式上报数据给zipkin server
      # web表示用http 还可以利用activemq rabbit kafka
      # 有一个小坑
      type: web
  sleuth:
    sampler:
      # 配置数据的采样率 默认值0.1(10%)
      probability: 1

启动各个微服务
在这里插入图片描述
通过网关请求课程微服务

http://localhost:8040/lesssons/buy/1
在这里插入图片描述
查看gateway:

详细的JSON数据：

[
  {
    "traceId": "7f7b9a1b903a1c92",
    "parentId": "88e26713e7f1d5a1",
    "id": "c86eef59144012b8",
    "kind": "SERVER",
    "name": "get /users/{id}",
    "timestamp": 1583933301555149,
    "duration": 10416,
    "localEndpoint": {
      "serviceName": "ms-user",
      "ipv4": "192.168.7.201"
    },
    "remoteEndpoint": {
      "ipv4": "192.168.7.201",
      "port": 52276
    },
    "tags": {
      "http.method": "GET",
      "http.path": "/users/3",
      "mvc.controller.class": "UserController",
      "mvc.controller.method": "findById"
    },
    "shared": true
  },
  {
    "traceId": "7f7b9a1b903a1c92",
    "parentId": "cc88faf9c632541b",
    "id": "75eaa64e48af19a6",
    "kind": "PRODUCER",
    "name": "send",
    "timestamp": 1583933301706230,
    "duration": 384551,
    "localEndpoint": {
      "serviceName": "ms-user",
      "ipv4": "192.168.7.201"
    },
    "remoteEndpoint": {
      "serviceName": "broker"
    },
    "tags": {
      "channel": "input"
    }
  },
  {
    "traceId": "7f7b9a1b903a1c92",
    "parentId": "7f7b9a1b903a1c92",
    "id": "e460c23da789a7a6",
    "kind": "CLIENT",
    "name": "get",
    "timestamp": 1583933301375632,
    "duration": 515376,
    "localEndpoint": {
      "serviceName": "ms-gateway",
      "ipv4": "192.168.7.201"
    },
    "tags": {
      "http.method": "GET",
      "http.path": "/lesssons/buy/1"
    }
  },
  {
    "traceId": "7f7b9a1b903a1c92",
    "id": "7f7b9a1b903a1c92",
    "kind": "CLIENT",
    "name": "get",
    "timestamp": 1583933301380069,
    "duration": 511321,
    "localEndpoint": {
      "serviceName": "ms-gateway",
      "ipv4": "192.168.7.201"
    },
    "tags": {
      "http.method": "GET",
      "http.path": "/"
    }
  },
  {
    "traceId": "7f7b9a1b903a1c92",
    "parentId": "88e26713e7f1d5a1",
    "id": "c86eef59144012b8",
    "kind": "CLIENT",
    "name": "get",
    "timestamp": 1583933301551426,
    "duration": 18188,
    "localEndpoint": {
      "serviceName": "ms-class",
      "ipv4": "192.168.7.201"
    },
    "remoteEndpoint": {
      "ipv4": "192.168.7.201",
      "port": 8081
    },
    "tags": {
      "http.method": "GET",
      "http.path": "/3"
    }
  },
  {
    "traceId": "7f7b9a1b903a1c92",
    "parentId": "e460c23da789a7a6",
    "id": "88e26713e7f1d5a1",
    "kind": "CLIENT",
    "name": "get",
    "timestamp": 1583933301542872,
    "duration": 29375,
    "localEndpoint": {
      "serviceName": "ms-class",
      "ipv4": "192.168.7.201"
    },
    "tags": {
      "http.method": "GET",
      "http.path": "/users/3"
    }
  },
  {
    "traceId": "7f7b9a1b903a1c92",
    "parentId": "e460c23da789a7a6",
    "id": "cc88faf9c632541b",
    "kind": "PRODUCER",
    "name": "send",
    "timestamp": 1583933301645201,
    "duration": 51070,
    "localEndpoint": {
      "serviceName": "ms-class",
      "ipv4": "192.168.7.201"
    },
    "remoteEndpoint": {
      "serviceName": "broker"
    },
    "tags": {
      "channel": "output"
    },
    "shared": true
  },
  {
    "traceId": "7f7b9a1b903a1c92",
    "parentId": "7f7b9a1b903a1c92",
    "id": "e460c23da789a7a6",
    "kind": "SERVER",
    "name": "get /lesssons/buy/{id}",
    "timestamp": 1583933301382136,
    "duration": 507161,
    "localEndpoint": {
      "serviceName": "ms-class",
      "ipv4": "192.168.7.201"
    },
    "remoteEndpoint": {
      "ipv6": "::1"
    },
    "tags": {
      "http.method": "GET",
      "http.path": "/lesssons/buy/1",
      "mvc.controller.class": "LessonController",
      "mvc.controller.method": "buyById"
    },
    "shared": true
  }
]