After migrating my application (stack spring boot, camel, hystrix) to spring boot 2 from spring boot 1.5. I can't get the hystrix metrics to show up in /actuator/prometheus.
As many solutions and tutorials suggest, I've made sure I had the following dependencies
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
<version>${micrometer-version}</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-core</artifactId>
<version>${micrometer-version}</version>
</dependency>
And added the following configuration class, which I'm sure it's being instantiated, as I've checked in spring boot autoconfiguration logs:
import io.micrometer.core.instrument.binder.hystrix.HystrixMetricsBinder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration
public class HystrixMetricsConfig {
@Bean
HystrixMetricsBinder registerHystrixMetricsBinder() {
return new HystrixMetricsBinder();
}
}
To avoid assuming such tutorials/answers were correct by definition, I've set up a simple spring boot project from scratch with a simple controller, hystrix and the above dependencies, and metrics do appear at /actuator/prometheus, without additional steps.
My application has more dependencies that the simple project, so I think something else might be overriding/disabling the metrics binding.
[...]
<dependencies>
<!-- Spring Boot -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-hystrix</artifactId>
<exclusions>
<exclusion>
<groupId>om.netflix.archaius</groupId>
<artifactId>archaius-core</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-thymeleaf</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-config-server</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-web</artifactId>
</dependency>
<dependency>
<groupId>org.jolokia</groupId>
<artifactId>jolokia-core</artifactId>
</dependency>
<!-- Camel -->
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-spring-security</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-swagger-java-starter</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-servlet-starter</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-http4</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-spring</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-hystrix-starter</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-metrics</artifactId>
</dependency>
<!-- Spring Framework Caching Support -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-cache</artifactId>
</dependency>
<dependency>
<groupId>com.hazelcast</groupId>
<artifactId>hazelcast</artifactId>
</dependency>
<dependency>
<groupId>com.hazelcast</groupId>
<artifactId>hazelcast-spring</artifactId>
</dependency>
<!-- logging -->
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-core</artifactId>
</dependency>
<dependency>
<groupId>net.logstash.logback</groupId>
<artifactId>logstash-logback-encoder</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.ws.security</groupId>
<artifactId>wss4j</artifactId>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</dependency>
<!-- Test -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<scope>provided</scope>
</dependency>
<!-- stream -->
<dependency>
<groupId>com.netflix.archaius</groupId>
<artifactId>archaius-core</artifactId>
<version>0.7.6</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-core</artifactId>
</dependency>
<dependency>
<groupId>org.reactivestreams</groupId>
<artifactId>reactive-streams</artifactId>
<version>1.0.2</version>
</dependency>
</dependencies>
[...]
More than being told what the actual problem is, I'm struggling trying to understand how to troubleshoot such issues. Is there any micrometer/Spring boot log (other than autoconfiguration report) that could be activated to understand what's going on ?
After having followed the whole custom hystrix metrics binding procedure, I've ended up discovering everything was getting bound correctly, at least in principle.
I've then tried to trigger a breakpoint at the first execution of an hystrix command to check what was going on with the registry and the publisher. Placing a breakpoint in the method
HystrixMetricsPublisherThreadPool getPublisherForThreadPool(HystrixThreadPoolKey threadPoolKey, HystrixThreadPoolMetrics metrics, HystrixThreadPoolProperties properties)
of HystrixMetricsPublisherFactory class, I found out that the HystrixPlugins instance was different with respect to the the moment in which the publisher was set. After inspecting all the code of the application, I found out that, to set a custom event notifier, the instance was getting reset.
@EventListener(ApplicationReadyEvent.class)
public void doAfterStartup() {
Hystrix.reset();
registerCustomHystrixEventNotifier(circuitBreakerHystrixEventNotifier);
logger.info("hello world, application started up");
}
I've then modified the method in order to register the event notifier, without resetting what the autoconfiguration configured before, and hystrix metrics are now showing up in the prometheus endpoint.