Spring Cloud config ter: config-server because the health check server and client-side service timeouts lead to blocking problems

springcloud online a problem when the config-server Rom git, micro-services cluster are slowly hang up.

In the entrance layer increases the tracking log:

org.springframework.cloud.config.server.environment.EnvironmentController.java

    @RequestMapping("/{name}/{profiles}/{label:.*}")
    public Environment labelled(@PathVariable String name, @PathVariable String profiles,
            @PathVariable String label) {
        if (name != null && name.contains("(_)")) {
            // "(_)" is uncommon in a git repo name, but "/" cannot be matched
            // by Spring MVC
            name = name.replace("(_)", "/");
        }
        if (label != null && label.contains("(_)")) {
            // "(_)" is uncommon in a git branch name, but "/" cannot be matched
            // by Spring MVC
            label = label.replace("(_)", "/");
        }
        StopWatch sw = new StopWatch("labelled");
        sw.start();
        logger.info("EnvironmentController.labelled()开始,name={},profiles={},label={}", name, profiles, label);
        Environment environment = this.repository.findOne(name, profiles, label);
        sw.stop();
        logger.info("EnvironmentController.labelled()结束,name={},profiles={},label={},耗时={}毫秒,耗时={}秒", name, profiles, label, sw.getTotalTimeMillis(), sw.getTotalTimeSeconds());
        return environment;
    }

Health check ConfigServerHealthIndicator.java increase the log entry:

@Override
    protected void doHealthCheck(Health.Builder builder) throws Exception {
        StopWatch sw = new StopWatch("doHealthCheck");
        sw.start();
        logger.info("ConfigServerHealthIndicator.doHealthCheck()开始,builder={}", builder);
        builder.up();
        List<Map<String, Object>> details = new ArrayList<>();
        for (String name : this.repositories.keySet()) {
            Repository repository = this.repositories.get(name);
            String application = (repository.getName() == null)? name : repository.getName();
            String profiles = repository.getProfiles();

            try {
                Environment environment = this.environmentRepository.findOne(application, profiles, repository.getLabel());

                HashMap<String, Object> detail = new HashMap<>();
                detail.put("name", environment.getName());
                detail.put("label", environment.getLabel());
                if (environment.getProfiles() != null && environment.getProfiles().length > 0) {
                    detail.put("profiles", Arrays.asList(environment.getProfiles()));
                }

                if (!CollectionUtils.isEmpty(environment.getPropertySources())) {
                    List<String> sources = new ArrayList<>();
                    for (PropertySource source : environment.getPropertySources()) {
                        sources.add(source.getName());
                    }
                    detail.put("sources", sources);
                }
                details.add(detail);
            } catch (Exception e) {
                HashMap<String, String> map = new HashMap<>();
                map.put("application", application);
                map.put("profiles", profiles);
                builder.withDetail("repository", map);
                builder.down(e);
                return;
            }
        }
        builder.withDetail("repositories", details);
        sw.stop();
        logger.info("ConfigServerHealthIndicator.doHealthCheck () ends, Processed ms = {}, {} = Processed seconds, Builder = {}" , sw.getTotalTimeMillis (), sw.getTotalTimeSeconds (), Builder); 
    }

After the time-consuming statistical analysis of log and found that EnvironmentController and ConfigServerHealthIndicator call too many times, both calls will eventually call JGitEnvironmentRepository.fetch () method, this method will go fetch request git, the timeout period is about 5 seconds.

Due to the excessive number of requests, service requests, however, the thread is blocked for a long time.

analysis:

1, call EnvironmentController that each micro-initiated service module, and why?

2, call ConfigServerHealthIndicator is config-server health check, can alleviate the problem by setting the interval to check.

    consul:
      host: 10.200.110.100
      port: 8500
      enabled: true
      discovery:
        enabled: true
        hostname: 10.200.110.100
        healthCheckInterval: 30s
        queryPassing: true

 

When EnvironmentController request with the health check on the client side config-server-initiated calls. Look Source:

After each client connection to the registry to get to the center of the configuration example, it calls the code above logic configured to obtain from the data center to the Environment variables, the on-line environment, encountered a problem, view the log and found that this logic is kept call every 20 seconds will be called once, application's name to app, to know whether Config Server through a health indicator to detect configuration EnvironmentRepository working properly by looking SpringCloudConfig official documents. By default, the application will ask for a name for the app to configure EnvironmentRepository, EnvironmentRepository examples responded default configuration. That is when the health monitor turned on by default, will not stop calling findOne to detect configuration is available, whether there will be an exception,

this code is org.springframework.cloud.config.server.config.ConfigServerHealthIndicator class initialization name for the application code name for the app

@ConfigurationProperties("spring.cloud.config.server.health")
public class ConfigServerHealthIndicator extends AbstractHealthIndicator {

    private EnvironmentRepository environmentRepository;

    private Map<String, Repository> repositories = new LinkedHashMap<>();

    public ConfigServerHealthIndicator(EnvironmentRepository environmentRepository) {
        this.environmentRepository = environmentRepository;
    }

    @PostConstruct
    public void init() {
        if (this.repositories.isEmpty()) {
            this.repositories.put("app", new Repository());
        }
    }
       //...
}

If you want to stop out such detection spring.cloud.config.server.health.enabled = false can be configured to turn this feature off.

Look Source: org.springframework.cloud.config.client. ConfigClientAutoConfiguration.java

@Configuration
public class ConfigClientAutoConfiguration {
    @Configuration
    @ConditionalOnClass(HealthIndicator.class)
    @ConditionalOnBean(ConfigServicePropertySourceLocator.class)
    @ConditionalOnProperty(value = "health.config.enabled", matchIfMissing = true)
    protected static class ConfigServerHealthIndicatorConfiguration {

        @Bean
        public ConfigServerHealthIndicator configServerHealthIndicator(
                ConfigServicePropertySourceLocator locator,
                ConfigClientHealthProperties properties, Environment environment) {
            return new ConfigServerHealthIndicator(locator, environment, properties);
        }
    }
//...