When multiple microservices are called, suppose microservice A calls microservice B and microservice C, and microservice B and microservice C call other microservices. This is the so-called "fan-out". If the call response time of a microservice on the fan-out link is too long or unavailable, the call to microservice A will occupy more and more system resources, causing the system to crash. There will be an avalanche of services.

The so-called "avalanche effect" for high-traffic applications where a single backend dependency can saturate all resources on all servers within seconds. Worse than failing, these applications can also cause increased latency between services, straining backup queues, threads and other system resources, leading to more cascading failures throughout the system. These all represent the need to isolate and manage failures and latencies so that the failure of a single dependency cannot bring down an entire application or system.

In the microservice architecture, we split the system into service units one by one, and the applications of each unit depend on each other through service registration and subscription. Since each unit runs in a different process, and the dependencies are executed through remote calls, there may be call failures or delays due to network reasons or the problems of the dependent services themselves, and these problems will directly cause delays in the external services of the caller. If the caller’s requests continue to increase at this time, there will be a backlog of tasks due to waiting for the response of the failed dependent party, and thread resources cannot be released, which eventually leads to the paralysis of its own services, and even the spread of faults eventually leads to the paralysis of the entire system. If such a structure has such serious hidden dangers, it will be more unstable than the traditional structure, as shown in the figure below.

2. Solutions

There are usually ways to solve the catastrophic avalanche effect: downgrade, isolation, circuit breaker, request caching, and request merging. In order to solve the above existing problems, a series of service protection mechanisms such as circuit breakers have been produced.

' The "circuit breaker" itself is a switching device. When a service unit fails, through the fault monitoring of the circuit breaker (similar to blowing a fuse), an expected and processable alternative response (FallBack) is returned to the caller, instead of waiting for a long time or throwing an exception that the caller cannot handle. This ensures that the thread of the service caller will not be occupied for a long time and unnecessarily, thereby avoiding the spread of faults in the distributed system, or even an avalanche .

3. Introduction to Hystrix

Hystrix is an open source library used to deal with delay and fault tolerance of distributed systems. In distributed systems, many dependencies will inevitably fail to call, such as timeouts, exceptions, etc. Hystrix can ensure that in the case of a dependency problem, it will not cause the overall service to fail , avoid cascading failures , and improve the elasticity of distributed systems. Hystrix implements powerful functions such as service degradation , service fusing , thread isolation , request caching , request merging, and service monitoring . The goal is to provide greater fault tolerance to latency and failures by controlling the nodes that access remote systems, services, and third-party libraries.

4. Hystrix service fuse

When a microservice of the fanout link is unavailable or the response time is too long, the service will be degraded , and then the call of the microservice of the node will be cut off, and an "error" response message will be returned quickly . When it is detected that the microservice call response of the node is normal, the call link is resumed. Therefore, the fuse mechanism is a microservice link protection mechanism to deal with the avalanche effect.

When the proportion of abnormal requests (request timeout, network failure, service exception, etc.) reaches the threshold within a certain period of time, the fuse will be activated. Once the fuse is activated, it will stop calling the specific service logic and quickly return the backing data through fallback to ensure the integrity of the service chain.

In the SpringCloud framework, the fuse mechanism is implemented through Hystrix. Hystrix will monitor the status of calls between microservices. When the failed calls reach a certain threshold, the default is that 20 failed calls within 5 seconds will start the fuse mechanism. The annotation of the fuse mechanism is @HystrixCommand.

The circuit breaker has an automatic recovery mechanism. For example, when the circuit breaker is activated, a new request is sent to the service provider every 5 seconds. If the service can be executed normally and the result is returned, the circuit breaker will be closed and the service will be restored. If the call still fails, the backing data will continue to be returned, and the fuse will remain on.

Service degradation situations include: program exception, timeout, service fuse triggering degradation, thread pool/semaphore fullness will also trigger degradation

Service degradation, service interruption, and service current limiting.

2. Build the basic environment of the project

1. Create parent project and public api, see Chapter 3

2. Create Eureka service , see Chapter 3

3. Create order (order-service) service

Create an interface in the order service, methods in the interface: findById( ), deleteOrderById( )

Create a controller in the order service, methods in the controller: buy(), deleteOrderById()

3.1 Modify the pom file as follows

Modify the pom file and add the Hystix fuse component dependency, as follows:

<!--Introduce the Hystix fuse component to start--> 
<dependency> 
    <groupId>org.springframework.cloud</groupId> 
    <artifactId>spring-cloud-starter-netflix-hystrix</artifactId> 
</dependency> 
<!--Introduce the Hystix fuse component to end-->

Complete pom file:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <parent>
        <artifactId>springcloudbase</artifactId>
        <groupId>com.hwadee.springcloud2022</groupId>
        <version>0.0.1-SNAPSHOT</version>
    </parent>
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.hwadee.springcloud</groupId>
    <artifactId>orderServer9000</artifactId>

    <dependencies>
        <!--引入Hystix熔断器组件 开始-->
        <dependency>
           <groupId>org.springframework.cloud</groupId>
           <artifactId>spring-cloud-starter-netflix-hystrix</artifactId>
        </dependency>
        <!--引入Hystix熔断器组件 结束-->
        <!--告诉微服务注册中心我是微服务的client端 Feign 需要和eureak整合-->
        <!-- Feign相关 开始 -->
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-openfeign</artifactId>
        </dependency>
        <!-- Feign相关 结束 -->
        <!--web依赖-->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <!-- 管理公共api -->
        <dependency>
            <groupId>com.hwadee.springboot2022</groupId>
            <artifactId>springcloud-api</artifactId>
            <version>0.0.1-SNAPSHOT</version>
        </dependency>

        <!--Eureka Client-->
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
        </dependency>
    </dependencies>

</project>

3.2 Create interface IOrderFeignService in order service

Create the interface IOrderFeignService that calls the remote service in the service directory of the order service, and use the annotation @FeignClient ("PRODUCT-SERVICE") on the interface to indicate that when the controller calls the interface method, it uses the load balancing mechanism to search for the remote service with the service name "PRODUCT-SERVICE" . The method in the interface is similar to the method in the controller. code show as below:

import com.hwadee.springcloud.entity.Product;
import org.springframework.cloud.openfeign.FeignClient;
import org.springframework.stereotype.Component;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;

@Component // 让 spring 可以识别，不加也行，但是在注入的时候 IDEA 会报错，不会影响运行，但有条红线让自己不舒服
@FeignClient("PRODUCT-SERVICE")
public interface IOrderFeignService {
    @RequestMapping(value = "/product/buy/{id}")
    Product findOrderById(@PathVariable Long id);

    @RequestMapping(value = "/product/delete/{id}")
    Product deleteOrderById(@PathVariable Long id);
}

3.3 Create OrderController in Order Service

Create an OrderController in the order service, inject the IOrderFeignService interface into the OrderController, and call the remote service through the IOrderFeignService interface method, the code is as follows:

import com.hwadee.springcloud.entity.Product;
import com.hwadee.springcloud.service.IOrderFeignService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping("/order")
public class OrderController {

    @Autowired
    IOrderFeignService orderFeignService;

    @RequestMapping("/buy/{id}")
    public Product buy(@PathVariable Long id) {
        System.out.println("进入OrderController orderFeignService 准备调用远端接口");
        Product product = orderFeignService.findOrderById(id);
        return product;
    }

    @RequestMapping(value = "/delete/{id}")
    Product deleteOrderById(@PathVariable Long id){
        System.out.println("进入OrderController orderFeignService 准备调用远端接口");
        Product product = orderFeignService.deleteOrderById(id);
        return product;
    }

}

3.4 Configure the configuration file of the order service

Configure the configuration file application.yml in the order service (order-service), modify it as follows:

server:
  port: 9000
spring:
  application:
    name: order-service # 为当前订单服务命名为 order-service

# 配置eureka客户端信息
eureka:
  client:
    service-url:
      # 配置eureka客户端服务order-service的注册地址，与 eureka-server 中暴露地址要保持一致
      defaultZone: http://localhost:8000/eureka/
  instance:
    prefer-ip-address: true # 是否使用 IP 地址注册，默认 false
    # instance-id: order-service  # 实例 id，服务的唯一标识，会自动的找到order-service的ip和端口
    instance-id: ${spring.cloud.client.ip-address}:${server.port} # 如果想在控制页面看到服务地址与端口，可以将 instance-id 这样配置

3.5 Create the main startup class of the order service

Create the main startup class OrderServerApplication of the order service, and add the annotation @EnableFeignClients to start openFeign and the annotation @EnableCircuitBreaker to start the Hystrix service. code show as below:

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.netflix.eureka.EnableEurekaClient;
import org.springframework.cloud.openfeign.EnableFeignClients;

@SpringBootApplication
@EnableEurekaClient// 启动 eureka 客户端
@EnableFeignClients  // 启动 feign
public class OrderServerApplication {
    public static void main(String[] args) {
        SpringApplication.run(OrderServerApplication.class, args);
    }
}

4 Create a commodity (product-service) service cluster

4.1 Create a product (product-service) service

Create a project: productServer9001, introduce the web launcher and eureka client, and fuse and downgrade Histrix dependencies, modify the pom file as follows:

<dependency>

<groupId>org.springframework.cloud</groupId>

<artifactId>spring-cloud-starter-netflix-hystrix</artifactId>

</dependency>

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <parent>
    <artifactId>springcloudbase</artifactId>
    <groupId>com.hwadee.springcloud2022</groupId>
    <version>0.0.1-SNAPSHOT</version>
  </parent>
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.hwadee.springcloud</groupId>
  <artifactId>productServer9001</artifactId>
  <version>0.0.1-SNAPSHOT</version>


  <dependencies>
     <!--引入Hystix熔断器组件 开始-->
     <dependency>
         <groupId>org.springframework.cloud</groupId>
         <artifactId>spring-cloud-starter-netflix-hystrix</artifactId>
     </dependency>
     <!--引入Hystix熔断器组件 结束-->

     <!--web依赖-->
     <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
     </dependency>

    <!-- 管理公共api -->
    <dependency>
      <groupId>com.hwadee.springboot2022</groupId>
      <artifactId>springcloud-api</artifactId>
      <version>0.0.1-SNAPSHOT</version>
    </dependency>

    <!--Eureka Client-->
    <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
    </dependency>
  </dependencies>
</project>

4.2 Create the main startup class of the productServer service

Create the main startup class of the productServer service, and use the annotations @SpringBootApplication and @EnableEurekaClient// to start the eureka client

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.netflix.eureka.EnableEurekaClient;

@SpringBootApplication
@EnableEurekaClient// 启动 eureka 客户端
public class ProductServerApplication {
    public static void main(String[] args) {
        SpringApplication.run(ProductServerApplication.class, args);
    }
}

4.3. Configure productServer configuration file

Configure the productServer configuration file application.yml information:

server:
  port: 9001
spring:
  application:
    name: product-service # 为当前商品服务命名
eureka:
  client:
    service-url: # 配置服务注册地址，与 eureka-server 中暴露地址保持一致
      defaultZone: http://localhost:8000/eureka
  instance:
    prefer-ip-address: true  # 是否使用 IP 地址注册，默认 false
    # instance-id: product-service  # 实例 id，服务的唯一标识
    instance-id: ${spring.cloud.client.ip-address}:${server.port} # 如果想在控制页面看到服务地址与端口，可以将 instance-id 这样配置
    lease-renewal-interval-in-seconds: 5  # 发送心跳的间隔，单位秒，默认 30
    lease-expiration-duration-in-seconds: 10 # 续约到期时间，单位秒，默认90

4.4. Create ProductController

Create a ProductController as follows:

import com.hwadee.springcloud.entity.Product;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import java.math.BigDecimal;

@RestController
@RequestMapping("/product")
public class ProductController {
    //方便后面讲负载均衡，查看ip，此处获取配置中的端口号和ip
    @Value("${server.port}")
    private String port;
    @Value("${spring.cloud.client.ip-address}")
    private String ip;

    @RequestMapping("/buy/{id}")
    public Product findById(@PathVariable Long id) {
        Product product = new Product();
        product.setId(id);
        // 后面需要测试负载均衡，所以返回 ip 地址及端口号
        product.setName("当前访问商品服务地址：" + ip + ":" + port+"  "+"查询商品订单，订单号："+id);
        product.setPrice(new BigDecimal(10000.0));
        System.out.println(product);
        return product;
    }

    @RequestMapping(value = "/delete/{id}")
    Product deleteOrderById(@PathVariable Long id){
        Product product = new Product();
        product.setId(id);
        // 后面需要测试负载均衡，所以返回 ip 地址及端口号
        product.setName("当前访问商品服务地址：" + ip + ":" + port+"  "+"从购物车删除订单，订单号："+id);
        product.setPrice(new BigDecimal(10000.0));
        System.out.println(product);
        return product;
    }

}

4.5 Create commodity service provider cluster

Use the following method: Duplicate multiple projects without creating a project.

5. Test

First start the registry service: eurekaServer main startup class EurekaServerApplication
Second, start the order service: order-service main startup class OrderServerApplication
Start the commodity service productServer9001: the main startup class ProductServerApplication of product-servie
Start the commodity service productServer9002: the main startup class ProductServerApplication of product-servie
Start the commodity service productServer9003: the main startup class ProductServerApplication of product-servie

Access the order service multiple times: http://localhost:9000/order/buy/1 View the returned ip and port information

3. Problems before using Hystrix's service

1. Method exception and timeout in ProductController

Add exceptions and timeouts to the methods in the product service ProductController, and modify the methods in ProductController as follows:

package com.hwadee.springcloud.controller;
import com.hwadee.springcloud.entity.Product;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import java.math.BigDecimal;

@RestController
@RequestMapping("/product")
public class ProductController {
    //方便后面讲负载均衡，查看ip，此处获取配置中的端口号和ip
    @Value("${server.port}")
    private String port;
    @Value("${spring.cloud.client.ip-address}")
    private String ip;

    @RequestMapping("/buy/{id}")
    public Product findById(@PathVariable Long id) {
        Product product = new Product();
        product.setId(id);
        // 后面需要测试负载均衡，所以返回 ip 地址及端口号
        product.setName("当前访问商品服务地址：" + ip + ":" + port+"  "+"查询商品订单，订单号："+id);
        product.setPrice(new BigDecimal(10000.0));
        System.out.println(product);

        //测试超时熔断
        try {
            Thread.sleep(5000);
            //测试并发熔断
            //Thread.sleep(500);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }

        return product;
    }

    @RequestMapping(value = "/delete/{id}")
    public Product deleteOrderById(@PathVariable Long id){
        Product product = new Product();
        product.setId(id);
        // 后面需要测试负载均衡，所以返回 ip 地址及端口号
        product.setName("当前访问商品服务地址：" + ip + ":" + port+"  "+"从购物车删除订单，订单号："+id);
        product.setPrice(new BigDecimal(10000.0));
        System.out.println(product);

        //测试异常熔断
        System.out.println(10/0);

        return product;
    }

}

2. Access to view the effect

Visit http://localhost:9000/order/buy/1

Visit http://localhost:9000/order/delete/1

When the method in the product service ProductController is called in the OrderController in the order service, the method in the ProductController is abnormal and timed out. The abnormal result of accessing the browser and the internal exception in the order service are as follows:

3. Problem Analysis

4. Partial downgrade of commodity service Hystrix

See Chapter 7

5. Partial downgrade of order service Hystrix

See Chapter 8

6. Global downgrade

See Chapter 9

7. Global decoupling

See Chapter 10

Chapter 5: Detailed explanation of Feign load balancing

Chapter 7: Detailed Explanation of Hystrix Circuit Breaker - Partial Downgrade of Service Downgrade

Chapter 6 Detailed Explanation of Hystrix Circuit Breaker + Environment Construction

1. Background and introduction of Hystrix

1. Problems faced by distributed systems