When calling between multiple microservices, suppose microservice A calls microservices B and C, and B and C call other microservices, that is, " fan-out ".
The call response time of a microservice on the fan-out link is too long or unavailable, and the call to A will occupy more and more system resources, which will cause system crash (cascade failure), that is, "avalanche effect "

1.2 What is Hystrix

1. Hystrix is an open source library for dealing with latency and fault tolerance of distributed systems

2. Hystrix can ensure that in the case of a dependency problem, it will not cause the overall service to fail, avoid cascading failures, and improve the elasticity of distributed systems

3. Hystrix acts as a "circuit breaker". When a service unit fails, through the fault monitoring of the circuit breaker, it returns an expected and processable alternative response (FallBack) to the caller instead of waiting for a long time Or throw an exception that the caller cannot handle. Thus avoiding avalanches.

1.3, Hystrix role

1. Service downgrade (FallBack): When the server is busy, the client is not allowed to wait and returns a friendly prompt immediately.

Occurrence: abnormal program operation, timeout, service fuse triggers service degradation, thread pool/semaphore will also cause service degradation

2. Service break (Break): After reaching the maximum service access, directly deny access, call the service downgrade method and return a friendly prompt

3. Close to real-time monitoring (FlowLimit): seckill with high concurrency, etc., it is strictly forbidden to swarm over and crowd, N per second, and proceed in an orderly manner

1.4, Hystrix workflow

1. Construct a HystrixCommand or HystrixObservableCommand object to encapsulate the request

2. To execute the command, Hystrix provides 4 methods to execute the command;

3. Determine whether to use the cache to respond to the request. If the cache is enabled and the cache is available, directly use the cache to respond to the request.

4. Determine whether the fuse is open, if it is open, skip to step 8;

5. Determine whether the thread pool/queue/semaphore is full, and skip to step 8 if it is full;

6. Execute HystrixObservableCommand.construct() or HystrixCommand.run(), if execution fails or times out, skip to step 8; otherwise, skip to step 9;

7. Statistical fuse monitoring indicators;

8. Follow the fallback backup logic;

9. Return the request response.

2. Hystrix case

2.1. Build normal microservices (service providers)

controller: provide two items, url mapping rules

@RestController
@Slf4j
public class PaymentController {
    @Resource
    private PaymentService paymentService;

    @Value("${server.port}")
    private String serverPort;

    @GetMapping("/payment/hystrix/ok/{id}")
    public String paymentInfo_OK(@PathVariable("id") Integer id){
        String result = paymentService.paymentInfo_OK(id);
        log.info("result:" + result);
        return result;
    }

    @GetMapping("/payment/hystrix/timeout/{id}")
    public String paymentInfo_TimeOut(@PathVariable("id") Integer id){
        String result = paymentService.paymentInfo_TimeOut(id);
        log.info("result:" + result);
        return result;
    }
}

2.2. Stress test

2.2.1, Jmeter pressure test

2.2.2. Client stress test

service: call the microservice interface provided in 2.1

@Component
@FeignClient(value = "Provider-hystrix-payment")
public interface OrderHystrixService {
    @GetMapping("/payment/hystrix/ok/{id}")
    public String paymentInfo_OK(@PathVariable("id") Integer id);

    @GetMapping("/payment/hystrix/timeout/{id}")
    public String paymentInfo_TimeOut(@PathVariable("id") Integer id);
}

3. Solutions

3.1. Service downgrade

Set the peak value of its own call timeout time, and it can run normally within the peak value
Exceeding the need to have a bottom-up method to deal with, do service downgrade fallback

3.1.1. Server-side microservice downgrade

1. POM file: add hystrix development scenario

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-netflix-hystrix</artifactId>
</dependency>

2. Business class: @HystrixCommand declares the bottom-up logic

Cover business logic, for timeout exceptions, calculation exceptions (int i= 10/0) can be covered

@Service
public class PaymentService {
    //注解中声明paymentInfo_TimeOutHandler方法为兜底业务逻辑，
    //同时声明该业务逻辑的正常执行时长为3秒钟以内
    @HystrixCommand(fallbackMethod = "paymentInfo_TimeOutHandler",
                    commandProperties = {@HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "3000")})
    public String paymentInfo_TimeOut(Integer id) {
        int timeNumber = 10;
        try{
            TimeUnit.SECONDS.sleep(timeNumber);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        return "线程池：" + Thread.currentThread().getName() + "paymentInfo_TimeOut,id:" + id + "\t";
    }

    //兜底业务逻辑
    public String paymentInfo_TimeOutHandler(Integer id) {
        return "线程池：" + Thread.currentThread().getName() + "paymentInfo_TimeOutHandler,id:" + id + "\t";
    }
}

3. Main startup class: add @EnableCircuitBreaker to activate

@SpringBootApplication
@EnableEurekaClient
@EnableCircuitBreaker
public class PaymentHystrixMain8001 {
    public static void main(String[] args) {
        SpringApplication.run(PaymentHystrixMain8001.class, args);
    }
}

3.1.2. Client microservice downgrade

1. POM file: add hystrix development scenario

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-netflix-hystrix</artifactId>
</dependency>

2. yaml configuration file: enable Hystrix service downgrade

#开启hystrix
feign:
  hystrix:
    enabled: true

3. Main startup class: Add @EnableHystrix to activate Hystrix

@SpringBootApplication
@EnableFeignClients
@EnableHystrix
public class OrderHystrixMain80 {
    public static void main(String[] args) {
        SpringApplication.run(OrderHystrixMain80.class, args);
    }
}

4. Business class: declare the bottom-line logic

@HystrixCommand(fallbackMethod = "paymentTimeOutFallbackMethod", commandProperties = {@HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "1500")})

@RestController
@Slf4j
public class OrderHystrixController {

    @GetMapping("/consumer/payment/hystrix/timeout/{id}")
    @HystrixCommand(fallbackMethod = "paymentTimeOutFallbackMethod", commandProperties = {
            @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "1500")
    })
    public String paymentInfo_TimeOut(@PathVariable("id") Integer id) {
        return orderHystrixService.paymentInfo_TimeOut(id);
    }

    public String paymentTimeOutFallbackMethod(@PathVariable("id") Integer id) {
        return "我是消费者80,对方支付系统繁忙请10秒种后再试或者自己运行出错请检查自己,o(╥﹏╥)o";
    }
}

5. Test results

3.1.3, code optimization

1) Each method has a bottom line that will cause code expansion ( business timeout ) - client microservice code optimization (controller)

In addition to special business logic, for ordinary business logic, you can design a general bottom-up method and configure it globally to prevent code expansion

1. @DefaultProperties (defaultFallback = " payment_Global_FallbackMethod ") declares the fallback method

2. @HystrixCommand is used to mark the bottom-up method, using this annotation

3. Write the global fallback logic payment_Global_FallbackMethod

@RestController
@Slf4j
@DefaultProperties(defaultFallback = "payment_Global_FallbackMethod")
public class OrderHystrixController {

    @Resource
    private OrderHystrixService orderHystrixService;

    @GetMapping("/consumer/payment/hystrix/timeout/{id}")
    @HystrixCommand
    public String paymentInfo_TimeOut(@PathVariable("id") Integer id) {
        return orderHystrixService.paymentInfo_TimeOut(id);
    }

    //下面是全局fallback方法
    public String payment_Global_FallbackMethod() {
        return "Global异常处理信息,请稍后重试.o(╥﹏╥)o";
    }
}

Test Results:

2) The bottom line is written together with the business, and the code is chaotic ( the server is down or shut down ) - client microservice decoupling (service)

1. Create a new service class and implement the service interface

The service class is the bottom-up logic for server downtime when remote calls to server-side microservices:

@Component
public class OrderFallbackService implements OrderHystrixService {
    @Override
    public String paymentInfo_OK(Integer id) {
        return "OrderFallbackService fallback, paymentInfo_OK";
    }

    @Override
    public String paymentInfo_TimeOut(Integer id) {
        return "OrderFallbackService fallback, paymentInfo_TimeOut";
    }
}

2. Service interface, @FeignClient annotation, add fallback attribute

The service interface is a remote call to the server-side interface. You need to add fallback = OrderFallbackService.class to the @FeignClient annotation to declare that the service class is the back-and-forth logic of the remote call interface

@Component
@FeignClient(value = "Provider-hystrix-payment", fallback = OrderFallbackService.class)
public interface OrderHystrixService {
    @GetMapping("/payment/hystrix/ok/{id}")
    public String paymentInfo_OK(@PathVariable("id") Integer id);

    @GetMapping("/payment/hystrix/timeout/{id}")
    public String paymentInfo_TimeOut(@PathVariable("id") Integer id);
}

3. Configure the yaml file and enable the Hystrix function

#开启hystrix
feign:
  hystrix:
    enabled: true

4. Test: Open the Eureka server, server microservice, and client microservice in turn, and call the client url interface before and after manually closing the server microservice to make remote calls to the server and view the results.

4. Service fuse

4.1. Fuse concept

The fuse mechanism is a microservice link protection mechanism to deal with the avalanche effect . When a microservice of the fanout link fails or the response time is too long, the service will be downgraded ,the call of the node microservice will be broken . , quickly returns an error response message. When it is detected that the microservice call response of the node is normal ,the call link is resumed .

4.2. Fuse operation (server-side microservice)

4.2.1, service layer

1. Add the @HystrixCommand annotation to the interrupted business logic:

1) The back-and-forth logic after declaring the circuit breaker

2) Set fuse parameters

2. Add paymentCircuitBreaker_fallback fuse logic

@Service
public class PaymentService {
    /** 服务熔断 */
    //表示在时间窗口期中10次请求有6次是错误的，就有调用断路器
    @HystrixCommand(fallbackMethod = "paymentCircuitBreaker_fallback", commandProperties = {
            //是否开启断路器
            @HystrixProperty(name = "circuitBreaker.enabled", value = "true"),
            //请求次数
            @HystrixProperty(name = "circuitBreaker.requestVolumeThreshold", value = "10"),
            //时间窗口期
            @HystrixProperty(name = "circuitBreaker.sleepWindowInMilliseconds", value = "10000"),
            //失败率达到多少后跳闸
            @HystrixProperty(name = "circuitBreaker.errorThresholdPercentage", value = "60"),
    })
    public String paymentCircuitBreaker(@PathVariable("id") Integer id) {
        if(id < 0){
            throw new RuntimeException("id,不能为负数");
        }

        String serialNumber = IdUtil.simpleUUID();

        return Thread.currentThread().getName() + "\t" + "调用成功，流水号：" + serialNumber;
    }

    public String paymentCircuitBreaker_fallback(@PathVariable("id") Integer id) {
        return "id不能为负数，请稍后再试，/(ToT)/~~ id:" + id;
    }
}

4.2.2, controller layer

@RestController
@Slf4j
public class PaymentController {
    @Resource
    private PaymentService paymentService;

    @Value("${server.port}")
    private String serverPort;

    @GetMapping("/payment/circuit/{id}")
    public String paymentCircuitBreaker(@PathVariable("id") Integer id) {
        String result = paymentService.paymentCircuitBreaker(id);
        log.info("result:" + result);
        return result;
    }
}

4.3. Fuse principle

4.3.1, fuse type

1) Fuse open: the request no longer calls the current service, and the internal setting is generally MTTR (mean failure processing time),

When the open time reaches the set time, it will enter the half-fuse state.

2) Fuse off: After the fusing is off, the service will not be fusing.

3) Fuse half open: Some requests call the current service according to the rules. If the request is successful and meets the rules,

Then it is considered that the current service is back to normal, and the circuit breaker is turned off.

4.3.2. Circuit breaker process

1) Circuit breaker open settings: snapshot time window, total request threshold, error percentage threshold

Snapshot time window: that is, the statistical time range, whether the circuit breaker is open or not needs statistical request and error data (default 10s)

Threshold of the total number of requests: within the snapshot time window, the threshold of the total number of requests must be met before being eligible for fuse (default 20)

Error Percentage Threshold: When the total number of requests exceeds the threshold, the circuit breaker is turned on when the proportion of response errors reaches the threshold (default 50%)

2) Circuit breaker operation process:

When the total number of requests and the failure rate meet a certain threshold, the circuit breaker is turned on
When enabled, all requests will not be forwarded
After a period of time (5 seconds by default), the circuit breaker turns half-open, allowing one of the requests to be forwarded.:
1. If successful, the circuit breaker closes
2. If it fails, continue to open

3) After the circuit breaker is opened:

When there is another request, the main logic will not be called, but the downgrade fallback will be called directly. Through the circuit breaker, the error is automatically found and the degraded logic is switched to the main logic to reduce the effect of response delay.

5. Service monitoring Hystrix Dashboard

5.1 Overview of Hystrix Dashboard

Hystrix provides quasi-real-time call monitoring (Hystrix Dashboard). Hystrix will continuously record the execution information of all requests initiated through Hystrix, and display them to users in the form of statistical reports and graphics, including how many requests are executed, how many successes, failures, etc. . Spring Cloud also provides the integration of Hystrix Dashboard, which converts monitoring content into a visual interface.

5.2, Hystrix Dashboard service construction

5.2.1, POM file:

Add hystrix-dashboard development scene

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-netflix-hystrix-dashboard</artifactId>
</dependency>

5.2.2. Configure yaml file (port number)

server:
  port: 9001

5.2.3, write the main startup class

Add the @EnableHystrixDashboard annotation to activate the Hystrix Dashboard component.

@SpringBootApplication
@EnableHystrixDashboard
public class HystrixDashBoardMain9001 {
    public static void main(String[] args) {
        SpringApplication.run(HystrixDashBoardMain9001.class, args);
    }
}

5.2.4. Visit http://localhost:9001/hystrix and get the following page

5.3, circuit breaker demonstration (9001 monitoring 8001)

5.3.1. Fill in the monitoring address http://localhost:8001/hystrix.stream

5.3.2. Call the microservice multiple times and open the Monitor stream on the monitoring interface

1) View monitoring results:

2) Description of monitoring indicators:

Hystrix principle and its use

Table of contents

1. Concept

1.1. Problems faced by distributed systems