Hystrix fuse mechanism principle

Related configuration

circuitBreaker.enabled  是否开启熔断
circuitBreaker.requestVolumeThreshold  熔断最低触发请求数阈值
circuitBreaker.sleepWindowInMilliseconds  产生熔断后恢复窗口
circuitBreaker.errorThresholdPercentage  错误率阈值
circuitBreaker.forceOpen  强制打开熔断
circuitBreaker.forceClosed  强制关闭熔断

State diagram

Implementation process

Called before the command is executed circuitBreaker.attemptExecution(), it will return true under normal circumstances, but if a fuse occurs, you need to restore it through sleepWindows

public boolean attemptExecution() {
    if (properties.circuitBreakerForceOpen().get()) {
        return false;
    }
    if (properties.circuitBreakerForceClosed().get()) {
        return true;
    }
    if (circuitOpened.get() == -1) {
        return true;
    } else {
        if (isAfterSleepWindow()) {
            if (status.compareAndSet(Status.OPEN, Status.HALF_OPEN)) {
                //only the first request after sleep window should execute
                return true;
            } else {
                return false;
            }
        } else {
            return false;
        }
    }
}

Fusing process occurs

In the new version 1.5.12, there will be a background thread that subscribes to the metrics stream for real-time calculation:

1. If the RequestVolume is not reached, it will return directly without calculating whether it needs to be blown

2. If the current error rate is greater than the set threshold, the fuse is triggered, the state is switched from CLOSED to OPEN, and the current fuse time is set (for subsequent SleepWindows judgment use)

   if (hc.getTotalRequests() < properties.circuitBreakerRequestVolumeThreshold().get()) {
   // we are not past the minimum volume threshold for the stat window,
   // so no change to circuit status.
   // if it was CLOSED, it stays CLOSED
   // if it was half-open, we need to wait for a successful command execution
   // if it was open, we need to wait for sleep window to elapse
   } else {
   if (hc.getErrorPercentage() < properties.circuitBreakerErrorThresholdPercentage().get()) {
       //we are not past the minimum error threshold for the stat window,
       // so no change to circuit status.
       // if it was CLOSED, it stays CLOSED
       // if it was half-open, we need to wait for a successful command execution
       // if it was open, we need to wait for sleep window to elapse
   } else {
       // our failure rate is too high, we need to set the state to OPEN
       if (status.compareAndSet(Status.CLOSED, Status.OPEN)) {
           circuitOpened.set(System.currentTimeMillis());
       }
   }
   }

Fuse recovery process

 When a fuse occurs and the time specified by SleepWindows is reached, the state will be converted from OPEN to HALF_OPEN. At this time, only one request will pass. If the request is successfully executed, the state will be converted to CLOSED, otherwise it will return to the OPEN state and fuse The time is set to the current time, and the time to wait for SleepWindows again

// 执行成功后调用
public void markSuccess() {
    if (status.compareAndSet(Status.HALF_OPEN, Status.CLOSED)) {
        //This thread wins the race to close the circuit - it resets the stream to start it over from 0
        metrics.resetStream();
        Subscription previousSubscription = activeSubscription.get();
        if (previousSubscription != null) {
            previousSubscription.unsubscribe();
        }
        Subscription newSubscription = subscribeToStream();
        activeSubscription.set(newSubscription);
        circuitOpened.set(-1L);
    }
}

// 执行失败后调用
public void markNonSuccess() {
    if (status.compareAndSet(Status.HALF_OPEN, Status.OPEN)) {
        //This thread wins the race to re-open the circuit - it resets the start time for the sleep window
        circuitOpened.set(System.currentTimeMillis());
    }
}

refer to

1. https://github.com/Netflix/Hystrix/wiki/Configuration
2. https://github.com/Netflix/Hystrix/wiki/How-it-Works#CircuitBreaker
3. HystrixCircuitBreaker source code