Before delving into circuit breakers, we need to take a look at several important default configurations of Hystrix, which are listed HystrixCommandPropertiesin
//时间窗（ms）
static final Integer default_metricsRollingStatisticalWindow = 10000;
//最少请求次数
private static final Integer default_circuitBreakerRequestVolumeThreshold = 20;
//熔断器打开后开始尝试半开的时间间隔
private static final Integer default_circuitBreakerSleepWindowInMilliseconds = 5000;
//错误比例
private static final Integer default_circuitBreakerErrorThresholdPercentage = 50;
These attributes together form the core logic of the circuit breaker, namely:
- During every 10-second window, when the number of requests exceeds 20 and the error rate exceeds 50%, the circuit breaker will be triggered to open.
- When the fuse is 5 seconds later, it will try to release some of the traffic for testing
Fuse initialization
The initialization of the fuse is in HystrixCircuitBreaker.Factorythe getInstancemethod
private static ConcurrentHashMap<String, HystrixCircuitBreaker> circuitBreakersByCommand = new ConcurrentHashMap<String, HystrixCircuitBreaker>();
public static HystrixCircuitBreaker getInstance(HystrixCommandKey key, HystrixCommandGroupKey group, HystrixCommandProperties properties, HystrixCommandMetrics metrics) {
// this should find it for all but the first time
HystrixCircuitBreaker previouslyCached = circuitBreakersByCommand.get(key.name());
if (previouslyCached != null) {
return previouslyCached;
}
// if we get here this is the first time so we need to initialize
// Create and add to the map ... use putIfAbsent to atomically handle the possible race-condition of
// 2 threads hitting this point at the same time and let ConcurrentHashMap provide us our thread-safety
// If 2 threads hit here only one will get added and the other will get a non-null response instead.
HystrixCircuitBreaker cbForCommand = circuitBreakersByCommand.putIfAbsent(key.name(), new HystrixCircuitBreakerImpl(key, group, properties, metrics));
if (cbForCommand == null) {
// this means the putIfAbsent step just created a new one so let's retrieve and return it
return circuitBreakersByCommand.get(key.name());
} else {
// this means a race occurred and while attempting to 'put' another one got there before
// and we instead retrieved it and will now return it
return cbForCommand;
}
}
As can be seen from the above code, each fuse is HystrixCircuitBreakerImplimplemented by, and all fuses are maintained in circuitBreakersByCommandthisConcurrentHashMap
Fuse implementation
Construction method
class HystrixCircuitBreakerImpl implements HystrixCircuitBreaker {
private final HystrixCommandProperties properties;
private final HystrixCommandMetrics metrics;
enum Status {
CLOSED, OPEN, HALF_OPEN
}
private final AtomicReference<Status> status = new AtomicReference<Status>(Status.CLOSED);
private final AtomicLong circuitOpened = new AtomicLong(-1);
private final AtomicReference<Subscription> activeSubscription = new AtomicReference<Subscription>(null);
protected HystrixCircuitBreakerImpl(HystrixCommandKey key, HystrixCommandGroupKey commandGroup, final HystrixCommandProperties properties, HystrixCommandMetrics metrics) {
this.properties = properties;
this.metrics = metrics;
//On a timer, this will set the circuit between OPEN/CLOSED as command executions occur
Subscription s = subscribeToStream();
activeSubscription.set(s);
}
}
Let’s first introduce some basic properties:
HystrixCommandProperties: Current fuse configurationHystrixCommandMetrics: request statistics componentStatus: fuse state enumeration, including three types, closed, open and half-openstatus: Current fuse statuscircuitOpened: Current opening time of the fuseactiveSubscription: Handler function for subscription request statistics
Request statistical processing
private Subscription subscribeToStream() {
/*
* This stream will recalculate the OPEN/CLOSED status on every onNext from the health stream
*/
return metrics.getHealthCountsStream()
.observe()
.subscribe(new Subscriber<HealthCounts>() {
@Override
public void onCompleted() {
}
@Override
public void onError(Throwable e) {
}
@Override
public void onNext(HealthCounts hc) {
// check if we are past the statisticalWindowVolumeThreshold
if (hc.getTotalRequests() < properties.circuitBreakerRequestVolumeThreshold().get()) {
// we are not past the minimum volume threshold for the stat window,
// so no change to circuit status.
// if it was CLOSED, it stays CLOSED
// if it was half-open, we need to wait for a successful command execution
// if it was open, we need to wait for sleep window to elapse
} else {
if (hc.getErrorPercentage() < properties.circuitBreakerErrorThresholdPercentage().get()) {
//we are not past the minimum error threshold for the stat window,
// so no change to circuit status.
// if it was CLOSED, it stays CLOSED
// if it was half-open, we need to wait for a successful command execution
// if it was open, we need to wait for sleep window to elapse
} else {
// our failure rate is too high, we need to set the state to OPEN
if (status.compareAndSet(Status.CLOSED, Status.OPEN)) {
circuitOpened.set(System.currentTimeMillis());
}
}
}
}
});
}
Look directly at onNextthe processing method in the method:
- Whether the number of requests in the time window meets the standard, according to the default configuration, whether the number of requests in 10 seconds exceeds 20 times, if it does not meet the standard, the fuse cannot be turned on
- In else, first determine whether the error ratio reaches the ratio, which is 50% by default
- Satisfy the opening conditions, use CAS to modify the status to open, and record the opening time
circuitOpenedas the current time
After the statistics of the current application are recorded, it can be judged whether the circuit breaker should be turned on according to the data at each request.
request filtering
I wonder if you still remember a method mentioned in the first article of the series applyHystrixSemantics. This method contains the logic to determine whether it should be blown. If the fuse is opened, it will directly enter the downgrade logic. The method of this judgment is as follows:
public boolean attemptExecution() {
if (properties.circuitBreakerForceOpen().get()) {
return false;
}
if (properties.circuitBreakerForceClosed().get()) {
return true;
}
if (circuitOpened.get() == -1) {
return true;
} else {
if (isAfterSleepWindow()) {
if (status.compareAndSet(Status.OPEN, Status.HALF_OPEN)) {
//only the first request after sleep window should execute
return true;
} else {
return false;
}
} else {
return false;
}
}
}
- The first if, if the configuration is forced to fuse, it will return false to turn on the fuse and enter the degraded logic
- Second, if the configuration is forced to close, it will return to normal without subsequent judgment.
- Third, if the opening time is empty, it must not have been opened.
- Fourth, judge whether the trial time is satisfied, the default is 5 seconds. Time is calculated as follows:
private boolean isAfterSleepWindow() {
final long circuitOpenTime = circuitOpened.get();
final long currentTime = System.currentTimeMillis();
final long sleepWindowTime = properties.circuitBreakerSleepWindowInMilliseconds().get();
return currentTime > circuitOpenTime + sleepWindowTime;
}
- When the attempt is satisfied, use the CAS method to modify the fuse to a half-open state
When the request is successful, the following method will be called to clear the statistics and change the circuit breaker state to off
public void markSuccess() {
if (status.compareAndSet(Status.HALF_OPEN, Status.CLOSED)) {
//This thread wins the race to close the circuit - it resets the stream to start it over from 0
metrics.resetStream();
Subscription previousSubscription = activeSubscription.get();
if (previousSubscription != null) {
previousSubscription.unsubscribe();
}
Subscription newSubscription = subscribeToStream();
activeSubscription.set(newSubscription);
circuitOpened.set(-1L);
}
}
If the request fails, turn on the circuit breaker again and update the opening time
public void markNonSuccess() {
if (status.compareAndSet(Status.HALF_OPEN, Status.OPEN)) {
//This thread wins the race to re-open the circuit - it resets the start time for the sleep window
circuitOpened.set(System.currentTimeMillis());
}
}
{{o.name}}
{{m.name}}