Fuse

 We know that if a concurrent software system of the number of requests exceeds the optimal number of threads in the system, then it will lead to fierce competition for resources, even with the lack of resources depletion, the entire system will be faced with disaster. Therefore, many software systems to ensure that even in the event of concurrent users> when the optimal number of threads, that does not lead to the collapse of the entire 40,000 Web sites will use some techniques to avoid systemic disaster. These techniques more typical is the current limit, demotion and fuse

1. Why do we need fuse

 Now many websites are behind a huge " distributed interaction between the majority" system, multiple systems are based on " RPC " way, but because it is a long-distance call, so the availability of caller services actually It is not controllable. The more extensive system, the call chain downstream will be longer, and if in a very long call chain, one service for some reason, the response time is very long, or completely unresponsive, it can be able to entire distributed systems collapse.

 

 

 

 

Above the call chain, if one service which due to their own causes slow to respond, then it may lead to affect upstream service is also very slow, so the cycle will lead to the collapse of the entire system across the board, this is the service avalanche .

In fact, in a distributed system, in order to ensure overall service availability and consistency, many systems will be introduced retry mechanism, in some cases, can actually be retried solve the problem, such as network problems, etc., can be retried by solve.

However, in some cases, not solve the problem and retry, but will return to aggravate the problem, because the downstream system such as a request amount is too large, resulting in the CPU has been played, said database connection pool is filled, this time upstream The system will continue to adjust unreasonable retry, retry this request, for the downstream system is undoubtedly worse, cause secondary damage to the downstream system.

And distributed systems, most services are also due to avalanches caused by the continuous retry, retry this may be the framework level of automatic retries, there may be code-level retry logic, there may be users take the initiative and try again.

Some of retries can not be avoided, but if for preventing avalanches, do not retry mechanism design, but also a unworthy.

2, fuse mode

Fuse mode (Circuit Breaker Pattern), is a modern software development design patterns. For detecting errors and avoid constantly trigger the same error (such as maintenance service is not available, the temporary system problem or unknown system error). Suppose there are two applications will communicate with databases hundreds of times per second, when suddenly a database error occurs, the programmer does not want the error Shihai continue to access the database. So we will try to deal directly with this error, and enter the normal end of the program. In short, the fuse will detect the error and "prevention" Applications continue to retry a call almost no response service (unless the security services have to retry the connection).

 

 

 

 

Fuse mode is an important means of micro Chronicles service system to prevent avalanches.

A relatively sound fuses, generally include three states:

• shut down

  • In the fuse is presented by default closed, the fuse itself with the counting function, whenever the one, and the counter will be "cumulative" operation error occurs, an error occurs to a certain number of times the circuit breaker will be "oN", this time will enable a timer inside, once the time is up it will switch into a semi-open state.

• Open

  • In the open state of any request will be "direct" message is rejected and thrown.

• Semi-open

  • In this state, the circuit breaker will allow part of the request, if these requests are successful, then it means that the error does not exist, it will be switched back off and reset the count. If there is an error in the request "any one" occurs, it will return to the "on" state, and re-timing, a break time administering the system.

The figure is a three-state conversions fuse.

If you call the process of micro-services system, the introduction of a fuse, then the whole system will naturally be able to:

1. Rapid-Fail : When calling remote services because too many failures, fuse open, upstream to downstream service call to the service will fail quickly, to avoid the upstream service collapse.

2. Seamless recovery : because the fuse can regularly check whether the downstream system recovery, you can recover once again returned to the closed state, all requests will be normal request to downstream services. So that the system does not need to consider intervention.

 

 

 

3, fusing tool

Fuse To achieve fast and seamless recovery failure on the need for the number of service calls statistics, service call cut and other operations, if you want to implement a fuse yourself actually possible.

However, there are some market framework has helped us to do these things. As Hystrix and Sentinel, resilience4j like.

3.1, Hystrix

Hystrix (https://github.com/Netflix/Hystrix) open source Netflix is ​​a fault-tolerant system that can help users yards out with a strong fault tolerance and robustness of the program. Provide downgrade, fuse and other functions.

However, at the end of 2018, Hystrix in their Github page announced, no longer open new features, the developers recommend the use of other open source projects still active.

Hystrix Although no longer develop new features, but the impact on users should not be too large, first, because developers can continue to use the latest version of 1.5.18 Hystrix

3.2、resilience4j

Hystrix more after stopping, Netflix official recommended resilience4j (https://github.com/resilience4j/resilience4j), it is a lightweight, easy to use, highly available framework can be assembled to support fusing, high-frequency control, isolation, limited flow, limit, retry, and other high availability mechanisms.

Compared with Hystrix, it has the following major differences:

• Hystrix HystrixCommand call must be encapsulated, the decorator and resilience4j manner to provide decoration of nested function interface, lambda expression like, so you can combine a plurality of high-availability mechanisms simple manner;

• Hystrix statistical frequency of use of the sliding window, rather resilience4j by way of a ring buffer;

• About fuse state when the half open state transition executed only once using MAMMALIA, determines whether the state transition, and execution count resilience4j is used with the threshold value may be configured to determine whether the state transition, the fuse mechanism in this way to improve the stability;

• About Quarantine mechanism, Hystrix provide isolation based on the thread pool and semaphores, and resilience4j only provides isolation based semaphore.

3.3、Sentinel

Sentinel (https://github.com/alibaba/Sentinel) Ali middleware team is open source, lightweight high availability flow control assembly for distributed services architecture, mainly in the flow as the starting point, the flow control, fuse downgrade the system load in the dimension of protection to help users protect the stability of the service.

Hystrix's focus is to isolate and fuse-based fault tolerance, time out or blown call will fail quickly, and can provide a fallback mechanism.

The Sentinel's focus is on:

• A variety of flow control

• Fuse downgrade

• System load protection

• Real-time monitoring and console

The figure is in contrast Sentinel's home page on GitHub and Hystrix the Sentinel:

 

 

 

 

 

4, the fuse mechanism :

Refers to the trading hours of the stock market, when price volatility reaches a certain defined target (fusing point), the suspension of trading of its mechanism for some time. This mechanism as fuses in the fuse when too much current is quite similar, hence the name.

Fuse mechanism introduced purpose is to guard against systemic risk, time to market, more calm and avoid panic spread lead to market volatility, so as to prevent the occurrence of the phenomenon of large-scale decline in the stock price. However, because of fusing mechanism also cut off the flow of funds, it will also result in increased market sentiment, and the market risk continues to expand at the end of the fuse.

American fuse mechanism :

U.S. fuse mechanism of the benchmark S & P 500 index, single drop threshold value of 7%, 13%, 20%. When the index fell by 7% compared with the previous day closing level, 13% of the nation's stock market trading will be suspended for 15 minutes, when the index fell 20% compared with the previous day closing level, the same day the transaction is stopped. 2010 Wall Street stocks began to implement fuse mechanism.

Fuse mechanism was first proposed in 1987 by the New York Stock Exchange in the United States, in order to avoid similar "Black Monday" stock market crash occurred. At this point the only blown fuse mechanism for the market index. 1997 October 27, the Dow Jones industrial average plunged 7.18 percent, to close at 7161.15 points, this is the first time fuse mechanism is triggered after the introduction of 1988.

US stocks triggered a second fuse mechanism is in EST March 9, 2020, by the 2019 coronavirus disease outbreak and the collapse of oil prices impact, at 9:34 on March 9, the S & P 500 index fell 7 percent after the opening bell trigger The first layer fuse mechanism to suspend trading for 15 minutes. Three days later, on March 12, Standard & Poor's 500 index fell more than a short time after the opening of the first layer 7% trigger mechanism fuse again.

Chinese mainland fuse mechanism :

People's Republic of China since the beginning of 2016 at the Shanghai Stock Exchange, Shenzhen Stock Exchange and China Financial Futures Exchange at the same time a trial fuse mechanism. Which is blown CSI 300 benchmark set 5%, 7% two steps fusing threshold index, ups and downs are blown.