Micro-service product line, each service to concentrate on its own business logic, and provide the external interface seems very clear, in fact, there are many things to consider, such as: automatic expansion of services, and current limiting fuse with the expansion of business, the number of services will increase, and will be more complex logic, a logic of a service need to rely on a number of other services to complete. Once a dependency can not provide the service is likely to have an avalanche effect, leading to the entire service inaccessible.
When rpc or http calls between micro-services, we will generally set to call a timeout, failure retry mechanisms to ensure the successful implementation services, looks beautiful, if you do not consider limiting fuse and services, is the source of the avalanche.
Suppose we have two to access a greater amount of service A and B, respectively, dependent on both services C and D, C and D are dependent services Service E
A and B continue to call C, D handle customer requests and return the data needed. When E service can not provide services, timeouts and retry mechanism C and D will be executed
Due to constantly generate new calls, will lead to a large number of calls to the backlog C and D E services, a large number of calls waiting and retry the call, it will slowly run out of resources C and D such as memory or CPU, and then also down out.
A and B, C and D will be repeated service operations, resource depletion, and then fall down, eventually the whole services are inaccessible.
Common cause avalanche situation are the following:
- Program bug causes the service is not available or is slow
- Cache breakdown, leading to full access to a service call, leading to fall down
- The sudden surge in traffic.
- Hardware problem, this feeling can only say that points back of ⊙︿⊙.
Although the avalanche effect of 10 million, to ensure that services do not hang up and running smoothly is our responsibility, the corresponding avalanche effect there are still many protection schemes.
The lateral expansion of services
Now we can use a lot of tools to ensure that services will not hang up, then the flow is relatively large, it may be laterally expanded services to ensure smooth operations. For example, we use most often k8s, can ensure the operation status of the service, the service also allows automatic horizontal expansion. For the surge in user traffic to this process is still very good, but there is also the end of the lateral expansion, if the response time in a certain environment E services is too long, there are still likely to lead to an avalanche effect.
Limiting
Limit the client's call to reach the current limit of practice is very common, for example, we limit the maximum processing 200 requests per second, more than two orders directly deny the request. Common algorithms such as token bucket algorithm
At a certain speed to put in the bucket token, when a client requests a service, must first get a token from the bucket, in order to be processed, if the bucket runs out of tokens, access is denied.
Fuse
Control access to the client dependent, dependence is unavailable, the call if the call is no longer directly returns an error, or downgrade. A library source code analysis of open source libraries such as hystrix-go, but also I am going to write. To achieve a good fuse and downgrade function. His main idea is to set some threshold, for example, the maximum number of concurrent error rate percentage, try to fuse recovery time. Able to convert these thresholds fuse status:
- Off, allowing calls rely
- Open state, the caller does not depend directly returns an error, or call fallback
- Half-open state, according to fuse try to open the recovery time, allows the caller dependent, if the call is successful then failed to close it continues to open
This article is over! Like a friend to help forward the article and look! thank! ! !
Java learning, interviews; documentation, free access to video resources
Reproduced in: https: //juejin.im/post/5cfe59f46fb9a07ebb0528e5