content
- What is Hystrix
- what is Hystrix used for
- What problem does Hystrix solve
- Hystrix Design Principles
- How Hystrix achieves its goals
What is Hystrix
In a distributed environment, there are inevitably many scenarios where calls between services fail. Hystrix is a library that helps you control the interaction between these distributed services by adding latency tolerance and fault tolerance logic. Hystrix implements cascading failures between applications by isolating access points between services (upstream services are unavailable due to downstream service failures), and provides alternatives to improve overall application availability in the event of a failure.
Example: During a snap-up activity, due to the overtime or collapse of the inventory service, you can prevent the snap-up app from being unavailable by prompting that the snap-up failed, please try again later, etc.
what is Hystrix used for
Hystrix can do a few things:
- Provides protection for control delays and call failures between dependencies through third-party client libraries.
- Prevent cascading failures in complex distributed systems
- Fast failover
- Fallback and graceful degradation where possible
- Real-time monitoring, alarming, operation and maintenance control as much as possible
What problem does Hystrix solve
A complex distributed application has many dependent calls, and at some point there is bound to be a failure between them, which is the risk of causing the overall application failure.
For example, for an application that depends on 30 services, each with 99.99% uptime, here's what you can expect.
99.99 to the 30th power = 99.7% (meaning 0.3% failure)
1 billion * 0.3% = 3 million (1 billion requests will have 3 million failed requests)
Even with good uptime every month, there will be 2 hours of downtime. Usually the situation is made worse by this.
When all requests are normal, the request flow will look like the following figure:
When many backend systems are unstable, the entire request will be blocked, as shown in the following figure:
Under a large number of requests, a suboptimal dependency of a backend will become a potential risk of blocking the entire application request
Any request sent by the network or the client in the application may become a potential risk of network request failure. What is worse than the request failure is that the delay between application calls and dependencies increases, even more queues, thread pools, Cascading failures caused by other system resources
When accessing the network through a third-party client, it is a black-box execution with unclear implementation details and may change at any time. For each client, the network and resource configuration are different, so it is difficult to monitor and change.
Worse, between dependent calls, perform expensive and error-prone network calls without being explicitly called by the application
Network connection failure or degradation, service-to-service call failure or delay, new library or service deployment changes or performance characteristics
All of these failures and delays need to be isolated or managed so that the failure of a single dependency does not destroy the entire application and system.
Hystrix Design Principles
- Prevent a single application from exhausting all available threads on a server (such as tomcat)
- Drop heavy loads, fail fast instead of queuing
- Protection against failure in any situation
- Use isolation techniques such as baffles, swim lanes, and circuit breaker patterns to limit the impact of any one dependency
- Optimize rapid fault detection with near real-time metrics, monitoring, and alerting
- Hystrix relies heavily on low-latency provisioning and dynamic provisioning to optimize recovery time, allowing real-time operational adjustments with fast return results
- Protects against failure in the entire dependent client implementation, not just in network traffic
How Hystrix achieves its goals
- Encapsulate all calls to external systems (or "dependencies") in a HystrixCommand or HystrixObservableCommand object, which is usually executed in a separate thread ( this is an example of the command pattern )
- Hystrix has a default value. If the call duration exceeds this default value, it is regarded as a timeout call. Most dependencies can customize this value through configuration, so they are slightly higher than 99.5% of the performance of the dependencies.
- A small thread pool (or semaphore) is maintained for each dependency; if it is full, requests specified for that dependency will be rejected immediately, rather than queued.
- Measure success, failure (exceptions thrown by clients), timeouts and thread rejections
- Trigger a circuit breaker to stop all requests to a service for a period of time, either manually or automatically if the service's error percentage exceeds a threshold
- Execute fallback logic when a request fails, is rejected, times out, or short-circuits
- Near real-time monitoring of metrics and configuration changes
The architecture shown in the diagram changes when you use Hystrix to wrap each underlying dependency, as shown in the following diagram. Each dependency is isolated from each other, when a delay occurs, it may be bound in resources, and overridden in fallback logic, when any kind of failure occurs in a dependency, it decides what to make what response
