Cloud Technology - Chaos Engineering

Table of contents

chaos engineering

Chaos Engineering is an experimental system reliability engineering method that actively introduces faults and exceptions to test the resilience and fault tolerance of the system. The core idea of chaos engineering is to verify the behavior of the system under various abnormal conditions by simulating fault scenarios , so as to discover potential problems in advance and improve the reliability and stability of the system .

Chaos engineering usually includes the following steps:

Define the normal behavior of the system: First, it is necessary to clarify the normal behavior and performance indicators of the system, such as response time, throughput, error rate, etc. (this generally requires the assistance of a monitoring system, such as prometheus).
Design experiments: Based on the system architecture and components, design fault injection experiments, such as simulating server downtime, network delay, disk failure, etc.
Monitoring and observation: During the experiment, the performance indicators and behavior of the system are continuously monitored, and the performance of the system in failure scenarios is observed.
Analysis and optimization: According to the experimental results, analyze the weaknesses and deficiencies of the system, optimize the architecture and implementation of the system, and improve the flexibility and fault tolerance of the system.

A simple Chaos Engineering example to test the behavior of a web application in the event of a database failure:

Define normal behavior: The normal behavior of a web application is to respond to user requests within 500ms with an error rate of less than 1%.
Design experiments: Simulate a database failure, for example by shutting down the database server or disconnecting from the network.
Monitoring and observation: During the experiment, monitor performance indicators such as response time and error rate of the web application.
Analysis and optimization: Based on the experimental results, evaluate the behavior of the web application when the database fails. If the performance indicators do not meet expectations, the architecture and implementation of the web application can be optimized, such as using caching, downgrading services, retry strategies, etc.

The use of chaos engineering techniques and tools in the industry mainly involves fault injection, monitoring and observation, automation and continuous integration.

fault injection

Fault injection is the core technology of chaos engineering, which is used to simulate various fault scenarios.

Techniques include:

Hardware fault injection: For example, shutting down the server, disconnecting the power supply, unplugging the network cable, etc.
Software Fault Injection: For example, simulate OS bugs, memory leaks, CPU overload, etc.
Network Fault Injection: For example, simulate network delays, packet loss, bandwidth limitations, etc.
Application fault injection: For example, simulate service downtime, interface errors, performance bottlenecks, etc.

etc.

The industry has tools such as:

Chaos Monkey: Netflix's open source chaos engineering tool is used to randomly shut down virtual machines or container instances in the production environment to test the resilience and failure recovery capabilities of the system.
Gremlin: A commercial chaos engineering platform that provides a range of fault injection scenarios, such as resource consumption, network failure, application failure, etc.
Pumba: An open source Docker container fault injection tool for simulating scenarios such as container failures, network failures, and performance issues.
Toxiproxy: Shopify's open source network fault injection proxy, used to simulate fault scenarios such as network delay, packet loss, and connection interruption.

The last step here can also include: simulation and modeling

Chaos engineering can use simulation and modeling techniques to predict how a system will behave under failure scenarios. For example, state machine, Petri net, queuing theory and other methods can be used to establish a mathematical model of the system, and then analyze the performance index and stability of the system under failure scenarios.

monitoring and observation

three aspects:

Log collection and analysis: collect system application logs, error logs, audit logs, etc., and analyze system behavior in failure scenarios.
Performance indicator monitoring: monitor system performance indicators, such as response time, throughput, error rate, etc.
Distributed tracing: In a distributed system, collect and analyze the call link and performance data of requests between various services.

Common tool:

(1) Prometheus: An open source monitoring and alerting system for collecting and storing system performance indicators and events. Prometheus is widely used to monitor chaos engineering experiments
(2) Grafana: an open source data visualization and analysis platform that can be integrated with monitoring systems such as Prometheus to display performance indicators and trend graphs of chaos engineering experiments.
(3) Jaeger: An open source distributed tracking system for collecting and analyzing request call links and performance data in distributed systems. Jaeger can help analyze the impact of chaos engineering experiments on distributed systems.
(4) Elastic Stack: An open source log collection, search and analysis platform, including components such as Elasticsearch, Logstash and Kibana. Elastic Stack can be used to collect and analyze log data of chaos engineering experiments.

Automation and Continuous Integration

Automated execution and continuous integration facilitate continuous verification of system resilience and fault tolerance during development and deployment.

(1) Automated testing framework: Use an automated testing framework (such as JUnit, pytest, etc.) to write chaos engineering experiments for automatic execution during the continuous integration process. In addition, you can also use Jenkins—an open source CI/CD server, etc.

(2) Continuous integration and continuous deployment (CI/CD): Integrate chaos engineering experiments into the CI/CD process to ensure that the system is verified for resilience and fault tolerance every time it is changed and deployed. Some large enterprises often have internal gitlab or self-built Git warehouse for CI/CD.

Below is a simple Jenkins Pipeline configuration example for fault injection using the Litmus Chaos tool in a Kubernetes cluster. The scenario is to deploy a simple Nginx application and use the Litmus Chaos tool to perform Pod deletion experiments:

pipeline {
    agent any

    stages {
        stage('Deploy Nginx') {
            steps {
                sh 'kubectl apply -f https://raw.githubusercontent.com/litmuschaos/chaos-workflows/master/App/nginx/app/nginx.yml'
            }
        }

        stage('Install Litmus Chaos') {
            steps {
                sh 'kubectl apply -f https://litmuschaos.github.io/litmus/litmus-operator-v1.13.8.yaml'
            }
        }

        stage('Run Pod Delete Experiment') {
            steps {
                git 'https://github.com/litmuschaos/chaos-charts.git'
                sh 'kubectl apply -f chaos-charts/charts/generic/pod-delete/engine.yaml'
                sh 'kubectl apply -f chaos-charts/charts/generic/pod-delete/rbac.yaml'
                sh 'kubectl apply -f chaos-charts/charts/generic/pod-delete/experiment.yaml'
            }
        }

        stage('Clean Up') {
            steps {
                sh 'kubectl delete -f chaos-charts/charts/generic/pod-delete/experiment.yaml'
                sh 'kubectl delete -f chaos-charts/charts/generic/pod-delete/rbac.yaml'
                sh 'kubectl delete -f chaos-charts/charts/generic/pod-delete/engine.yaml'
                sh 'kubectl delete -f https://litmuschaos.github.io/litmus/litmus-operator-v1.13.8.yaml'
                sh 'kubectl delete -f https://raw.githubusercontent.com/litmuschaos/chaos-workflows/master/App/nginx/app/nginx.yml'
            }
        }
    }
}