OpenShift Istio-Tutorial 4 of (7) using the chaotic VirtualService fault injection test Chaos Testing

This series OpenShift Servic Mesh tutorial is based on Red Hat official public offering " Introducing Istio Service Mesh for Micoservices " publication, I will verify all operations carried out in OpenShift 4.2.x environment. Like to read or need to learn more English and knowledge of the relevant scene small partners can download the book by reading the above slowly.

In VirtualService can be simulated fault injection request for HTTP, so chaotic test. This is achieved by providing HTTPFaultInjection delay and abort the properties VirtualService.

• delay to delay access.
• abort to terminate access.

The following recommendation to the micro-services, for example to achieve delay and abort test chaos. Need to follow the "OpenShift Istio-Tutorial (6) 4 of the service resilience" and "prepare the environment" deployment recommendation v1 and recommendation 2, and for the recommendation to remove all the VirtualService and DestinationRule.

1. First check istiofiles / destination-rule-recommendation.yml file, which defines the name of the recommendation DestinationRule, it will send a request to the called recommendation Service assigned to the background average lable comprising all "app = recommendation" of Pod.

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: recommendation
spec:
  host: recommendation
  subsets:
  - labels:
      app: recommendation
    name: app-recommendation

2. Execute commands, create DestinationRule named for the Service's recommendation.

$ oc apply -f istiofiles/destination-rule-recommendation.yml

Delay fault injection

1. See istiofiles / virtual-service-recommendation-delay.yml, which defines the delay of the request 7s 50% of injected VirtualService.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: recommendation
spec:
  hosts:
  - recommendation
  http:
  - fault:
      delay:
        fixedDelay: 7.000s
        percent: 50
    route:
    - destination:
        host: recommendation
        subset: app-recommendation

2. Execute command to create VirtualService.

$ oc apply -f istiofiles/virtual-service-recommendation-delay.yml

3. Run, the response time of the test validation request. Some may find request returns faster, some slower return.

$ export INGRESS_GATEWAY=$(oc get route istio-ingressgateway -n istio-system -o 'jsonpath={.spec.host}')
$ ./scripts/run.sh $INGRESS_GATEWAY/customer

4. Kiali into the console Distributed Trace, select recommendation in Service, select 1m in Lookback, and then click Search Traces button on the right. You can then track to see the time of the request and consumed below. Where the request is fast ms level, slow requests over 7s.
   

Fault injection termination

1. View istiofiles / virtual-service-recommendation-503.yml file, he defines the termination abort fault, HTTP 503 return 50% of the recommendation request.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: recommendation
spec:
  hosts:
  - recommendation
  http:
  - fault:
      abort:
        httpStatus: 503
        percent: 50
    route:
    - destination:
        host: recommendation
        subset: app-recommendation

2. Execute command to create VirtualService.

$ oc apply -f istiofiles/virtual-service-recommendation-503.yml

3. Run micro continuous access to customer service. We can see part 503 will return results is wrong.

$ export INGRESS_GATEWAY=$(oc get route istio-ingressgateway -n istio-system -o 'jsonpath={.spec.host}')
$ ./scripts/run.sh $INGRESS_GATEWAY/customer
customer => preference => recommendation v1 from '67976848-4l4s7': 13
customer => Error: 503 - preference => Error: 503 - fault filter abort
customer => Error: 503 - preference => Error: 503 - fault filter abort
customer => preference => recommendation v1 from '67976848-4l4s7': 14
customer => preference => recommendation v1 from '67976848-4l4s7': 15
customer => Error: 503 - preference => Error: 503 - fault filter abort