OpenShift Service 4 of Mesh Tutorial (6) - DestinationRule breaker Circuit Breaker and load balancing

This article will simulate a running Pod problems backend-v1 and set breaker Circuit Breaker in DestinationRule sent to backend-v1 micro service. The problem will be temporarily removed when the circuit breaker Pod find access to micro-forwarding service problems from the target 15 minutes before trying to run whether the normal access in the Pod micro service.

After the operation - "Kiali monitoring service run by micro OpenShift Service 4 of Mesh Tutorial (3)" article can be completed. Need to run before the official start of the operation the following command to run my-istio-app project applications to empty the contents.

$ scripts/teardown.sh

1. Deployment image above frontend-v1 and backend-v1 micro-services, and will run the backend-v1 pod extended to three examples.

$ oc apply -f ocp/frontend-v1-deployment.yml -n my-istio-app
$ oc apply -f ocp/frontend-service.yml -n my-istio-app
$ oc apply -f ocp/frontend-route.yml -n my-istio-app
$ oc apply -f ocp/backend-v1-deployment.yml -n my-istio-app
$ oc apply -f ocp/backend-service.yml -n my-istio-app
$ oc scale deployment backend-v1 --replicas=3 

2. View the number of currently running pod.

$ oc get pod
NAME                          READY   STATUS    RESTARTS   AGE
backend-v1-6ddf9c7dcf-2qlls   2/2     Running   0          77s
backend-v1-6ddf9c7dcf-pk57w   2/2     Running   0          3m15s
backend-v1-6ddf9c7dcf-qj64x   2/2     Running   0          77s
frontend-v1-655f4478c-97xbg   2/2     Running   0          3m40s

3. Test run command issued request, confirm th backend-v1 pod 3 can receive the request.

$ scripts/run-50.sh
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-pk57w, Elapsed Time:0.777830 sec
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-2qlls, Elapsed Time:0.778008 sec
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-qj64x, Elapsed Time:0.796773 sec
...

4. First enter within one of the pod by pod name

$ oc rsh pod/backend-v1-6ddf9c7dcf-2qlls
Defaulting container name to backend.

5. In the pod first see a background service, and then it stopped, and finally exit the pod environment.

$ curl http://localhost:8080
Backend version:v1,Response:200,Host:backend-v1-6ddf9c7dcf-2qlls, Message: Hello World!!
$ curl http://localhost:8080/stop
Backend version:v1,Response:200,Message: backend-v1-6ddf9c7dcf-2qlls is stopped
$ exit

6. To view the configuration of the circuit breaker Circuit DestinationRule Breaker, then create it.

$ cat istio-files/destination-rule-backend-circuit-breaker-with-pool-ejection.yml
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: backend-destination-rule
spec:
  host: backend
  trafficPolicy:
      connectionPool:
        http: {}
        tcp: {}
      loadBalancer:
        simple: ROUND_ROBIN
      outlierDetection:
      outlierDetection:          #Detect error with condition
        consecutiveErrors: 1     #If found 1 consecutive error (consecutiveErrors)
        baseEjectionTime: 15m    #then eject that pod from pool for 15 minutes (baseEjectionTime)
        interval: 15m            #check again within 15 minutes (interval)
        maxEjectionPercent: 100  #All of pods can be ejected (maxEjectionPercent)
$ oc apply -f istio-files/destination-rule-backend-circuit-breaker-with-pool-ejection.yml

7. Run test request command issued, a pod has been confirmed not received the request.

$ scripts/run-50.sh
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-qj64x, Elapsed Time:1.176571 sec
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-pk57w, Elapsed Time:1.071071 sec
Backend:v1, Response Code: 504, Host:, Elapsed Time:0.011490 sec
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-qj64x, Elapsed Time:0.816486 sec
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-pk57w, Elapsed Time:0.805425 sec
Backend:v1, Response Code: 504, Host:, Elapsed Time:0.007417 sec
...

8. Configuring breaker Circuit Breaker

$ oc apply -f istio-files/destination-rule-backend-circuit-breaker-with-pool-ejection.yml

9. Run test request command issued, a confirmation appears "Response Code: 504" no longer occurs after, this is because the problem Pod DestinationRule isolated by the circuit breaker.

$ scripts/run-50.sh
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-pk57w, Elapsed Time:0.797786 sec
Backend:v1, Response Code: 504, Host:, Elapsed Time:0.009550 sec
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-qj64x, Elapsed Time:0.819576 sec

10. Into that again just inside the pod

$ oc rsh pod/backend-v1-6ddf9c7dcf-2qlls
Defaulting container name to backend.

11. Start stopped services, and finally exit the pod environment.

$ curl http://localhost:8080/start
Backend version:v1,Response:200,Message: backend-v1-6ddf9c7dcf-2qlls is started
$ curl http://localhost:8080
Backend version:v1,Response:200,Host:backend-v1-6ddf9c7dcf-2qlls, Message: Hello World!!
$ exit

12. Run test request command issued, now confirmed that the request can only be sent to the two rear pod example, this is because there is no 15 minutes, DestinationRule defined breaker can not forward the request instance called backend-v1-6ddf9c7dcf- 2qlls the pod. After 15 minutes, DestinationRule can restore forwarded to the background three pod.

$ scripts/run-50.sh
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-qj64x, Elapsed Time:1.133394 sec
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-pk57w, Elapsed Time:1.096140 sec
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-qj64x, Elapsed Time:0.788913 sec
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-pk57w, Elapsed Time:0.815003 sec

13. Run editing DestinationRule, the "loadBalancer" configuration from "simple: ROUND_ROBIN" changed to "simple: RANDOM".

$ oc edit destinationrule backend-destination-rule

14. Test run command issued request, confirmation request is sent to the random Pod 3 micro-th run backend-v1 and services.

$ scripts/run-50.sh
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-2qlls, Elapsed Time:0.801652 sec
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-2qlls, Elapsed Time:0.877466 sec
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-qj64x, Elapsed Time:0.812104 sec
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-pk57w, Elapsed Time:0.821273 sec
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-2qlls, Elapsed Time:0.776068 sec
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-qj64x, Elapsed Time:0.796318 sec
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-qj64x, Elapsed Time:0.807531 sec