OpenShift 4 之Service Mesh教程（6）- DestinationRule的断路器Circuit Breaker和负载均衡功能

本文将模拟一个运行了backend-v1的Pod出问题，并在发给backend-v1微服务的DestinationRule中设置断路器Circuit Breaker。当断路器发现访问微服务出问题后会将出问题的Pod暂时从转发目标中移除，15分钟后再尝试运行在该Pod的微服务是否能正常访问。

本文可在完成《OpenShift 4 之Service Mesh教程（3）- 用Kiali监控微服务运行》后进行操作。在开始正式操作前需要运行以下命令将运行应用的my-istio-app项目里的内容清空即可。

$ scripts/teardown.sh

1. 部署上图中的frontend-v1和backend-v1微服务，并将运行backend-v1的pod扩展到3个实例。

$ oc apply -f ocp/frontend-v1-deployment.yml -n my-istio-app
$ oc apply -f ocp/frontend-service.yml -n my-istio-app
$ oc apply -f ocp/frontend-route.yml -n my-istio-app
$ oc apply -f ocp/backend-v1-deployment.yml -n my-istio-app
$ oc apply -f ocp/backend-service.yml -n my-istio-app
$ oc scale deployment backend-v1 --replicas=3 

2. 查看当前运行的pod的数量。

$ oc get pod
NAME                          READY   STATUS    RESTARTS   AGE
backend-v1-6ddf9c7dcf-2qlls   2/2     Running   0          77s
backend-v1-6ddf9c7dcf-pk57w   2/2     Running   0          3m15s
backend-v1-6ddf9c7dcf-qj64x   2/2     Running   0          77s
frontend-v1-655f4478c-97xbg   2/2     Running   0          3m40s

3. 运行命令发测试请求，确认3个backend-v1的pod都可以接收到请求。

$ scripts/run-50.sh
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-pk57w, Elapsed Time:0.777830 sec
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-2qlls, Elapsed Time:0.778008 sec
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-qj64x, Elapsed Time:0.796773 sec
...

4. 先通过pod名称进入其中一个pod内部

$ oc rsh pod/backend-v1-6ddf9c7dcf-2qlls
Defaulting container name to backend.

5. 在pod中先查看后台服务运行，然后再将其停掉，最后再退出pod环境。

$ curl http://localhost:8080
Backend version:v1,Response:200,Host:backend-v1-6ddf9c7dcf-2qlls, Message: Hello World!!
$ curl http://localhost:8080/stop
Backend version:v1,Response:200,Message: backend-v1-6ddf9c7dcf-2qlls is stopped
$ exit

6. 先查看配置有断路器Circuit Breaker的DestinationRule，然后在创建它。

$ cat istio-files/destination-rule-backend-circuit-breaker-with-pool-ejection.yml
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: backend-destination-rule
spec:
  host: backend
  trafficPolicy:
      connectionPool:
        http: {}
        tcp: {}
      loadBalancer:
        simple: ROUND_ROBIN
      outlierDetection:
      outlierDetection:          #Detect error with condition
        consecutiveErrors: 1     #If found 1 consecutive error (consecutiveErrors)
        baseEjectionTime: 15m    #then eject that pod from pool for 15 minutes (baseEjectionTime)
        interval: 15m            #check again within 15 minutes (interval)
        maxEjectionPercent: 100  #All of pods can be ejected (maxEjectionPercent)
$ oc apply -f istio-files/destination-rule-backend-circuit-breaker-with-pool-ejection.yml

7. 运行命令发测试请求，确认有1个pod已无法接收到请求。

$ scripts/run-50.sh
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-qj64x, Elapsed Time:1.176571 sec
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-pk57w, Elapsed Time:1.071071 sec
Backend:v1, Response Code: 504, Host:, Elapsed Time:0.011490 sec
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-qj64x, Elapsed Time:0.816486 sec
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-pk57w, Elapsed Time:0.805425 sec
Backend:v1, Response Code: 504, Host:, Elapsed Time:0.007417 sec
...

8. 配置断路器Circuit Breaker

$ oc apply -f istio-files/destination-rule-backend-circuit-breaker-with-pool-ejection.yml

9. 运行命令发测试请求，确认在出现一次“Response Code: 504”后就不再出现了，这是因为通过DestinationRule的断路器将出问题的Pod隔离开了。

$ scripts/run-50.sh
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-pk57w, Elapsed Time:0.797786 sec
Backend:v1, Response Code: 504, Host:, Elapsed Time:0.009550 sec
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-qj64x, Elapsed Time:0.819576 sec

10. 再次进入刚刚那个pod内部

$ oc rsh pod/backend-v1-6ddf9c7dcf-2qlls
Defaulting container name to backend.

11. 启动停掉的服务，最后再退出pod环境。

$ curl http://localhost:8080/start
Backend version:v1,Response:200,Message: backend-v1-6ddf9c7dcf-2qlls is started
$ curl http://localhost:8080
Backend version:v1,Response:200,Host:backend-v1-6ddf9c7dcf-2qlls, Message: Hello World!!
$ exit

12. 运行命令发测试请求，确认现在请求还只能发到后端2个pod实例上，这是因为还没有到15分钟，定义断路器的DestinationRule还不能将请求转发实例名为backend-v1-6ddf9c7dcf-2qlls的pod。在15分钟后，DestinationRule可以恢复转发给后台3个pod。

$ scripts/run-50.sh
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-qj64x, Elapsed Time:1.133394 sec
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-pk57w, Elapsed Time:1.096140 sec
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-qj64x, Elapsed Time:0.788913 sec
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-pk57w, Elapsed Time:0.815003 sec

13. 执行命令编辑DestinationRule，将“loadBalancer”的配置从“simple: ROUND_ROBIN”改为“simple: RANDOM”。

$ oc edit destinationrule backend-destination-rule

14. 运行命令发测试请求，确认请求是被随机发送到3个运行backend-v1微服务的Pod上。

$ scripts/run-50.sh
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-2qlls, Elapsed Time:0.801652 sec
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-2qlls, Elapsed Time:0.877466 sec
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-qj64x, Elapsed Time:0.812104 sec
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-pk57w, Elapsed Time:0.821273 sec
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-2qlls, Elapsed Time:0.776068 sec
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-qj64x, Elapsed Time:0.796318 sec
Backend:v1, Response Code: 200, Host:backend-v1-6ddf9c7dcf-qj64x, Elapsed Time:0.807531 sec