OpenShift Istio-Tutorial (6) 4 of the service recovery (retry, timeout circuit breaker)

This series OpenShift Servic Mesh tutorial is based on Red Hat official public offering " Introducing Istio Service Mesh for Micoservices " publication, I will verify all operations carried out in OpenShift 4.2.x environment. Like to read or need to learn more English and knowledge of the relevant scene small partners can download the book by reading the above slowly.

As Service Mesh architecture, Istio provides awareness and resilience to the problem of micro-services. Mainly timeout (Timeout), the retry (Try), circuit breaker (Circuit Breaker) and reject pool (Pool Ejection) function.

Prepare the environment
before starting the operation, we first DestinationRule and VirtualService previously defined for Recommendation deleted, and then run the recommendation-v2 micro services Pod set a.

$ oc delete -f istiofiles/destination-rule-recommendation_lb_policy_app.yml -n tutorial
$ oc get istio-io -n tutorial
NAME                                             GATEWAYS        HOSTS   AGE
virtualservice.networking.istio.io/customer-vs   [customer-gw]   [*]     5h
NAME                                      AGE
gateway.networking.istio.io/customer-gw   5h
$ oc scale deployment recommendation-v2 --replicas=1 -n tutorial
$ oc get pod -n tutorial
customer-77dc47d7f8-szhd5            2/2     Running   4          6h
preference-v1-55476494cf-xm4dq       2/2     Running   0          3h
recommendation-v1-67976848-4l4s7     2/2     Running   0          3h
recommendation-v2-599867df6c-5ccdx   2/2     Running   0          19m

Retry (Fail Try)

When accessing a service through Service Pod micro errors (such as 503), Istio automatically (default configuration can be modified) attempt to access other operating Pod micro services.

1. Run the following commands in a window, and can be seen at this time recommendation v1 recommendation v2 are alternately be called.

$ export INGRESS_GATEWAY=$(oc get route istio-ingressgateway -n istio-system -o 'jsonpath={.spec.host}')
$ ./scripts/run.sh $INGRESS_GATEWAY/customer
customer => preference => recommendation v2 from '3cbba7a9cde5': 50
customer => preference => recommendation v1 from '67976848-4l4s7': 1451
customer => preference => recommendation v2 from '3cbba7a9cde5': 51
customer => preference => recommendation v1 from '67976848-4l4s7': 1452

At this point in Kiali you can see recommendation v1 and recommendation v2 have a 50% chance of call.

2. Execute the command into the container of micro recommendation v2 and services running in another window in order to simulate the service recommendation v2 micro not operating properly.

$ oc exec -it $(oc get pods|grep recommendation-v2|awk '{ print $1 }'|head -1) -c recommendation /bin/bash

3. In the run inside the container recommendation v2 micro services execute the following command to invoke the service interface micro-simulation runtime exception (return to 503), then exit the container.

bash-4.4$ curl localhost:8080/misbehave
Following requests to / will return a 503
bash-4.4$ exit
exit

4. In the first window to confirm only call micro-recommendation-v1 served.

$ export INGRESS_GATEWAY=$(oc get route istio-ingressgateway -n istio-system -o 'jsonpath={.spec.host}')
$ ./scripts/run.sh $INGRESS_GATEWAY/customer
customer => preference => recommendation v1 from '67976848-4l4s7': 1444
customer => preference => recommendation v1 from '67976848-4l4s7': 1445
customer => preference => recommendation v1 from '67976848-4l4s7': 1446

At this point in Kiali you can see recommendation v1 have a 100% chance of call. Recommandation selected block, then the right side of the mouse on the App red: the Recommendation, then you can see: Although Pod recommandation state 2 are normal, but some Inbound request error.

Recommandation v2 small box is selected, and then the mouse on the right side of the App red: the Recommendation, then you can see the prompt: 100% Inbound request error.

Preference selected block, and then the mouse on the right side of the App red: the preference, then you can see the prompt: Outbound request error part.

The service then resume normal operation micro recommendation v2, then the first window and have access to the services of the micro recommendation v2.

$ oc exec -it $(oc get pods|grep recommendation-v2|awk '{ print $1 }'|head -1) -c recommendation /bin/bash
bash-4.4$ curl localhost:8080/behave
Following requests to / will return a 503
bash-4.4$ exit
exit

6. File istiofiles / virtual-service-recommendation-try.yml modify the default Servic recommendation on access to retry configuration.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: recommendation
spec:
  hosts:
  - recommendation
  http:
  - route:
    - destination:
        host: recommendation
    retries:
      attempts: 3
      perTryTimeout: 2s

Run modified retry configuration.

$ oc apply -f istiofiles/virtual-service-recommendation-try.yml

Timeout (Timeout)

Set timeout to access micro-services, when the time exceeds the Timeout automatic end of the visit.

1. According to this paper, we first began to explain, re-prepare the environment to ensure the recommendation v1 and recommendation v2 can be a normal visit.
2. Execute the following command, delete Deployment normal recommendation v2, and then deploy a timeout version of the recommendation v2.

$ oc delete -f recommendation/kubernetes/Deployment-v2.yml
$ oc apply -f recommendation/kubernetes/Deployment-v2-timeout.yml

3. Execute scripts continuous access to customer, you can find recommendation v2 returns the result slower.

$ export INGRESS_GATEWAY=$(oc get route istio-ingressgateway -n istio-system -o 'jsonpath={.spec.host}')
$ ./scripts/run.sh $INGRESS_GATEWAY/customer
customer => preference => recommendation v1 from '67976848-4l4s7': 3078
customer => preference => recommendation v2 from '379afb614fb1': 1
customer => preference => recommendation v1 from '67976848-4l4s7': 3079
customer => preference => recommendation v2 from '379afb614fb1': 2
CTL+C

4. 为recommendation服务创建一个VirtualService。文件istiofiles/virtual-service-recommendation-timeout.yml内容如下，其中为访问recommendation定义了timeout为1s。

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: recommendation
spec:
  hosts:
  - recommendation
  http:
  - route:
    - destination:
        host: recommendation
    timeout: 1.000s

执行命令，创建VirtualService

$ oc apply -f istiofiles/virtual-service-recommendation-timeout.yml -n tutorial

5. 执行脚本持续访问customer，可以看到由于recommendation v2超时，所以返回的只有recommendation v1。

$ export INGRESS_GATEWAY=$(oc get route istio-ingressgateway -n istio-system -o 'jsonpath={.spec.host}')
$ ./scripts/run.sh $INGRESS_GATEWAY/customer
customer => preference => recommendation v1 from '67976848-4l4s7': 3084
customer => preference => recommendation v1 from '67976848-4l4s7': 3085
customer => preference => recommendation v1 from '67976848-4l4s7': 3086

6. 在Kiali控制台中，选中recommendation的红色方框，鼠标点到右上方App：recommendation后，可以看到50%的Inbound出现错误。由于此步骤没有配置“retry”
   
7. 最后恢复环境，删除超时的recommendation v2部署和recommendation的VirtualService，并重新部署正常的recommendation v2。

$ oc delete -f istiofiles/virtual-service-recommendation-timeout.yml -n tutorial
$ oc delete -f recommendation/kubernetes/Deployment-v2-timeout.yml
$ oc apply -f recommendation/kubernetes/Deployment-v2.yml

断路器（Circuit Breaker）

当一个被调用服务出现错误后，下一次Istio还会将请求发给出错的服务。回顾本文的“重试(Fail Try)”中的第二章截图，访问recommendation v2的失败率是33%。这是由于Istio会轮训将请求发给recommendation v2和recommendation v2，当发现recommendation v2出现错误后再尝试发给recommendation v1。我们可以在DestinationRule上设置断路器（Circuit Breaker），以便在访问某个服务失败后暂时断开对这个服务实例的访问。Istio会在指定的时间后尝试将请求再次发给recommendation v2，如果此时失效的recommendation v2已经恢复正常，Istio会停止作用于recommendation v2的断路器。

1. 根据本文“准备环境”的说明，删除所有recommendation微服务相关的VirtualService、DestinationRule。此时请求可以被轮训发到recommendation v1和recommendation v2上。
2. 进入运行recommendation v2的容器里，执行命令把状态设为disbehave状态，然后退出。

$ oc exec -it -n tutorial $(oc get pods -n tutorial|grep recommendation-v2|awk '{ print $1 }'|head -1) -c recommendation /bin/bash
bash-4.4$ curl localhost:8080/misbehave
Following requests to / will return a 503
bash-4.4$ exit
exit

3. 执行命令，连续访问customer。此时从调用客户端看到的是recommendation v1响应处理的请求。

$ export INGRESS_GATEWAY=$(oc get route istio-ingressgateway -n istio-system -o 'jsonpath={.spec.host}')
$ ./scripts/run.sh $INGRESS_GATEWAY/customer
customer => preference => recommendation v1 from '67976848-4l4s7': 3492
customer => preference => recommendation v1 from '67976848-4l4s7': 3493
customer => preference => recommendation v1 from '67976848-4l4s7': 3494
customer => preference => recommendation v1 from '67976848-4l4s7': 3495
...

4. 在另一个窗口执行以下命令，查看运行recommendation v2的容器日志。我们可以发现recommendation v2微服务还继续不断有日志输出，说明有请求还是不断发给它。因为此时我们还没有为recommendation v2设置断路器。

$ oc logs -f $(oc get pods |grep recommendation-v2|awk '{ print $1 }'|head -1) -c recommendation
recommendation request from 3cbba7a9cde5: 11
recommendation request from 3cbba7a9cde5: 12
recommendation request from 3cbba7a9cde5: 13
...

5. 在第三个窗口执行命令，为recommendation服务创建带有断路器的DestinationRule。文件istiofiles/destination-rule-recommendation_cb_policy_version_v2.yml定义了DestinationRule：

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: recommendation
spec:
  host: recommendation
  subsets:
  - labels:
      version: v1
    name: version-v1
  - labels:
      version: v2
    name: version-v2
  trafficPolicy:
    connectionPool:
      http:
        http1MaxPendingRequests: 1
        maxRequestsPerConnection: 1
      tcp:
        maxConnections: 1
    outlierDetection:
      baseEjectionTime: 3m
      consecutiveErrors: 1
      interval: 1s
      maxEjectionPercent: 100

执行命令为recommendation服务创建DestinationRule。

$ oc apply -f istiofiles/destination-rule-recommendation_cb_policy_version_v2.yml -n tutorial

6. 此时可从从窗口1看到还不断发送对customer的请求，但是窗口2已经不再有日志输出，说明请求没有被发到recommendation
   v2微服务上，这是由于断路器暂时不再将请求发给出问题的recommendation v2微服务。
7. 再次进入运行recommendation v2的容器里，执行命令把状态设为behave状态，然后退出。这样recommendation v2微服务又恢复了正常访问。

$ oc exec -it -n tutorial $(oc get pods -n tutorial|grep recommendation-v2|awk '{ print $1 }'|head -1) -c recommendation /bin/bash
bash-4.4$ curl localhost:8080/behave
Following requests to / will return a 503
bash-4.4$ exit
exit

8. 继续观察窗口1和窗口2的输出，需要等3分钟左右。从以下窗口1的日志可以看到请求再次被转发到recommendation v2，而窗口2也会有新的日志输出，说明recommendation v2已经恢复接收到转发的请求。

customer => preference => recommendation v1 from '67976848-4l4s7': 4522
customer => preference => recommendation v2 from '3cbba7a9cde5': 20
customer => preference => recommendation v1 from '67976848-4l4s7': 4523
customer => preference => recommendation v2 from '3cbba7a9cde5': 21
customer => preference => recommendation v1 from '67976848-4l4s7': 4524
customer => preference => recommendation v2 from '3cbba7a9cde5': 22
customer => preference => recommendation v1 from '67976848-4l4s7': 4525
customer => preference => recommendation v2 from '3cbba7a9cde5': 23