使用Kubernetes Ingress来实现类似Istio条件路由

微服务 Istio / SpringCloud日益被越来越多的客户关注，Istio提供了各种酷炫的流量控制功能，但Istio距离生产部署可用仍然还有差距。条件路由是否可以在已有的Kubernetes Ingress架构中实现，以最小的代价实现应用的微服务化迁移。答案是肯定的，通过对ingress自定义location/server块的定义，以及upsteam自动生产的规则，可以实现复杂条件路由的支持，类似istio, match。
 

 [不满足] 1. 尝试一，修改Ingress的server-snippet/location-snippet属性来重定向请求到后台服务, 不能按期望跳转

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/load-balance: ip_hash
    nginx.ingress.kubernetes.io/server-snippet: |
      set $agentflag 0;
 
      if ($http_user_agent ~* "(Mobile)" ){
        set $agentflag 1;
      }
 
     if ( $agentflag = 1 ) {
      .  return 301 http://default-cardinfo-homepage-80;
     }
    nginx.ingress.kubernetes.io/upstream-hash-by: ip_hash
    nginx.ingress.kubernetes.io//location-snippet: |
                  if ( $agentflag = 1 ) {
                     proxy_pass http://default-cardinfo-homepage-80;
                  }
  creationTimestamp: 2018-08-27T12:06:32Z
  generation: 2
  name: nginx-test
  namespace: default
  resourceVersion: "43290755"
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/nginx-test
  uid: a310ca06-a9f1-11e8-a613-00163e0c87f1
spec:
  rules:
  - host: stickyingress.example.com
    http:
      paths:
      - backend:
          serviceName: http-svc
          servicePort: 80
        path: /
  - host: mobile.example.com
    http:
      paths:
      - backend:
          serviceName: cardinfo-homepage
          servicePort: 80
        path: /
status:
  loadBalancer:
    ingress:
    - {}

检查ingressController自动生成的nginx server定义, 仅仅依靠server-snippet 以及location-snippet是不能完成请求的自动跳转。server-snippet at ingress level does not work for reverse proxy server-snippet spec

[不满足] 尝试二, 使用IngressController级别的定义 location-snippet， location-snippet会传播到所有的location-block, 不满足期望。

location-snippet of ingress controller is applied to all location block, it does not work as expected
 
location-snippet
 

apiVersion: v1
data:
  location-snippet: |
    if ( $agentflag = 1 ) {
                     proxy_pass http://default-cardinfo-homepage-80;
     }
  proxy-body-size: 20m
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"proxy-body-size":"20m"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app":"ingress-nginx"},"name":"nginx-configuration","namespace":"kube-system"}}
  creationTimestamp: 2018-02-07T09:06:33Z
  labels:
    app: ingress-nginx
  name: nginx-configuration
  namespace: kube-system
  resourceVersion: "43308775"
  selfLink: /api/v1/namespaces/kube-system/configmaps/nginx-configuration
  uid: 312dd92f-0be6-11e8-856b-00163e0c87f1

[解决] 尝试三，使用configuration-snippet + server-snippet + upstream default name 来实现路由跳转到不同的kubernetes service服务。

configuration-snippet is alternative of location snippet, combination configuration-snippet and server-snippet to parse header, set global variable, then proxy specific backend service with upstream name created by IngressController. e.g. default-cardinfo-homepage-80

metadata:
  annotations:
     nginx.ingress.kubernetes.io/server-snippet: |
      set $agentflag 0;

      if ($http_user_agent ~* "(Mobile)" ){
        set $agentflag 1;
      }
    nginx.ingress.kubernetes.io/configuration-snippet: |
      if ( $agentflag = 1 ) {
         proxy_pass http://default-cardinfo-homepage-80;
      }

Put everything together

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/load-balance: ip_hash
    nginx.ingress.kubernetes.io/server-snippet: |
      set $agentflag 0;

      if ($http_user_agent ~* "(Mobile)" ){
        set $agentflag 1;
      }
    nginx.ingress.kubernetes.io/upstream-hash-by: ip_hash
    nginx.ingress.kubernetes.io/configuration-snippet: |
      if ( $agentflag = 1 ) {
         proxy_pass http://default-cardinfo-homepage-80;
      }
  name: nginx-condition
  namespace: default
spec:
  rules:
  - host: stickyingress.example.com
    http:
      paths:
      - backend:
          serviceName: cardinfo-recommendation
          servicePort: 80
        path: /
  - host: mobile.example.com
    http:
      paths:
      - backend:
          serviceName: cardinfo-homepage
          servicePort: 80
        path: /
status:
  loadBalancer:
    ingress:
    - {}

测试

使用不同的Header 模拟不同用户浏览器请求，用户请求根据浏览器的类型被重定向到不同的后台服务和入口。以stickyingress.example.com为入口的用户使用"user-agent: (Mobile)"的用户，自动跳转到后台服务cardinfo-homepage，其他用户仍然进入cardinfo-recommendation。

curl stickyingress.example.com:32619 -v
* Rebuilt URL to: stickyingress.example.com:32619/
* Hostname was NOT found in DNS cache
*   Trying 192.168.33.239...
* Connected to stickyingress.example.com (192.168.33.239) port 32619 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.37.0
> Host: stickyingress.example.com:32619
> Accept: */*
>
< HTTP/1.1 200
* Server nginx/1.13.7 is not blacklisted
< Server: nginx/1.13.7
< Date: Wed, 14 Nov 2018 04:46:19 GMT
< Content-Type: text/plain;charset=UTF-8
< Content-Length: 17
< Connection: keep-alive
<
* Connection #0 to host stickyingress.example.com left intact
recommendation v2

dns:~ # curl stickyingress.example.com:32619 -v -H "user-agent: (Mobile)"

* Rebuilt URL to: stickyingress.example.com:32619/
* Hostname was NOT found in DNS cache
*   Trying 192.168.33.239...
* Connected to stickyingress.example.com (192.168.33.239) port 32619 (#0)
> GET / HTTP/1.1
> Host: stickyingress.example.com:32619
> Accept: */*
> user-agent: (Mobile)
>
< HTTP/1.1 200
* Server nginx/1.13.7 is not blacklisted
< Server: nginx/1.13.7
< Date: Wed, 14 Nov 2018 04:46:31 GMT
< Content-Type: text/plain;charset=UTF-8
< Content-Length: 26
< Connection: keep-alive
< Vary: Accept-Encoding
<
* Connection #0 to host stickyingress.example.com left intact
OK. Vist /cardinfo please.

Conclusion 总结

使用Ingress的自定义location/server block，以及upstream的隐含定义，依然可以实现类似Istio的自动流量路由功能，后台Pod出现更新升级并不会影响到条件跳转。用户可以使用最小的代价通过Kubernetes Ingress实现ABTest，灰度，流量路由，并且享受到Kubernetes服务发现，和滚动更新带来的优势。