table of Contents
Background introduction
In some cases, we are using Kubernetes as the cloud platform for business applications. We want to implement blue-green deployment of applications to iterate application versions. Using lstio is too heavy and complicated, and it is positioned in flow control and grid governance; Ingress -Nginx introduced the Canary function in version 0.21, which can configure multiple versions of applications for the gateway entrance, and use annotations to control the traffic distribution of multiple back-end services.
Ingress-Nginx-Annotation Canary function introduction
If you want to enable the Canary function, you must first set nginx.ingress.kubernetes.io/canary: "true", then you can enable the following comment to configure Canary
- nginx.ingress.kubernetes.io/canary-weightThe percentage of requests to the service specified in Canary Ingress. The value is an integer from 0-100. According to the set value, the percentage of traffic will be allocated to the backend service specified in Canary Ingress.
- nginx.ingress.kubernetes.io/canary-by-header Traffic segmentation based on the request header is suitable for grayscale publishing or A/B testing. When the set hearer value is always, the request traffic will always be allocated to the Canary entrance. When the hearer value is set to never, the request traffic Will not be assigned to the Canary entry, for other hearder values, will be ignored, and request traffic will be assigned to other rules by priority
- nginx.ingress.kubernetes.io/canary-by-header-valueThis configuration should nginx.ingress.kubernetes.io/canary-by-headerbe used together. When the hearer key and value nginx.ingress.kubernetes.io/canary-by-headernginx.ingress.kubernetes.io/canary-by-header-valuein the request match, the request traffic will be allocated to the Canary Ingress entry. For any other hearer values, it will be ignored, and the request traffic will be assigned to other rules by priority
- nginx.ingress.kubernetes.io/canary-by-cookieThis configuration is based on cookie traffic segmentation. It is also suitable for grayscale publishing or A/B testing. When the cookie value is set to always, the request traffic will be routed to the Canary Ingress entrance. When the cookie value is set to never, the request traffic will be routed to the Canary Ingress entry. Will not be routed to the Canary entry, for other values, will be ignored, and request traffic will be assigned to other rules by priority
The canary rules are sorted by priority as follows: canary-by-header -> canary-by-cookie -> canary-weight
Weight-based small-scale version testing
- v1 version layout file
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
labels:
app: echoserverv1
name: echoserverv1
namespace: echoserver
spec:
rules:
- host: echo.chulinx.com
http:
paths:
- backend:
serviceName: echoserverv1
servicePort: 8080
path: /
---
kind: Service
apiVersion: v1
metadata:
name: echoserverv1
namespace: echoserver
spec:
selector:
name: echoserverv1
type: ClusterIP
ports:
- name: echoserverv1
port: 8080
targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: echoserverv1
namespace: echoserver
labels:
name: echoserverv1
spec:
template:
metadata:
labels:
name: echoserverv1
spec:
containers:
- image: mirrorgooglecontainers/echoserver:1.10
name: echoserverv1
ports:
- containerPort: 8080
name: echoserverv1
- View the resources created by the v1 version
$ [K8sSj] kubectl get pod,service,ingress -n echoserver
NAME READY STATUS RESTARTS AGE
pod/echoserverv1-657b966cb5-7grqs 1/1 Running 0 24h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/echoserverv1 ClusterIP 10.99.68.72 <none> 8080/TCP 24h
NAME HOSTS ADDRESS PORTS AGE
ingress.extensions/echoserverv1 echo.chulinx.com 80 24h
- Visit the v1 service, you can see that 10 requests are all access to a pod, that is, the v1 version of the service
$ [K8sSj] for i in `seq 10`;do curl -s echo.chulinx.com|grep Hostname;done
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
- Create a v2 version of the service
We turn on the canary function and set the weight of the v2 version to 50%. This percentage does not accurately distribute requests to the two versions of the service, but fluctuates around 50%.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "50"
labels:
app: echoserverv2
name: echoserverv2
namespace: echoserver
spec:
rules:
- host: echo.chulinx.com
http:
paths:
- backend:
serviceName: echoserverv2
servicePort: 8080
path: /
---
kind: Service
apiVersion: v1
metadata:
name: echoserverv2
namespace: echoserver
spec:
selector:
name: echoserverv2
type: ClusterIP
ports:
- name: echoserverv2
port: 8080
targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: echoserverv2
namespace: echoserver
labels:
name: echoserverv2
spec:
template:
metadata:
labels:
name: echoserverv2
spec:
containers:
- image: mirrorgooglecontainers/echoserver:1.10
name: echoserverv2
ports:
- containerPort: 8080
name: echoserverv2
- View the created resource again
$ [K8sSj] kubectl get pod,service,ingress -n echoserver
NAME READY STATUS RESTARTS AGE
pod/echoserverv1-657b966cb5-7grqs 1/1 Running 0 24h
pod/echoserverv2-856bb5758-f9tqn 1/1 Running 0 4s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/echoserverv1 ClusterIP 10.99.68.72 <none> 8080/TCP 24h
service/echoserverv2 ClusterIP 10.111.103.170 <none> 8080/TCP 4s
NAME HOSTS ADDRESS PORTS AGE
ingress.extensions/echoserverv1 echo.chulinx.com 80 24h
ingress.extensions/echoserverv2 echo.chulinx.com 80 4s
- Access test
It can be seen that 4 requests fell to the v2 version, and 6 fell to the v1 version. In theory, the more requests are said, the closer the number of requests falling to the v2 version is to the set weight of 50%.
$ [K8sSj] for i in `seq 10`;do curl -s echo.chulinx.com|grep Hostname;done
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
A/B test based on header
- Change the orchestration file of the v2 version
Add headerginx.ingress.kubernetes.io/canary-by-header: "v2"
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "50"
nginx.ingress.kubernetes.io/canary-by-header: "v2"
labels:
app: echoserverv2
name: echoserverv2
namespace: echoserver
spec:
rules:
- host: echo.chulinx.com
http:
paths:
- backend:
serviceName: echoserverv2
servicePort: 8080
path: /
---
kind: Service
apiVersion: v1
metadata:
name: echoserverv2
namespace: echoserver
spec:
selector:
name: echoserverv2
type: ClusterIP
ports:
- name: echoserverv2
port: 8080
targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: echoserverv2
namespace: echoserver
labels:
name: echoserverv2
spec:
template:
metadata:
labels:
name: echoserverv2
spec:
containers:
- image: mirrorgooglecontainers/echoserver:1.10
name: echoserverv2
ports:
- containerPort: 8080
name: echoserverv2
- Update access test
Tested the header value of v2:always v2:never v2:true these three hearder values, you can see that when the hearder is v2:always, all traffic will flow into v2, when v2:never, all traffic will flow into v1, when v2 : true, that is, not always/never, traffic will flow into the corresponding version of the service according to the configured weight
$ [K8sSj] kubectl apply -f appv2.yml
ingress.extensions/echoserverv2 configured
service/echoserverv2 unchanged
deployment.extensions/echoserverv2 unchanged
$ [K8sSj] for i in `seq 10`;do curl -s -H "v2:always" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
$ [K8sSj] for i in `seq 10`;do curl -s -H "v2:never" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
$ [K8sSj] for i in `seq 10`;do curl -s -H "v2:true" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
- Custom header-value
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "50"
nginx.ingress.kubernetes.io/canary-by-header: "v2"
nginx.ingress.kubernetes.io/canary-by-header-value: "true"
labels:
app: echoserverv2
name: echoserverv2
namespace: echoserver
spec:
rules:
- host: echo.chulinx.com
http:
paths:
- backend:
serviceName: echoserverv2
servicePort: 8080
path: /
---
kind: Service
apiVersion: v1
metadata:
name: echoserverv2
namespace: echoserver
spec:
selector:
name: echoserverv2
type: ClusterIP
ports:
- name: echoserverv2
port: 8080
targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: echoserverv2
namespace: echoserver
labels:
name: echoserverv2
spec:
template:
metadata:
labels:
name: echoserverv2
spec:
containers:
- image: mirrorgooglecontainers/echoserver:1.10
name: echoserverv2
ports:
- containerPort: 8080
name: echoserverv2
- Update test
It can be seen that only when the header is v2:never, the request traffic will flow into the v2 version, and the traffic of other values will flow into the service of the unreachable version according to the weight setting.
$ [K8sSj] kubectl apply -f appv2.yml
ingress.extensions/echoserverv2 configured
service/echoserverv2 unchanged
deployment.extensions/echoserverv2 unchanged
$ [K8sSj] for i in `seq 10`;do curl -s -H "v2:true" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
$ [K8sSj] for i in `seq 10`;do curl -s -H "v2:always" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
$ [K8sSj] for i in `seq 10`;do curl -s -H "v2:never" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Cookie-based flow control
In fact, the principle of cookie is roughly the same as that of header. It is also the ingress automatic cookie value. If the cookie matches the client's visit, the traffic will flow to the matching backend service.
- Update v2 version of the orchestration file
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "50"
nginx.ingress.kubernetes.io/canary-by-header: "v2"
nginx.ingress.kubernetes.io/canary-by-header-value: "true"
nginx.ingress.kubernetes.io/canary-by-cookie: "user_from_shanghai"
labels:
app: echoserverv2
name: echoserverv2
namespace: echoserver
spec:
rules:
- host: echo.chulinx.com
http:
paths:
- backend:
serviceName: echoserverv2
servicePort: 8080
path: /
---
kind: Service
apiVersion: v1
metadata:
name: echoserverv2
namespace: echoserver
spec:
selector:
name: echoserverv2
type: ClusterIP
ports:
- name: echoserverv2
port: 8080
targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: echoserverv2
namespace: echoserver
labels:
name: echoserverv2
spec:
template:
metadata:
labels:
name: echoserverv2
spec:
containers:
- image: mirrorgooglecontainers/echoserver:1.10
name: echoserverv2
ports:
- containerPort: 8080
name: echoserverv2
- Access test
It can be seen that the access effect is the same as that of the header, but the cookie cannot customize the value
$ [K8sSj] kubectl apply -f appv2.yml
ingress.extensions/echoserverv2 configured
service/echoserverv2 unchanged
deployment.extensions/echoserverv2 unchanged
$ [K8sSj] for i in `seq 10`;do curl -s --cookie "user_from_shanghai" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
# zlx @ zlxdeMacBook-Pro in ~/Desktop/unicom/k8syml/nginx-ingress-canary-deployment [16:01:52]
$ [K8sSj] for i in `seq 10`;do curl -s --cookie "user_from_shanghai:always" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
# zlx @ zlxdeMacBook-Pro in ~/Desktop/unicom/k8syml/nginx-ingress-canary-deployment [16:02:25]
$ [K8sSj] for i in `seq 10`;do curl -s --cookie "user_from_shanghai=always" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
to sum up
The grayscale release can ensure the stability of the overall system. The new version can be tested, found and adjusted at the initial grayscale to ensure its impact. The above content introduces Ingress-Nginx's actual combat Canary Annotation in detail through examples. You can easily implement blue-green release and canary release with the help of Ingress-Nginx