In some cases, we want to replicate traffic in Amazon EKS to other EKS clusters or Kubernetes clusters for traffic distribution and replication purposes. Here are some common scenarios:
Test the new version: When deploying a new version of the application, production traffic can be copied to the new version of the application in the test environment to test its performance and stability without affecting the production environment application.
Failure analysis: When data analysis or troubleshooting is required, the traffic can be copied to the test environment for more effective troubleshooting analysis. At this point, you can run specific tools in the test environment or add debugging information to help troubleshoot the issue without affecting the production environment.
Migration/Capacity Planning: When migration or capacity assessment of an application is required, traffic can be sent to different targets for benchmarking and performance analysis. At this point, different load and traffic patterns can be simulated by replicating the traffic into a test environment, and the test results can be analyzed for capacity planning or migration decisions.
Through the mirror function in Nginx Ingress, it is very convenient to mirror the request to other environments, and the response of the Mirror target will be ignored. This function is very useful to view the response of the request in the "test" backend.
In this article, we will deploy two EKS clusters to demonstrate the effect of Nginx mirror. EKS-Source-Cluster is used as the source cluster, and EKS-Destination-Cluster is used as the target cluster. Both clusters deploy nginx-ingress and httpbin applications for testing.
prerequisite
The environment information used in this article is as follows, and the configuration of different versions will be different:
Amazon EKS: 1.25
Nginx-ingress：v1.7.0
Amazon Load Balancer Controler: v2.5.1
To deploy Amazon Load Balancer Controller in two EKS clusters, you can refer to the link:
https://docs.aws.amazon.com/zh_cn/eks/latest/userguide/aws-load-balancer-controller.html
Deploy Nginx-ingress in two EKS clusters:
Solution deployment steps
helm upgrade --install ingress-nginx ingress-nginx \
--repo https://kubernetes.github.io/ingress-nginx \
--namespace ingress-nginx --create-namespaceSwipe left to see more
Check the external-ip of nginx-ingress in the target cluster EKS-Destination-Cluster and record it:
$ kubectl get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 172.20.246.159 a3fa8caf2f1ee4427852c0c9dc1d4249-1800757104.us-east-1.elb.amazonaws.com 80:30606/TCP,443:31247/TCP 29m
ingress-nginx-controller-admission ClusterIP 172.20.129.253 <none>Swipe left to see more
Deploy the httpbin application in two EKS clusters:
apiVersion: apps/v1
kind: Deployment
metadata:
name: httpbin
spec:
selector:
matchLabels:
app: httpbin
replicas: 1
template:
metadata:
labels:
app: httpbin
spec:
containers:
- name: httpbin
image: kennethreitz/httpbin
ports:
- containerPort: 80Swipe left to see more
Deploy httpbin-svc in two EKS clusters:
apiVersion: v1
kind: Service
metadata:
name: httpbin-svc
spec:
selector:
app: httpbin
type: ClusterIP
ports:
- name: http
port: 80
targetPort: 80Swipe left to see more
on Amazon EKS
Deploy ingress resources
Create mirror-source-ingress.yaml:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: httpbin-source-ingress
annotations:
# 指定使用nginx-ingress
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /
# mirror-target填入EKS-Destination-Cluster的nginx-ingress external-ip域名
nginx.ingress.kubernetes.io/mirror-target: https://a3fa8caf2f1ee4427852c0c9dc1d4249-1800757104.us-east-1.elb.amazonaws.com/$request_uri
spec:
rules:
# host填入EKS-Source-Cluster的nginx-ingress external-ip域名
- host: a1008e8ef462544b9ba6fb7e68352f7d-92988604.us-east-1.elb.amazonaws.com
http:
paths:
- path: /httpbin
pathType: Prefix
backend:
service:
name: httpbin-svc
port:
name: httpSwipe left to see more
Create httpbin-source-ingress in the source cluster EKS-Source-Cluster :
kubectl apply -f mirror-source-ingress.yaml
$ kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
httpbin-ingress <none> a1008e8ef462544b9ba6fb7e68352f7d-92988604.us-east-1.elb.amazonaws.com 80 10sSwipe left to see more
Create mirror-destination-ingress.yaml:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: httpbin-destination-ingress
annotations:
# 指定使用nginx-ingress
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
# host填入EKS-Destination-Cluster的nginx-ingress external-ip域名
- host: a3fa8caf2f1ee4427852c0c9dc1d4249-1800757104.us-east-1.elb.amazonaws.com
http:
paths:
- path: /httpbin
pathType: Prefix
backend:
service:
name: httpbin-svc
port:
name: httpSwipe left to see more
Create httpbin-destination-ingress in the target cluster EKS-Destination-Cluster :
kubectl apply -f mirror-source-ingress.yaml
$ kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
httpbin-source-ingress <none> a1008e8ef462544b9ba6fb7e68352f7d-92988604.us-east-1.elb.amazonaws.com 80 5sSwipe left to see more
Note: nginx.ingress.kubernetes.io/mirror-target: https://a3fa8caf2f1ee4427852c0c9dc1d4249-1800757104.us-east-1.elb.amazonaws.com/$request_uri
All requests for this ingress can be forwarded to
a3fa8caf2f1ee4427852c0c9dc1d4249-1800757104.us-east-1.elb.amazonaws.com.
Verify mirror effect
Visit httpbin-source-ingress:
curl http://a1008e8ef462544b9ba6fb7e68352f7d-92988604.us-east-1.elb.amazonaws.com/httpbinSwipe left to see more
In EKS-Destination-Cluster, check whether nginx-ingress has received the request from mirror:
kubectl logs --tail=0 -f ingress-nginx-controller-6b8bfd7f69-8jb7d -n ingress-nginxSwipe left to see more
Other configuration items
By default, the request body is sent to the mirror backend, but this can be turned off by applying the following command:
nginx.ingress.kubernetes.io/mirror-request-body: "off"Swipe left to see more
By default, the header Host of a mirror request will be set to be the same as the host part of the uri in the mirror-target annotation, this can be overridden via the mirror-host annotation:
nginx.ingress.kubernetes.io/mirror-host: "test.env.com"Swipe left to see more
Summarize
Using the Nginx ingress mirror function to implement traffic replication on the Amazon EKS cluster can help us perform performance testing, troubleshooting, and feature development more effectively. Without affecting the production environment, we can continuously receive and analyze actual traffic in the mirror environment. This article provides detailed steps to implement this function on the EKS cluster, hoping to help you in your development.
reference link
https://kubernetes.github.io/ingress-nginx/deploy/
https://docs.aws.amazon.com/zh_cn/eks/latest/userguide/aws-load-balancer-controller.html
https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#mirror
The author of this article
Chen Jiajun
Amazon cloud technology solution architect, currently mainly responsible for helping customers with cloud architecture design and technical consultation, and has an in-depth understanding of containerization and other technical directions.
I heard, click the 4 buttons below
You will not encounter bugs!
