Kubernetes relies on the kube-proxy component to implement service communication and load balancing. In this process, because SNAT is used to translate the source address, the service in the Pod cannot get the real client IP address information. This article mainly answers the question of how the load in the Kubernetes cluster obtains the real IP address of the client.
Create a backend service
Service selection
Select here containous/whoamias the backend service image. On the introduction page of Dockerhub, you can see that when accessing its port 80, the relevant information of the client will be returned. In the code, we can get this information in the Http header.
Hostname : 6e0030e67d6a
IP : 127.0.0.1
IP : ::1
IP : 172.17.0.27
IP : fe80::42:acff:fe11:1b
GET / HTTP/1.1
Host: 0.0.0.0:32769
User-Agent: curl/7.35.0
Accept: */*
Cluster environment
Briefly describe the status of the cluster. The cluster has three nodes, one master and two worker nodes. As shown below:
Create service
- Create corporate spaces, projects
As shown in the figure below, here the enterprise space and project are named realip
- Create service
Create a stateless service here, select containous/whoamiMirror , and use the default port.
- Change the service to NodePort mode
Edit the external network access mode of the service and change it to NodePort mode.
Check the NodePort port of the access service and find that the port is 31509.
- access service
When the browser opens the EIP+ :31509of the Master node, it returns the following:
Hostname: myservice-fc55d766-9ttxt
IP: 127.0.0.1
IP: 10.233.70.42
RemoteAddr: 192.168.13.4:21708
GET / HTTP/1.1
Host: dev.chenshaowen.com:31509
User-Agent: Chrome/86.0.4240.198 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate
Accept-Language: zh-CN,zh;q=0.9,en;q=0.8
Cookie: lang=zh;
Dnt: 1
Upgrade-Insecure-Requests: 1
It can be seen that RemoteAddr is the IP of the Master node, not the real IP address of the access client. The host here refers to the address of the access entrance. For the convenience of quick access, I use the domain name, which does not affect the test results.
Get real IP directly through NortPort access
In the above access, the reason why the real IP of the client cannot be obtained is that the source IP of the access SVC has changed due to SNAT. Changing the externalTrafficPolicy of the service to Local mode can solve this problem.
Open the configuration edit page for the service
Set the service's externalTrafficPolicy to Local mode.
Access the service, you can get the following:
Hostname: myservice-fc55d766-9ttxt
IP: 127.0.0.1
IP: 10.233.70.42
RemoteAddr: 139.198.254.11:51326
GET / HTTP/1.1
Host: dev.chenshaowen.com:31509
User-Agent: hrome/86.0.4240.198 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate
Accept-Language: zh-CN,zh;q=0.9,en;q=0.8
Cache-Control: max-age=0
Connection: keep-alive
Cookie: lang=zh;
Dnt: 1
Upgrade-Insecure-Requests: 1
Cluster hides the client source IP, potentially causing a second hop to another node, but with good overall load distribution. Local preserves the client source IP and avoids the second hop for LoadBalancer and NodePort type services, but there is a potential risk of uneven traffic spreading.
Here's a comparison chart:
When a request falls to a node that does not have a service pod, it will be unreachable. When accessing with curl, it will always stop at TCP_NODELAY, and then prompt timeout:
* Trying 139.198.112.248...
* TCP_NODELAY set
* Connection failed
* connect to 139.198.112.248 port 31509 failed: Operation timed out
* Failed to connect to 139.198.112.248 port 31509: Operation timed out
* Closing connection 0
Get real IP through LB -> Service access
In a production environment, there are usually multiple nodes that receive client traffic at the same time. Using only the Local mode will lead to lower service accessibility. The purpose of introducing LB is to use its probing feature to only forward traffic to nodes where service Pods exist.
Here we take Qingyun's LB as an example to demonstrate. Under the control of Qingyun, you can create LB, add listeners, and monitor port 31509. You can refer to the LB usage documentation ( https://docs.qingcloud.com/product/network/loadbalancer/ ), which will not be repeated here.
As you can see in the figure below, only the master node is active on port 31509 of the service, and the traffic will only be directed to the master node, as expected.
Then continue to increase the number of replicas to 3
Unfortunately, the Pods are not evenly distributed across the three nodes, two of which are on the master. So the backend nodes of the LB are also not fully lit. As shown below:
This requires adding an anti-affinity description to deploy. There are two options. The first is to configure a soft policy, but it cannot guarantee that all LB backends are turned on and the traffic is evenly distributed.
spec:
template:
metadata:
labels:
app: myservice
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- myservice
topologyKey: kubernetes.io/hostname
The other is to configure a hard policy that forces Pods to be allocated on different nodes, but limits the number of replicas, that is, the total number of Pods cannot exceed the total number of Nodes.
spec:
template:
metadata:
labels:
app: myservice
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- myservice
topologyKey: kubernetes.io/hostname
The configuration of the hard strategy is adopted, and all the backends are finally lit, as shown in the following figure:
Obtain real IP through LB -> Ingress -> Service access
If each service occupies an LB, the cost is high, and the configuration is not flexible enough. Every time a service is added, a new port mapping needs to be added to the LB.
Another solution is that the LB directs the traffic of 80 and 443 to the Ingress Controller, and then forwards the traffic to the Service, and then reaches the service in the Pod.
At this time, the LB needs to be able to do transparent transmission at the TCP layer, or forwarding with real IP at the HTTP layer, and set the externalTrafficPolicy of the Ingress Controller to Local mode, but Service does not need to be set to Local mode.
If you want to improve accessibility, you can also refer to the above to configure anti-affinity to ensure that there is an Ingress Controller on each backend node.
Traffic forwarding path:
LB(80/443) -> Ingress Controller(30000) -> myservice(80) -> myservice-fc55d766-xxxx(80)
First, you need to check the configuration of LB [Get Client IP]
Then open the project's external network access gateway
Then add the route for the service
Finally, you need to enter the cluster in [Platform Management] -> [Cluster Management], and find the gateway corresponding to the realip project in the system project kubesphere-controls-system.
Edit the configuration file of the service and change the externalTrafficPolicy to Local mode.
Access the service, you can get the following:
Hostname: myservice-7dcf6b965f-vv6md
IP: 127.0.0.1
IP: 10.233.96.152
RemoteAddr: 10.233.70.68:34334
GET / HTTP/1.1
Host: realip.dev.chenshaowen.com
User-Agent: Chrome/87.0.4280.67 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate
Accept-Language: zh-CN,zh;q=0.9,en;q=0.8
Cache-Control: max-age=0
Cookie: _ga=GA1.2.896113372.1605489938; _gid=GA1.2.863456118.1605830768
Cookie: lang=zh;
Upgrade-Insecure-Requests: 1
X-Forwarded-For: 139.198.113.75
X-Forwarded-Host: realip.dev.chenshaowen.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Original-Uri: /
X-Real-Ip: 139.198.113.75
X-Request-Id: 999fa36437a1180eda3160a1b9f495a4
X-Scheme: https
Summarize
This article introduces three deployment methods for obtaining real IP:
- Get real IP directly through NortPort access
Subject to the Local mode, the service may be inaccessible. It is necessary to ensure that the node that provides the entry to the outside world must have a service load.
- Get real IP through LB -> Service access
Using the detection capability of LB can improve the accessibility of services. It is suitable for scenarios where there are few services or one LB per service is preferred.
- Obtain real IP through LB -> Ingress -> Service access
The traffic of ports 80 and 443 is transferred to the Ingress Controller through the LB, and then the service is distributed. However, if the Ingress Controller uses the Local mode, each backend node of the LB is required to have a copy of the Ingress Controller. It is suitable for scenarios with a large number of externally exposed services.
Of course, it can also be used in combination. For services that do not need to obtain the real IP of the client, you can continue to use the Cluster mode.
refer to
- https://hub.docker.com/r/containous/whoami
- https://kubernetes.io/zh/docs/tutorials/services/source-ip/
- https://docs.qingcloud.com/product/network/loadbalancer/
About KubeSphere
KubeSphere ( https://kubesphere.io) is an open source container hybrid cloud built on top of Kubernetes, providing full-stack IT automation operation and maintenance capabilities and simplifying enterprise DevOps workflows.
KubeSphere has been approved by Aqara Smart Home, Original Life, Sina, Hua Xia Bank, Sichuan Airlines, Sinopharm Group, WeBank, Zijin Insurance, Zhongtong, PICC Life Insurance, China Taiping Insurance, China Mobile Jinke, Radore, ZaloPay, etc. Used by thousands of businesses. KubeSphere provides a developer-friendly wizard-style operation interface and rich enterprise-level functions, including multi-cloud and multi-cluster management, Kubernetes resource management, DevOps (CI/CD), application life cycle management, microservice governance (Service Mesh), multi- Tenant management, monitoring logs, alarm notifications, audit events, storage and network management, GPU support and other functions help enterprises quickly build a powerful and feature-rich container cloud platform.
GitHub: https://github.com/kubesphere official website (China station): https://kubesphere.com.cn WeChat group: Please search to add group assistant WeChat account kubesphere
This article is published by OpenWrite , a multi-post blog platform !