Author: Zheng Wei @Graphite Documentation
1. Background
As one of the user identity's attributes , the request IP is a very important basic data. In many scenarios, we will do network security attack prevention or access risk control based on client request IP. X-Forwarded-For Usually we can get the real IP through the HTTP protocol Request Headers header. However X-Forwarded-For , is the way to get the real IP through the header really reliable?
2. Concept
X-Forwarded-For is an HTTP extension header. There is no definition of it in the HTTP/1.1 (RFC 2616) standard. It was originally introduced by the caching proxy software Squid to represent the real IP of the HTTP requester. Now it has become the de facto standard and is used by major HTTP proxies. , load balancing and other forwarding services are widely used and written into the RFC 7239 (Forwarded HTTP Extension) standard.
Some time ago, after upgrading the Gin framework to 1.7.2, a certain HTTP service of Graphite document suddenly found a "Bug". After the upgrade, the server could not obtain the correct client IP, and instead it was the Nginx Ingress IP in the Kubernetes cluster. So we decided to get the corresponding source code of the client from Gin to check it out.
The business side service used the v1.6.3 version before. Let's take a look at the Context.ClientIP() implementation of this version method:
// ClientIP 方法可以获取到请求客户端的IPfunc (c *Context) ClientIP() string {// 1. ForwardedByClientIP 默认为 true，此处会优先取 X-Forwarded-For 值，// 如果 X-Forwarded-For 为空，则会再尝试取 X-Real-Ipif c.engine.ForwardedByClientIP {clientIP := c.requestHeader("X-Forwarded-For")clientIP = strings.TrimSpace(strings.Split(clientIP, ",")[0])if clientIP == "" {clientIP = strings.TrimSpace(c.requestHeader("X-Real-Ip"))}if clientIP != "" {return clientIP}}// 2. 如果我们手动配置 ForwardedByClientIP 为 false 且 X-Appengine-Remote-Addr 不为空，则取 X-Appengine-Remote-Addr 作为客户端IPif c.engine.AppEngine {if addr := c.requestHeader("X-Appengine-Remote-Addr"); addr != "" {return addr}}// 3. 最终才考虑取对端 IP 兜底if ip, _, err := net.SplitHostPort(strings.TrimSpace(c.Request.RemoteAddr)); err == nil {return ip}return ""}
Look at the v1.7.2 version again, the Contexnt.ClientIP() method implementation:
func (c *Context) RemoteIP() (net.IP, bool) {...remoteIP := net.ParseIP(ip) // 获取客户端 IP...// trustedCIDRs 由 engine 启动时配置的 TrustedProxies 数组解析而来，表示可以信任的前置代理 CIDR 列表。只有配置了 engine.TrustedProxies 才有可能解析出正确的可信任 CIDR 列表。// 只有 CIDR 列表不为空，这里才会将 remoteIP 和已配置可信 CIDR 列表进行比对。CIDR 列表中任一 CIDR 包含对端 IP，则将第二个返回值置为 true，表示对端 IP 可信任。if c.engine.trustedCIDRs != nil {for _, cidr := range c.engine.trustedCIDRs {if cidr.Contains(remoteIP) {return remoteIP, true}}}return remoteIP, false}func (c *Context) ClientIP() string {// 1. AppEngine 默认为 false，如果应用通过 Google Cloud App Engine 部署，或用户手动设置为 true 且 X-Appengine-Remote-Addr 不为空，则会取 X-Appengine-Remote-Addr 值作为客户端 IP。if c.engine.AppEngine {if addr := c.requestHeader("X-Appengine-Remote-Addr"); addr != "" {return addr}}// 2. 否则通过 RemoteIP() 方法判断对端 IP 是否可信，trusted 为 true 表示可信// 详见上文 Context.RemoteIP() 方法内部注释。remoteIP, trusted := c.RemoteIP()if remoteIP == nil {return ""}// 3. 如对端 IP 可信，且 ForwardedByClientIP 为 true(默认为 true），且// RemoteIPHeaders 不为空（默认不为空），则根据 RemoteIPHeaders 中配置的获取 ClientIP 的 Headers 列表中依次获取。默认读取顺序：1. X-Forwarded-For；2. X-Real-IP。if trusted && c.engine.ForwardedByClientIP && c.engine.RemoteIPHeaders != nil {for _, headerName := range c.engine.RemoteIPHeaders {// 对header进行处理，先通过","进行分割，并返回分割后 IP 列表的第一个合法 IPip, valid := validateHeader(c.requestHeader(headerName))if valid {return ip}}}// 3. 最终才考虑取对端 IP 兜底。return remoteIP.String()}// validateHeader 会对入参header进行校验，先通过","进行分割成 IP 列表后，对每个 IP 进行合法性检查，如果任一 IP 不合法，则此Header不合法；否则返回 IP 列表中第一个 IP。func validateHeader(header string) (clientIP string, valid bool) {if header == "" {return "", false}items := strings.Split(header, ",")for i, ipStr := range items {ipStr = strings.TrimSpace(ipStr)ip := net.ParseIP(ipStr)...if i == 0 {clientIP = ipStrvalid = true}}return}
For a detailed discussion of this "Bug", see: https://github.com/gin-gonic/gin/issues/2697.
3. Analysis
First introduce a few concepts/terms that may be covered later:
•$remote_addr : is the real address of the client obtained during the TCP connection between Nginx and the client. The Remote Address cannot be forged, because the establishment of a TCP connection requires a three-way handshake. If the source IP is forged, the TCP connection cannot be established, and there will be no subsequent HTTP request. •X-Client-Real-IP : is a custom header we customize on the cloud vendor's WAF/CDN. It is a header with a value set by the cloud vendor on the edge node $remote_addr , which ensures that we can obtain the real client IP. This feature is basically supported by most cloud vendors (Alibaba Cloud, Huawei Cloud, Tencent Cloud, etc.).
Network requests are usually requests sent by browsers (or other clients), forwarded by layers of network devices, and finally reach the server. Then the request received by each link $remote_addr must be the real IP of the upstream link, which cannot be forged. From the point of view of the whole link, if the source of the final request is required, it X-Forwarded-For can be traced through, and the IP ( $remote_addr ) of each link is added to the X-Forwarded-For field, so that X-Forwarded-For the whole link can be connected in series. which is:
X-Forwarded-For: client_ip, proxy1_ip, proxy2_ip3.1. Can X-Forwarded-For be forged?
Whether the client can forge IP depends on how the edge node (Edge Node) processes the X-Forwarded-For field. The first proxy node directly connected by the client is called an edge node, whether it is a gateway, CDN, LB, etc., as long as this layer is directly accessed by the client, then it is an edge node.
• Edge nodes that do not rewrite X-Forwarded-For If the edge node transparently transmits the HTTP X-Forwarded-For header, it is not safe. The client can forge X-Forwarded-For the value in the HTTP request, and the value will be transparently transmitted backwards.
Therefore, edge nodes that are not rewritten X-Forwarded-For are insecure edge nodes that users can forge X-Forwarded-For .
# 不安全X-Forwareded-For：clientX-Forwarded-For(用户请求中的 X-Forwarded-For)，proxy1，proxy2，proxy3...
• Override X-Forwarded-For Edge Node Edge Node If overwritten $remote_addr to X-Forwarded-For , then this is safe. The edge node obtains remote_addr the real IP of the client. Therefore, the rewritten X-Forwarded-For edge node is a safe edge node, and users cannot forge it X-Forwarded-For .
# 边缘节点用 $remote_addr 来覆盖用户请求中的 X-Forwarded-For：proxy_set_header X-Forwarded-For $remote_addr;# 安全X-Forwareded-For：ClientX-Forwarded-For(边缘节点获取的 remote_addr)，proxy1，proxy2，proxy3...
3.2. How can I get the real client IP?
We consider the solution to obtain the real client IP under the common network topology on the public cloud.
3.2.1. Client->WAF->SLB->Ingress->Pod
3.2.1.1. Using the Nginx real-ip module
To obtain it using the Nginx real-ip module, you need to configure it on the Ingress proxy-real-ip-cidr , and add both the WAF and SLB (layer 7) addresses. After the operation, the server X-Forwarded-For can obtain the real IP by using it, and obtain X-Original-Forwarded-For the fake IP through it.
This scheme has the following disadvantages:
• Since WAF is maintained by cloud vendors, there are many WAF address pools, and addresses will change at the same time. It is extremely difficult to maintain this dynamic configuration. If the update is not timely, the obtained client IP will be inaccurate. • Even if this solution is adopted, if the business party wants to use the new version of Gin ctx. ClientIP() , it still needs to change the code and configure all trusted proxies to TrustedProxies , which will lead to the coupling of infrastructure and business services. This solution is obviously unacceptable , unless the business side is willing to lock the dependent Gin version to v1.6.3.
3.2.1.2. Custom Header with WAF
Many cloud vendors provide custom headers to obtain the client's real IP ( $remote_addr ) capability. We can configure custom headers in advance in the cloud vendor's WAF terminal, such as X-Appengine-Remote-Addr or X-Client-Real-IP to obtain the client's real IP.
This scheme has the following disadvantages:
• If you directly reuse X-Appengine-Remote-Addr this Header, you need to set engine. AppEngine=trueit before you can ctx. ClientIP() get the client IP through the method. • If you use other headers, for example X-Client-Real-IP, you need to encapsulate the method X-Client-Real-IP to obtain the client IP from it yourself, and at the same time, you need to cooperate with the business for transformation.
The architecture is roughly as follows:
3.2.2. Client->CDN->WAF->SLB->Ingress->Pod
3.2.2.2. Using real-ip
To use the real-ip module to obtain, you need to configure the ingress to proxy-real-ip-cidr add the addresses of CDN, WAF and SLB (layer 7), the server can use X-Forwarded-For the real IP, and X-Original-Forwarded-For the fake IP can be obtained through the server.
Advantages and disadvantages of this scheme:
• Compared with 3.2.1, this scenario has more layers of CDN. The CDN address pool is larger than that of WAF, and the address pool changes more frequently. At the same time, the manufacturer does not provide a CDN address pool, so it is basically impossible to maintain the Ingress configuration. • Even if this solution is adopted, if the business party wants to use the new version of Gin ctx. ClientIP() , it still needs to change the code and configure all trusted proxies to TrustedProxies , which will lead to the coupling of infrastructure and business services, which is definitely unacceptable unless the business party Lock the Gin version to 1.6.3.
3.2.2.1. Customize Header with CDN
Advantages and disadvantages of this program: Same as 3.1.1. The architecture is roughly as follows:
3.2.3. Client->SLB->Ingress->Pod
Forgery can use-forwarded-headers be prevented by setting on Ingress .X-Forwarded-For
•use-forwarded-headers=false
Applicable to no proxy layer before Ingress, such as directly hanging on the 4-layer SLB, ingress is rewritten X-Forwarded-For by default to $remote_addr prevent forgery X-Forwarded-For .
•use-forwarded-headers=true
Applicable to the proxy layer in front of Ingress, such as 7-layer SLB or WAF, CDN, etc., is equivalent to adding the following configuration to nginx.conf:
real_ip_header X-Forwarded-For;real_ip_recursive on;set_real_ip_from 0.0.0.0/0; // 默认信任所有 IP，无法避免伪造 X-Forwarded-For
The architecture is roughly as follows:
4. Summary
It is not difficult to see from the above that under the complex and changeable network topology on the cloud, we will frequently maintain the configuration of various network facilities such as CDN, WAF, SLB, and Ingress. If it is necessary to completely guarantee that it X-Forwarded-For cannot be forged, there are only the following two options for the Go service that needs to upgrade the Gin framework:
• Continue to try by getting the client real IP. X-Forwarded-For • Try to get the real IP of the client through other Header.
4.1. Continue to try to get the real IP of the client through X-Forwarded-For
In the business, it is necessary to configure all front-end proxies of the infrastructure to TrustedProxies, including the CDN address pool, WAF address pool, and Kunernetest Nginx Ingress address pool. This solution basically cannot be implemented:
• The configuration is too complicated, once the IP is inaccurate, it is difficult to troubleshoot. • Causes the coupling of business configuration and infrastructure. If the infrastructure is changed to CDN, WAF, and Ingress, the business code must be changed synchronously. • Some trusted proxy IPs cannot be configured at all, such as CDN address pools.
4.2. Try to get the real IP of the client through a custom Header
The infrastructure team provides custom headers to get the real IP of the client, such as X-Client-Real-IP or X-Appengine-Remote-Addr . This solution requires the infrastructure team to make corresponding configurations on the cloud vendor's CDN or WAF terminal. This scheme:
• Simple and reliable configuration, low maintenance cost, you only need to configure custom headers on CDN and WAF terminals. • If used X-Appengine-Remote-Addr, no modifications are required for services using Google Cloud's App Engine. For the services of domestic cloud vendors used, explicit configuration is required engine. AppEngine = true, and then the ctx.ClientIP() method can be continued. • If you use other custom headers, such as X-Client-Real-IP to obtain the real IP of the client, it is recommended to consider encapsulating ClientIP(*gin.Context) string the function X-Client-Real-IP yourself to obtain the client IP from it.
Information link:
-
https://datatracker.ietf.org/doc/html/rfc7239
-
https://github.com/gin-gonic/gin/issues/2697