foreword
Recently, the department has a need to whitelist some client IPs. Only within the scope of the whitelist can some business operations be performed. According to the usual practice of our department, we will package a client package and provide it to the business side. ( Note: Our project is running on K8S) I thought it was not a difficult function. The small partner in the department realized the function in less than a day. He can get the correct client IP through local debugging. , but published to the test environment, and found that the obtained client IP has always been the IP of the node. The little partner behind has been checking for a long time, but has no clue, so he asked me to help and keep checking. Today's article is mainly to review the process
Troubleshooting process
First, check the implementation logic of obtaining the client IP.
public class IpUtils {
private static Logger logger = LoggerFactory.getLogger(IpUtils.class);
private static final String IP_UTILS_FLAG = ",";
private static final String UNKNOWN = "unknown";
private static final String LOCALHOST_IP = "0:0:0:0:0:0:0:1";
private static final String LOCALHOST_IP1 = "127.0.0.1";
/**
* 获取IP地址
*
*/
public static String getIpAddr(HttpServletRequest request) {
String ip = null;
try {
//以下两个获取在k8s中，将真实的客户端IP，放到了x-Original-Forwarded-For。而将WAF的回源地址放到了 x-Forwarded-For了。
ip = request.getHeader("X-Original-Forwarded-For");
System.out.println("X-Original-Forwarded-For:" + ip);
if (StringUtils.isEmpty(ip) || UNKNOWN.equalsIgnoreCase(ip)) {
ip = request.getHeader("X-Forwarded-For");
}
//获取nginx等代理的ip
if (StringUtils.isEmpty(ip) || UNKNOWN.equalsIgnoreCase(ip)) {
ip = request.getHeader("x-forwarded-for");
System.out.println("x-forwarded-for:" + ip);
}
if (StringUtils.isEmpty(ip) || UNKNOWN.equalsIgnoreCase(ip)) {
ip = request.getHeader("Proxy-Client-IP");
}
if (StringUtils.isEmpty(ip) || ip.length() == 0 || UNKNOWN.equalsIgnoreCase(ip)) {
ip = request.getHeader("WL-Proxy-Client-IP");
}
if (StringUtils.isEmpty(ip) || UNKNOWN.equalsIgnoreCase(ip)) {
ip = request.getHeader("HTTP_CLIENT_IP");
}
if (StringUtils.isEmpty(ip) || UNKNOWN.equalsIgnoreCase(ip)) {
ip = request.getHeader("HTTP_X_FORWARDED_FOR");
}
//兼容k8s集群获取ip
if (StringUtils.isEmpty(ip) || UNKNOWN.equalsIgnoreCase(ip)) {
ip = request.getRemoteAddr();
System.out.println("getRemoteAddr:" + ip);
if (LOCALHOST_IP1.equalsIgnoreCase(ip) || LOCALHOST_IP.equalsIgnoreCase(ip)) {
//根据网卡取本机配置的IP
InetAddress iNet = null;
try {
iNet = InetAddress.getLocalHost();
} catch (UnknownHostException e) {
logger.error("getClientIp error: {}", e);
}
ip = iNet.getHostAddress();
}
}
} catch (Exception e) {
logger.error("IPUtils ERROR ", e);
}
//使用代理，则获取第一个IP地址
if (!StringUtils.isEmpty(ip) && ip.indexOf(IP_UTILS_FLAG) > 0) {
ip = ip.substring(0, ip.indexOf(IP_UTILS_FLAG));
}
return ip;
}
}
This logic seems to be no problem, because the local debugging can get the correct client IP, but the test environment can't get it, there is a high probability that there is a problem with the environment. So the direction is turned to the difference of positioning environment
Environmental positioning
test environment
The access process of our test environment is client –> k8s service nodeport—>pod
Find the answer in this article by searching https://kubernetes.io/zh-cn/docs/tutorials/services/source-ip/
.
In the Kubernetes Service forwarding scenario, regardless of the load balancing forwarding mode of iptbales or ipvs, SNAT will be performed on the data packet during forwarding, that is, the real source IP of the client will not be retained
overall process
The link above is also posted to understand the solution
The specific steps are
1. Step 1: The configuration of the business pod is scheduled to the designated node
example
spec:
nodeName: node1 #指定pod节点配置
containers:
- name: pod-name
2. Step 2: Change the default configuration of externalTrafficPolicy: Cluster in the service yaml of the business to externalTrafficPolicy: Local
example
spec:
type: NodePort
externalTrafficPolicy: Local
3. Step 3: Access through the node node + nodeport specified on the pod
example
http://node1:nodeport
Assuming that node1 and node2 are deployed, specific services can only be accessed through node1:nodeport. If node2:nodeport is used, the requested data packet will be discarded
Through the above solution, the problem that the correct client ip cannot be obtained through the service nodeport in the test environment is solved
uat environment
When the test environment is ok, publish the project to the UAT environment, and if there is no accident, there will be another accident.
The access process of uat is client- -> nginx+keepalive --> ingress --> pod
Because the access methods are different, the solutions are different. Through the search, I learned that the transmission of * user ip depends on the X-Forwarded- parameter. But by default, ingress is not enabled, so we need to enable it. The following parameters are required to enable
- use-forwarded-headers : If set to True, pass the set X-Forwarded-* Header to the backend,
use this option when Ingress is behind L7 proxy/load balancer. If set to false, the incoming X-Forwarded-*Header will be ignored.
Use this option when the Ingress is directly exposed to the Internet or behind the load balancer of the L3/packet, and will not change the source IP in the packet. . - forwarded-for-header : Set the Header field used to identify the original IP address of the client. The default value is X-Forwarded-For. If you want to change it to a custom field name, you can add it under the data configuration block of configmap: forwarded-for-header: "THE_NAME_YOU_WANT"
- compute-full-forwarded-for : Append remote address to X-Forwarded-For
Header instead of replacing it. When this option is enabled the backend application is responsible for excluding and extracting the client IP based on its own list of trusted proxies.
Detailed introduction can be viewed on the official website
We add the following content to the Configmap of the Ingress Nginx Controller
apiVersion: v1
kind: ConfigMap
......
data:
compute-full-forwarded-for: "true"
use-forwarded-headers: "true"
forwarded-for-header:"X-Forwarded-For"
After configuration, it was found that it was useless and had no effect. After checking a lot of information, I found that the Internet is so configured. Later, I thought that there was something wrong with the nginx-keepalive link, so I asked the operation and maintenance to see if there is X-Forwarded- configured on nginx. For, he said no, then I asked him if he could configure it. His answer was that X-Forwarded-For could not be used because the ssl_preread module was enabled on nginx. Later, he asked if he could change it. He replied that it was the company
behind It is time to use F5, and it will be fine to configure it at that time. And he has a lot of things to do at the moment, so he doesn't have time to help me with this.
Since the business is rushing and there is no time for operation and maintenance, we communicated with the business side and adopted a compromise solution, that is, by customizing the request header, we configured an attribute in the client package. Fill in the list ip, example
lybgeek:
whilte-ips: 192.168.1.1,192.168.2.1
When the business project starts, the client package will automatically insert the configured whitelist into the request header
header("x-custom-forwarded-for",whilteIps)
On the server side, obtain the client ip and do the following modification
@Slf4j
public final class IPHelper {
private IPHelper(){
}
private static final String IP_UTILS_FLAG = ",";
private static final String UNKNOWN = "unknown";
private static final String LOCALHOST_IP = "0:0:0:0:0:0:0:1";
private static final String LOCALHOST_IP1 = "127.0.0.1";
private static final String[] headersToTry = {
//在k8s中，将真实的客户端IP，放到了x-Original-Forwarded-For。而将WAF的回源地址放到了 x-Forwarded-For了。
"X-Original-Forwarded-For",
"X-Forwarded-For",
"Proxy-Client-IP",
"WL-Proxy-Client-IP",
"HTTP_X_FORWARDED_FOR",
"HTTP_X_FORWARDED",
"HTTP_X_CLUSTER_CLIENT_IP",
"HTTP_CLIENT_IP",
"HTTP_FORWARDED_FOR",
"HTTP_FORWARDED",
"HTTP_VIA",
"REMOTE_ADDR",
// 自定义请求头
"X-Custom-Forwarded-For",
};
/**
* 获取用户的真正IP地址
*
* @param request request对象
* @return 返回用户的IP地址
*/
@SneakyThrows
public static String getIpAddr(HttpServletRequest request) {
String ip = null;
for (String header : headersToTry) {
ip = request.getHeader(header);
if (StringUtils.hasText(ip) && !UNKNOWN.equalsIgnoreCase(ip)){
log.info("hit the target client ip -> 【{}】 by header --> 【{}】",ip,header);
return ip;
}
}
//兼容k8s集群获取ip
if (org.springframework.util.StringUtils.isEmpty(ip) || UNKNOWN.equalsIgnoreCase(ip)) {
ip = request.getRemoteAddr();
if (LOCALHOST_IP1.equalsIgnoreCase(ip) || LOCALHOST_IP.equalsIgnoreCase(ip)) {
//根据网卡取本机配置的IP
InetAddress iNet = null;
try {
iNet = InetAddress.getLocalHost();
} catch (UnknownHostException e) {
log.error("getIpAddr error: {}", e);
}
ip = iNet.getHostAddress();
}
log.info("hit the target client ip -> 【{}】 by method 【getRemoteAddr】 ",ip);
}
//使用代理，则获取第一个IP地址
if (!org.springframework.util.StringUtils.isEmpty(ip) && ip.indexOf(IP_UTILS_FLAG) > 0) {
ip = ip.substring(0, ip.indexOf(IP_UTILS_FLAG));
}
return ip;
}
}
In fact, what I did was to refactor the original tool class a little bit, and add a custom request header X-Custom-Forwarded-For
Summarize
The summary of this review is that many things are not taken for granted, and some simple things may have pitfalls in them. When encountering cross-departmental cooperation, if we encounter some force majeure factors, in addition to upward feedback, we must also have a bottom-up plan, otherwise we will be very passive