proxy_protocol_V2 function

background

proxy_protocol introduction:

Proxy protocol is an Internet protocol developed and designed by Willy Tarreau, the author of HAProxy, in 2010. It adds a small header information to tcp to conveniently transfer client information (protocol stack, source IP, destination IP, source port, destination port, etc.), which is very useful when the network situation is complex and the user's real IP needs to be obtained.

The proxy protocol is divided into two versions: V1 and V2. V1 is human-readable, and V2 is in binary format and supports the tlv function.

v1 introduction:

The format of Proxy protocol V1 is as follows:

PROXY protocol stack source IP destination IP source port destination port rn

For example:

PROXY TCP4 192.168.0.1 192.168.0.11 56324 443\r\n
    GET / HTTP/1.1\r\n
    Host: 192.168.0.11\r\n
    \r\n

v2 introduction:

Compared to V1, v2 utilizes a binary format to achieve higher parsing efficiency and can add specific extended attributes (TLVs)

After the standard address information, if there is additional data, the data is a TLV array. Each tlv structure includes type, length, and value. The following type values have been standardized, and applications should follow the meaning of the type and should not be used for other purposes.

    #define PP2_TYPE_ALPN           0x01
        #define PP2_TYPE_AUTHORITY      0x02
        #define PP2_TYPE_CRC32C         0x03
        #define PP2_TYPE_NOOP           0x04
        #define PP2_TYPE_UNIQUE_ID      0x05
        #define PP2_TYPE_SSL            0x20
        #define PP2_SUBTYPE_SSL_VERSION 0x21
        #define PP2_SUBTYPE_SSL_CN      0x22
        #define PP2_SUBTYPE_SSL_CIPHER  0x23
        #define PP2_SUBTYPE_SSL_SIG_ALG 0x24
        #define PP2_SUBTYPE_SSL_KEY_ALG 0x25
        #define PP2_TYPE_NETNS          0x30

Common applications:

Pass the real IP of the client (especially for non-http applications, http classes can be passed using headers such as x-forward-realip).
Proxy protocol version 2 supports additional TLV fields. It can be forwarded by the front-end SSL protocol uninstaller to forward the client certificate information to the back-end non-HTTP protocol back-end server. One example is the VerneMQ MQTT broker, which can utilize proxy protocol version 2 to obtain the client's certificate details for identity authorization.
Used in K8s environment, ingress terminates SSL for non-Http protocols.
GCP/amazon/arure utilizes specific TLVs for load balancing.

2.Module description

On the basis of being compatible with nginx v1 version sending and parsing, OpenNJet implements the ability to send protocol v2 and customize TLV sending.

3. Configuration instructions:

This configuration takes effect in the stream module.

	Required	Configuration instructions
proxy_protocol	yes	Turn on proxy_protocol
proxy_pp2	yes	Enable proxy_protocol v2. on turns on, off turns off
proxy_pp2_set_tlv	no	proxy_pp2_set_tlv key value format. The key is hexadecimal (two digits, to avoid conflicts with the standard). value can be a constant or variable. For example: proxy_pp2_set_tlv 0x31 test; Get: $proxy_protocol_tlv_0x31 prefix variable. Note: Including the previous source IP, target IP, source port, target port, and tlv value, the total length cannot exceed 4096

4. Application examples

OpenNJet Sidecar application v2 passes original destination address

Business Process:

1. In the K8s environment, the client accesses a service through the domain name (for example, http request "api.test.com:8080"), sends the request to the service_ip corresponding to the service, the firewall hijacks the request, and redirects it to the local listening port 15001 Sidecar process processing.

2. The egress sidecar is a layer 4 proxy that can identify http traffic and forward the http traffic to the local "egress policy" for further processing.

3. The egress policy is a 7-layer proxy server. It will select the appropriate backend real server address (in this example, server instance1, ip:10.0.30.1 or instance2, ip:10.0.30.2). Assume that server1 is selected. , it will connect to 10.0.30.1:8080 and send the request.

3.1 After the request reaches 10.0.30.1, the incoming firewall rule control redirects all requests to "ingress sidecar", which is also a layer 4 proxy and listens to 15006. At the same time, the firewall records the original destination port (8080) in the IP header ( tcp option) .

4. The ingress process must also select http requests and forward them to the ingress policy, which is a layer 7 proxy that listens to 8086

5. After analyzing the content and processing, the ingress policy should be forwarded to the original destination port 8080 (stage 4.2). Since the ingress sidecar is a layer 4 device, it will not modify the http protocol. Therefore, the information of 8080 cannot be passed to the ingress. policy.

solution:

When the Ingress sidecar sends a request to the ingress policy (step 4.2), it passes this information at the beginning of the TCP connection through the proxy protocol. The ingress policy is obtained through standard NJet variables.

Ingress sidecar configuration

stream {
 
     server {
        listen 15006  ;
        njtmesh_dest on;    #参考下面该指令的备注 
        proxy_protocol on;  #开启protocol 功能
        proxy_pp2  on;      #开启protocol V2功能 
        
        proxy_pp2_set_tlv  0x41    $njtmesh_port;  #设置tlv 字段名，以及值
        proxy_pass 127.0.0.1:90;
     }
}

Ingress policy configuration

http {
 
   server {
        listen 8086 proxy_protocol;
        server_name server-90;
        location /{ 
          #todo: 其他的业务配置指令
          proxy_pass 127.0.0.1:$proxy_protocol_tlv_0x41;
        }
 }
}

Note: # njtmesh_dest NJet custom command is a command used to obtain the original target information set by firewall rules. After it is enabled, the original target port can be obtained through $njtmesh_port. Similarly, the original destination address can be obtained through $njtmesh_ip.

5. References and Acknowledgments

Open NJet, like nginx, accesses the information passed through variables, http://nginx.org/en/docs/http/ngx_http_core_module.html#var_proxy_protocol_tlv_
Part of the code refers to https://github.com/dedok/nginx-stream-proxy-protocol-v2 , and I would like to express my gratitude.
The commercial version of nginx has a module https://nginx.org/en/docs/http/ngx_http_proxy_protocol_vendor_module.html , which can parse the custom TLV of cloud service providers such as GCP/amazon/Azure for better LB control, but it has not been seen Setup module.
Protocol rfc reference: https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt

OpenNJet was first based on the basic fork of NGINX1.19 and evolved independently. It has the characteristics of high performance, stability, and easy expansion. It also solves the long-standing problems of NGINX such as difficulty in dynamic configuration and management functions affecting business.

Mail group official website

OpenNJet supports maintaining Proxy_protocol V2