Detailed explanation of kube-proxy proxy mode
kube-proxy currently supports the following proxy modes:
1. Userspace: The earliest load balancing solution, it listens to a port in user space, and all services are forwarded to this port through iptables, and then the internal load is balanced to the actual Pod. The main problem of this method is low efficiency and obvious performance bottleneck, so it is no longer recommended.
For example, to create a service, the corresponding IP: 1.2.3.4, port: 8888, the port randomly selected by kube-proxy is 32890, it will be added in iptables
-A PREROUTING -j KUBE-PORTALS-CONTAINER
-A KUBE-PORTALS-CONTAINER -d 1.2.3.4/32 -p tcp --dport 8888 -j REDIRECT --to-ports 32890
KUBE-PORTALS-CONTAINER is the rule chain created by kube-proxy in iptable, which is executed in the PREROUTING phase
Implementation process:
- When there is a request to access the service, in the PREROUTING stage, jumpKUBE-PORTALS-CONTAINER will be requested;
- This record in KUBE-PORTALS-CONTAINER works, redirecting the request to the port 32890 that kube-proxy just listened to, and the data packet enters the kube-proxy process;
- Then kube-proxy will select one from the endpoint corresponding to the service according to the SessionAffinity as the real service response request.
2. iptables: The currently recommended solution uses iptables rules to achieve service load balancing. The main problem with this method is that too many iptables rules are generated when there are many services, and non-incremental updates will introduce a certain delay, and there are obvious performance problems in large-scale cases.
For example, if a service is created, the corresponding IP: 1.2.3.4, port:8888, and a corresponding backend address: 10.1.0.8:8080, it will be added in iptables (main rules):
-A PREROUTING -j KUBE-SERVICES
-A KUBE-SERVICES -d 1.2.3.4/32 -p tcp –dport 8888 -j KUBE-SVC-XXXXXXXXXXXXXXXX
-A KUBE-SVC-XXXXXXXXXXXXXXXX -j KUBE-SEP-XXXXXXXXXXXXXXXX
-A KUBE-SEP-XXXXXXXXXXXXXXXX -p tcp -j DNAT –to-destination 10.1.0.8:8080
KUBE-SERVICES is the rule chain created by kube-proxy in iptable, which is executed in the PREROUTING stage
Implementation process:
- When requesting to access the service, iptables will jump to KUBE-SERVICES during the prerouting stage;
- Match the first criterion above in KUBE-SERVICES, continue to jump to KUBE-SVC-XXXXXXXXXXXXXXXX;
- Jump to KUBE-SEP-XXXXXXXXXXXXXXXX according to this rule;
- In the KUBE-SEP-XXXXXXXXXXXXXXXX rule, access the real pod address 10.1.0.8:8080 through the DNAT action.
3. ipvs: In order to solve the performance problem of iptables mode, v1.11 added ipvs mode (v1.8 began to support the beta version, and v1.11 GA), using incremental update, and can guarantee the connection during service update Keep it unbroken.
In IPVS mode, kube-proxy observes the changes of service and endpoint objects in Kubernetes, creates corresponding IPVS rules by calling the netlink interface, and periodically synchronizes service, endpoint and IPVS rules of Kubernetes. When accessing a service, IPVS is responsible for selecting a real backend to provide services.
IPVS mode is also based on netfilter's hook function (INPUT stage), which is the same as iptables mode, but uses the hash table of the kernel workspace as the stored data structure. In this mode, iptables can be verified by ipset Whether a specific request satisfies a rule. When the service changes, only the records in ipset need to be updated, and the rule chain of iptables does not need to be changed. Therefore, it can be guaranteed that the rules in iptables will not increase with the increase of services, and provide consistent performance in the case of ultra-large-scale service clusters Effect.
The performance when synchronizing rules is also higher than iptables (only a specific hash table is updated, rather than operating on the entire rule table in iptables mode), so it can provide higher network traffic.
IPVS also provides more flexible algorithms in the selection of backend services:
- rr: round-robin (round-robin algorithm)
- lc: least connection (minimum connection algorithm)
- dh: destination hashing (destination address hash algorithm)
- sh: source hashing (original address hashing algorithm)
- sed: shortest expected delay (shortest expected delay algorithm)
- nq: never queue (never queue algorithm)
4. winuserspace: Same as userspace, but only works on Windows nodes.
This is the end of today's sharing
