As container technology matures, more and more components are migrated to containers. In the process of technology migration, components such as databases, games, and AI put forward higher requirements on container network performance (latency, throughput, stability) . In order to obtain better latency and throughput performance, major cloud vendors are working to shorten the network access link of the node's internal device so that data packets can be forwarded to the container network card as quickly as possible.
Tencent Cloud Container Service TKE launched the next-generation container network solution with the help of smart network cards. This solution realized that a Pod can exclusively occupy a flexible network card without passing through the node network protocol stack (default namespace), greatly shortening the container access link and shortening Access delay, and make PPS can reach the upper limit of the whole machine. This solution achieves a 50%-70% increase in QPS in the short link scenario compared to the previous container network solution (strategic routing solution, bridge solution); QPS in the long link scenario increases by 40%-60%.
Since it no longer passes through the node network protocol stack, the traditional ClusterIP service access scheme based on iptables and IPVS cannot be directly applied to this scheme. In order to realize that the Pod can directly access the ClusterIP service under this scheme, TKE has launched the share-NS IPVS scheme, which makes it possible to access the IPVS rules of the node network protocol stack under the container network namespace, and at the same time cooperate with the CLB to pass through the Pod, achieving a complete sense The flexible network card is pass-through.
This solution achieves a 40%-60% increase in QPS compared to the iptables solution for the ClusterIP service short link scenario, and a 70%-90% increase in the IPVS solution; a 30%-50% increase in QPS compared to the iptables solution in the long link scenario, the IPVS solution Increase by 50%-70%.
Background of the launch of a new generation of container network solutions
Before introducing the new generation of container network solutions, let me introduce TKE's existing network solutions, the challenges faced by the existing network solutions, and the new demands of customers.
Introduction to existing network solutions
Tencent Cloud Container Service TKE currently provides two container network modes for users to choose from.
GlobalRouter mode: Based on the global routing mode implemented by vpc , it is currently the TKE default network solution. This mode relies on the underlying routing capabilities of vpc. It does not need to configure overlay devices such as vxlan on the nodes to achieve mutual access between the container network and the vpc network. Compared with network solutions such as calico/flannel, there is no additional decapsulation. Performance will also be better.
VPC-CNI mode: TKE is based on the container network capabilities of CNI and VPC elastic network cards. It is suitable for Pod fixed IP, CLB direct Pod, and Pod direct binding EIP. In this network mode, containers and nodes are distributed on the same network plane, and the container IP is the elastic network card IP assigned by the IPAMD component.
GlobalRouter and VPC-CNI modes have currently served tens of thousands of enterprise users of TKE. The two network modes also have certain usage restrictions. See: How to choose the TKE network mode . With the enrichment of customer usage scenarios, TKE customers have proposed more changes to the container network. High demands.
New customer needs for TKE network solutions
In addition to providing container network capabilities for Tencent’s external TKE customers, Tencent’s cloud container service TKE serves as the base for Tencent’s internal business cloud native, supporting Tencent’s internal self-developed businesses such as QQ, Tencent meetings, games, CDB, big data, etc. The following demand points were also received during the business process:
- On the basis of the VPC-CNI mode, further reduce resource consumption, reduce network delay, and increase network throughput (key points)
- Support Pod-level security isolation
- Support CLB pass-through Pod, no longer forwarded by NodePort, improve forwarding performance and have a unified load balancing view
Based on the above scenarios, the TKE team, together with the underlying Tencent Cloud VPC team and virtualization team, launched a new generation of independent network card VPC-CNI solution.
Introduction of TKE's new generation network solution
On the basis of the original VPC-CNI mode single network card multi-IP mode, TKE's new generation network solution is advanced for the container to directly use the elastic network card, seamlessly docking with all the functions of Tencent Cloud private network products, and has achieved great performance (See the performance introduction below for details).
Pressure test data description
-
In order to obtain the QPS under different network schemes, the variables are controlled here, so that nginx Pods of different network schemes run on the same node, use wrk to pressure test different Pods, and make the CPU of the server node close to 100%.
- In order to get the QPS under different service schemes, the variables are controlled here, so that kube-proxy and wrk Pod run on the same node, pressure test the same backend, and make the cpu of the client node close to 100%.
Function introduction
In the new generation VPC-CNI mode network solution, it can be added to the original network capacity
- Support Pod binding EIP/NAT, no longer rely on the external network access ability of the node, no need to do SNAT, can meet the high concurrency and high bandwidth external network access scenarios such as live broadcast, game, video conference, etc.
- Support Pod binding security group to achieve Pod-level security isolation
- Supports fixed IP based on the Pod name, and the IP remains unchanged after the Pod is rescheduled
- Support CLB pass-through Pod, no longer forwarded by NodePort, improve forwarding performance and have a unified load balancing view
- Will support Blackstone 2.0 physical server (recommended, use smart network card by default, higher network performance)
- Will support fixed EIP based on Pod name to meet Pod fixed external network export
Instructions
Apply for a new generation of network solutions container after the opening of closed beta, when you create a cluster container network TKE mode selection VPC-CNI / Pod independent NIC mode you can:
Introduction to the realization principle
The new-generation solution expands on the original VPC-CNI model. Relying on the elastic network card, the elastic network card bound to the node is configured to the container network namespace through CNI, so that the container can directly use the elastic network card exclusively.
You can follow Tencent Cloud's native official account, and the technical details of the implementation of TKE's new generation network solution will be pushed later .
Use restrictions in the current closed beta phase
- Only some S5 models can use this network mode.
- The number of Pods running on a node is limited to 5 times the number of node cores.
- Only the new cluster is supported, and the existing TKE cluster does not support changing the network plan.
TKE new generation network solution internal test invitation
We sincerely invite you to participate in the internal test of Tencent Cloud's next container independent network card network solution product capability. You can submit an application for the internal test through the following link: https://cloud.tencent.com/apply/p/85p1zs6x777