Original link: https://zhuanlan.zhihu.com/p/583436603
1 High performance SR-IOV network (SR-IOV)
1.1 Introduction to SR-IOV Principle
SR-IOV is the abbreviation of Single Root I/O Virtualization. SR-IOV technology is a hardware-based virtualization solution that improves performance and scalability. The SR-IOV standard allows efficient sharing of PCIe (Peripheral Component Interconnect Express) devices between virtual machines, and it is implemented in hardware to achieve I/O performance comparable to native performance . The SR-IOV specification defines a new standard by which new devices can be created to allow virtual machines to connect directly to I/O devices.
The SR-IOV specification is defined and maintained by the PCI-SIG at http://www.pcisig.com .
A single I/O resource can be shared by many virtual machines . Shared devices provide dedicated resources and also use shared generic resources. This way, each virtual machine has access to unique resources. Therefore, a PCIe device with SR-IOV enabled and proper hardware and OS support (such as an Ethernet port) can appear as multiple separate physical devices, each with its own PCIe configuration space.
SR-IOV technology
The two new feature types in SR-IOV are:
- Physical Function (PF)
A PCI function to support SR-IOV functionality, as defined in the SR-IOV specification. The PF contains the SR-IOV function structure for managing SR-IOV functions. PF is a full-featured PCIe function that can be discovered, managed, and handled like any other PCIe device. PF has fully configured resources and can be used to configure or control PCIe devices.
- Virtual Function (VF)
A function associated with a physical function. A VF is a lightweight PCIe function that can share one or more physical resources with a physical function and with other VFs associated with the same physical function. A VF is only allowed to have configuration resources for its own behavior.
Each SR-IOV device can have one Physical Function (PF), and each PF can have up to 64,000 Virtual Functions (VF) associated with it. PFs can create VFs through registers designed with attributes dedicated to this purpose.
Once SR-IOV is enabled in a PF, the PCI configuration space of each VF can be accessed through the PF's bus, device, and function numbers (routing ID). Each VF has a PCI memory space that maps its register set. The VF device driver manipulates the register set to enable its functionality and appear as a real PCI device. Once a VF is created, it can be assigned directly to an IO guest domain or to individual applications such as Oracle Solaris Zones on bare metal platforms. This feature enables virtual functions to share physical devices and perform I/O without CPU and hypervisor software overhead.
1.2 Advantages of SR-IOV
The SR-IOV standard allows efficient sharing of PCIe devices between IO guest domains. An SR-IOV device can have hundreds of Virtual Functions (VFs) associated with a Physical Function (PF). VF creation can be dynamically controlled by PF through registers designed to enable SR-IOV functionality. By default, SR-IOV functionality is disabled and PF acts as a legacy PCIe device.
Devices with SR-IOV capabilities can take advantage of the following advantages:
-
High Performance - Access hardware directly from the virtual machine environment.
-
Cost reductions - capital and operational savings include:
- energy saving
- Reduced number of adapters
- Simplified wiring
- Reduced switch ports
1.3 SR-IOV limitations
In the SR-IOV passthrough scenario, a virtual machine (VM) can obtain network performance comparable to that on a bare metal host. However, there are still two limitations:
(1) After the SR-IOV VF passthrough to the VM, the migration of the VM will be limited. The main reason is that the passthrough I/O of SR-IOV uses Intel CPU VT-d (Virtualization Technology for Directed I/O) or AMD With the IOMMU (I/O Memory Management Unit) technology, when the VF network card on the VM is initialized, the mapping table from the Guest virtual address to the Host physical address is established, so this "stateful" mapping table is used during live migration. will be lost.
(2) Since SR-IOV VF passthrough to VM, and SR-IOV PF is directly connected to TOR, in this deployment environment, the external network of virtual machine (VM) needs to be customized, such as automatic activation like OVS-DPDK network, it is necessary to add TOR to the management scope of the SDN controller, and the SDN controller will manage and control it in a unified manner. This usually makes network operations very complicated.
In response to the second problem above, Mellanox first proposed to support OVS Fastpath hardware offloading on its smart network card, combined with SR-IOV VF passthrough to VM, providing network capabilities close to line-speed forwarding, and solving the virtual machine (VM) tenant network. The problem of automatic orchestration and activation.
After OVS Fastpath is uninstalled, when OVS forwards the message, the first packet of the data flow is still forwarded by software. During the forwarding process, the Fastpath forwarding flow table is generated and configured on the hardware network card. The subsequent packets of this data flow are directly forwarded to the Virtual Machines (VMs). Since the early Mellanox smart network card has not yet integrated a general-purpose CPU core, the control plane of OVS still runs on the physical host.
1.4 SR-IOV Internal Architecture
The above figure is an example to explain the keywords one by one:
- PF is a PCI function supported by the physical network card. PF can expand several VFs.
- VF is a "network card" virtualized by a physical network card that supports SRIOV or a virtual instance. It will appear in the form of an independent network card. Each VF has its own exclusive PCI configuration area and may communicate with Other VFs share the same physical resource (share the same physical network port)
- The PF miniport driver, that is, the PF driver, works in the parent area of the Hyper-V virtualization platform and is loaded first before the VF
- The VF miniport driver, that is, the VF driver works in the sub-area of the Hyper-V virtualization platform, that is, the guestOS; it should be noted that the VF and PF are isolated, and any results driven by the VF or executed will not affect the Other VF or PF
- The Network Interface Card is the physical network card. After SRIOV is enabled, several vports will be generated. What the physical NIC needs to do is to forward the traffic between the physical port and the vport.
- The physical port, as the name implies, is the physical network port. In the SRIOV scenario, the physical port acts as an external network medium
- VPort is an abstracted interface, similar to physical network ports, which are mapped to each VF or PF for parentOS or guestOS to use
From the description of the above architecture, it can be seen that after SRIOV is enabled, the physical NIC will exchange data with the virtual machine (VF driver) through VF, and vice versa. In this way, the intermediate virtualization stack (ie, the VMM layer) can be skipped to achieve performance close to that of a pure physical environment; this is where the greatest value of SRIOV lies