NIC multi-queue, as the name suggests, means that the traditional NIC has multiple DMA queues, and the NIC has an allocation mechanism based on multiple DMA queues. Multi-queue network cards have become the mainstream of current high-speed network cards.
In the Linux kernel, RPS (Receive Packet Steering) provides such a mechanism at the receiving end. RPS mainly balances the load of soft interrupts to each core of the CPU. The network card driver generates a hash identifier for each flow. DIP, the purpose of the four-layer port DPORT) to calculate, and then the interrupt processing place is assigned to the corresponding core according to the hash identifier, so that the multi-core capability can be fully utilized.
DPDK multi-queue support
The DPDK Packet I/O mechanism has an inherent multi-queue support function. According to different platforms or requirements, the number of queues to be used can be selected, and the queues can be conveniently used to specify queues to send or receive messages. Due to this feature, it is easy to achieve affinity among CPU cores, caches, and network card queues, thereby achieving good performance. From the typical application l3fwd of DPDK, it can be seen that the program running on a certain core receives from the specified queue and sends it to the specified queue, which can achieve a high cache hit rate and high efficiency.
In addition to conveniently performing packet sending and receiving operations on the specified queue, the queue management mechanism of DPDK can also avoid unnecessary waiting caused by the use of spin locks by multiple sending and receiving processes in multi-core processors.
Taking the run to completion model as an example, we can understand how DPDK uses network card multi-queue technology to improve performance from the relationship between cores, memory and network card queues.
- Assign a receiving queue of the network card to a certain core, and all messages received from the queue should be processed on the designated core.
- Allocate a memory pool from the local storage corresponding to the core, and both the received message and the corresponding message descriptor are located in the memory pool.
- A separate send queue is allocated for each core, and the send message and the corresponding message descriptor are located in the local memory pool corresponding to the core and the send queue.
It can be seen that different cores operate different queues, thus avoiding the lock overhead caused by multiple threads accessing a queue at the same time. However, if the number of logical cores is greater than the number of transmit queues contained on each interface, then a mechanism is required to assign queues to these cores. Regardless of the strategy adopted, locks need to be introduced to protect the data in these queues.
How does the network card distribute the packets in the network to different queues? Commonly used methods include RSS proposed by Microsoft and Flow Director technology proposed by Intel. The former hopes to evenly distribute packets to multiple queues according to the hash value. The latter is based on the exact match of the lookup, distributing the package to the specified queue. In addition, the network card can also provide support for QoS by assigning queues according to priority.
flow classification
Advanced network card devices (such as Intel XL710) can analyze the packet type, and the packet type will be carried in the receiving descriptor, and the application program can quickly determine which type of packet the packet is based on the descriptor. The Mbuf structure of DPDK contains corresponding fields to indicate the type of packet analyzed by the network card.
RSS (Receive-Side Scaling, receiver scaling)
RSS is to calculate the hash value through the hash function according to the keyword, and then determine the queue by the hash value.
How are keywords determined?
Hash function generally chooses Microsoft Toeplitz algorithm (Microsoft Toeplitz Based Hash) or symmetric hash.
Flow Director
Flow Director technology is a technology proposed by Intel to assign packets to a specific queue based on exact matching of fields: a Flow Director table is stored on the network card. The size of the table is limited by hardware resources, and it records the fields that need to be matched. keywords and actions after matching; the driver is responsible for operating this table, including initialization, adding entries, and deleting entries; after the NIC receives data packets from the line, it checks the table of Flow Director according to the keywords. The action processing in the entry can be allocation queue, discarding, etc.
Compared with the load sharing function of RSS, it puts more emphasis on specificity. For example, a user can reserve a certain queue for certain specific TCP conversations (S-IP+D-IP+S-Port+D-Port), and then the application processing these TCP conversations can only care about this specific queue , thus eliminating the overhead of CPU filtering packets and improving the cache hit rate.
service quality
Multiple queues are applied to quality of service (QoS) traffic categories: assigning sending queues to different traffic categories allows network cards to schedule on the sending side; assigning packet receiving queues to different traffic categories enables flow-based rate limiting .
stream filtering
Which data packets from outside are local and can be received, and which ones cannot be received? The data packets that can be received will be sent by the network card to the host or the built-in management controller of the network card. The filtering is mainly focused on the Layer 2 function of the Ethernet, including VLAN and MAC filtering.
application
For the Intel® XL710 NIC, PF uses the i40e Linux Kernel driver, and VF uses the DPDK i40e PMD driver. Using the Linux Ethtool tool, you can complete the configuration operation cloud filter, and directly allocate a large number of data packets to the VF queue, and hand them over to the virtual machine application running on the VF for direct processing.
echo 1 > /sys/bus/pci/devices/0000:02:00.0/sriov_numvfs
modprobe pci-stub
echo "8086 154c" > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:02:02.0 > /sys/bus/pci/devices/0000:2:02.0/driver/unbind
echo 0000:02:02.0 > /sys/bus/pci/drivers/pci-stub/bind
qemu-system-x86_64 -name vm0 -enable-kvm -cpu host -m 2048 -smp 4 -drive file=dpdk-vm0.img -vnc :4 -device pci-assign,host=02:02.0
ethtool -N ethx flow-type ip4 dst-ip 2.2.2.2 user-def 0xffffffff00000000 action 2 loc 1
Original link: https://mp.weixin.qq.com/s/A1rtrDNr9pFE5zR6BrnRnA
Learn more dpdk videos
DPDK learning materials, teaching videos and learning roadmap: https://space.bilibili.com/1600631218
Dpdk/network protocol stack/vpp/OvS/DDos/NFV/virtualization/high performance expert learning address: https://ke.qq.com/course/5066203?flowToken=1043799
DPDK develops learning materials, teaching videos and learning roadmap sharing. If necessary, you can add learning exchanges by yourself q Junyang 909332607 Remarks (XMG) Get