A Preliminary Study of DPDK (Data Plane Development Kit)

What is DPDK?

DPDK (Data Plane Development Kit) is an open source data plane development kit designed to accelerate the speed and performance of data packet processing. It provides users with an optimized set of APIs and libraries that allow them to achieve high-performance packet processing on general-purpose processors, often used in network function virtualization (NFV) and high-performance network applications. DPDK was originally developed by Intel and has now become a project jointly maintained by the community.

Advantages of DPDK

Compared with the traditional Linux network stack, the advantages of DPDK are mainly reflected in the following aspects:

High performance: DPDK is highly optimized for packet processing, making full use of the advantages of modern multi-core processors to greatly improve packet processing performance. This makes DPDK ideal for handling high traffic and packet densities.
Low latency: By bypassing the traditional kernel networking stack, DPDK can significantly reduce packet processing latency. This is especially important for web applications that require real-time response.
Lightweight: The design of DPDK is simple and efficient, avoiding the complex abstraction and protocol processing in the traditional kernel stack. This allows DPDK to perform well in resource-constrained environments.
Scalability: DPDK allows users to flexibly configure and manage threads and queues for packet processing, so as to easily cope with different network loads and processing requirements.
Hardware acceleration: DPDK can be tightly integrated with specific hardware platforms to make full use of hardware acceleration functions to further improve performance.

DPDK usage scenarios

DPDK is mainly used for high-performance network applications and data packet processing, especially for the following scenarios:

Virtualization environment: In a virtualization environment, DPDK can be used to implement high-performance virtual machine network functions (VNFs), providing virtual machines with performance close to physical networks.
Network function virtualization (NFV): NFV involves a large number of network function chains, such as firewalls, load balancing, etc. These functions usually require high-performance packet processing capabilities, and DPDK just meets this demand.
SDN controller: DPDK can be used to accelerate the traffic processing of the SDN controller and improve the performance and response speed of the controller.
Network monitoring and analysis: In network monitoring and analysis applications, it is usually necessary to capture and process a large number of data packets, and DPDK can help achieve efficient data packet processing and analysis.
High-performance server applications: For server applications that require high concurrency and low latency, DPDK can accelerate the processing of data packets and improve server performance.

The basic concept of DPDK

Before learning DPDK in depth, there are several basic concepts to understand:

Poll Mode Drivers (PMDs): PMD is the interface between DPDK and network devices, responsible for sending and receiving data packets. DPDK provides a series of pre-optimized PMDs to adapt to different types of network devices.
Memory Pools: A memory pool is a pre-allocated memory area used to store memory buffers for data packets. DPDK uses memory pools to avoid frequent memory allocation and deallocation and improve performance.
Queues: DPDK uses multiple queues to achieve parallel processing of packets. Queues can be divided into input queues and output queues, which are used to transfer data packets between different processing cores.
Polling mode: DPDK uses polling mode to process packets, rather than interrupt driven. This means that the application needs to periodically poll the device to check if new packets have arrived.
Master-slave mode: DPDK usually runs in master-slave mode, one of the cores is responsible for initialization and configuration as the master core, and the other cores are responsible for packet processing as slave cores.

A Preliminary Study on DPDK (Data Plane Development Kit) - Part 2

Install and configure DPDK on Linux system

1. Confirm system requirements

Before starting to install DPDK, make sure the following system requirements are met:

Linux distribution: It is recommended to use mainstream Linux distributions such as Ubuntu, CentOS, or Fedora.
Kernel version: DPDK requires a newer Linux kernel, 4.0 and above is generally recommended.
Hardware requirements: Make sure the system supports the hardware features required by DPDK, such as CPU supports VT-x (Intel) or AMD-V (AMD), etc.

2. Install dependent libraries

Before installing DPDK, you need to install some necessary dependent libraries. You can use a package manager (such as apt, yum, etc.) to install these dependent libraries.

For Ubuntu/Debian systems:

sudo apt update
sudo apt install build-essential linux-headers-$(uname -r) numactl

For CentOS/Fedora system:

sudo yum groupinstall "Development Tools"
sudo yum install numactl-devel

3. Download and compile DPDK

First, download the latest version of the DPDK source code package from the DPDK official website (https://www.dpdk.org/), and decompress it.
Enter the decompressed DPDK directory, and configure the DPDK compilation options. Usually we can use x86_64-native-linuxapp-gcctarget to compile DPDK library for x86_64 architecture.

make config T=x86_64-native-linuxapp-gcc

Next, compile DPDK.

make

4. Set environment variables

In order to link and use the DPDK library correctly when using the DPDK application, we need to set some environment variables.

//临时环境变量，只对当终端界面生效
export RTE_SDK=/path/to/dpdk   # 将/path/to/dpdk替换为实际的DPDK源代码路径
export RTE_TARGET=x86_64-native-linuxapp-gcc

It is recommended to add the above two lines to your ~/.bashrcfile so that these environment variables are automatically set every time you log in.

5. Bind the network card to the DPDK driver

Before using a network card in a DPDK application, the network card needs to be bound to DPDK's Poll Mode Driver (PMD). dpdk-devbind.pyThis can be done using scripts provided by DPDK .

sudo $RTE_SDK/usertools/dpdk-devbind.py --status   # 查看可用的网卡和其绑定情况
sudo $RTE_SDK/usertools/dpdk-devbind.py -b igb_uio 0000:01:00.0   # 将网卡绑定到igb_uio驱动

6. Write a simple DPDK application

Here is an example of a simple DPDK application that receives packets from a network card and echoes them back:

#include <stdio.h>
#include <rte_eal.h>
#include <rte_ethdev.h>

#define RX_RING_SIZE 128
#define NUM_MBUFS 8191
#define MBUF_CACHE_SIZE 250
#define BURST_SIZE 32

static const struct rte_eth_conf port_conf_default = {
    
    
    .rxmode = {
    
    
        .max_rx_pkt_len = ETHER_MAX_LEN,
    },
};

int main(int argc, char *argv[]) {
    
    
    int ret;
    unsigned nb_ports;
    uint16_t portid;
    struct rte_mempool *mbuf_pool;
    struct rte_eth_dev_info dev_info;
    struct rte_eth_txconf txconf;

    // 初始化EAL
    ret = rte_eal_init(argc, argv);
    if (ret < 0)
        rte_exit(EXIT_FAILURE, "Error with EAL initialization\n");

    // 获取网卡数量
    nb_ports = rte_eth_dev_count_avail();
    printf("Detected %u port(s)\n", nb_ports);

    // 创建内存池
    mbuf_pool = rte_pktmbuf_pool_create("mbuf_pool", NUM_MBUFS, MBUF_CACHE_SIZE, 0, RTE_MBUF_DEFAULT_BUF_SIZE, rte_socket_id());
    if (mbuf_pool == NULL)
        rte_exit(EXIT_FAILURE, "Cannot create mbuf pool\n");

    // 配置和初始化每个网卡
    for (portid = 0; portid < nb_ports; portid++) {
    
    
        // 配置网卡
        ret = rte_eth_dev_configure(portid, 1, 1, &port_conf_default);
        if (ret < 0)
            rte_exit(EXIT_FAILURE, "Cannot configure device: err=%d, port=%u\n", ret, portid);

        // 获取网卡设备信息
        rte_eth_dev_info_get(portid, &dev_info);

        // 设置网卡的接收队列
        ret = rte_eth_rx_queue_setup(portid, 0, RX_RING_SIZE, rte_eth_dev_socket_id(portid), NULL, mbuf_pool);
        if (ret < 0)
            rte_exit(EXIT_FAILURE, "RX queue setup failed: err=%d, port=%u\n", ret, portid);

        // 设置网卡的发送队列
        ret = rte_eth_tx_queue_setup(portid, 0, RX_RING_SIZE, rte_eth_dev_socket_id(portid), NULL);
        if (ret < 0)
            rte_exit(EXIT_FAILURE, "TX queue setup failed: err=%d, port=%u\n", ret, portid);

        // 启动网卡
        ret = rte_eth_dev_start(portid);
        if (ret < 0)
            rte_exit(EXIT_FAILURE, "Error starting device: err=%d, port=%u\n", ret, portid);
    }

    printf("Initialization completed.\n");

    // 开始数据包收发处理
    for (;;) {
    
    
        struct rte_mbuf *bufs[BURST_SIZE];
        uint16_t nb_rx;

        // 从网卡接收数据包
        for (portid = 0; portid < nb_ports; portid++) {
    
    
            nb_rx = rte_eth_rx_burst(portid, 0, bufs, BURST_SIZE);
            if (nb_rx > 0) {
    
    
                // 回显收到的数据包
                rte_eth_tx_burst(portid, 0, bufs, nb_rx);
            }
        }
    }

    return 0;
}

Compile the above code and run the application on the system with the NIC bound, it will start listening for packets on the NIC and echoing them. Note that this is just a simple example, real DPDK applications may require more complex logic and processing.

Advanced features of DPDK

1. Multi-queue processing

DPDK supports multi-queue processing, allowing packets to be processed in parallel on different processing cores, further improving performance. By using different queues, higher concurrency and load balancing can be achieved. To achieve multi-queue processing, you need to configure multiple queues of the network card and use different cores to process the packets of each queue.

// 配置网卡的多个接收队列
for (portid = 0; portid < nb_ports; portid++) {
    
    
    for (int queue_id = 0; queue_id < num_queues; queue_id++) {
    
    
        ret = rte_eth_rx_queue_setup(portid, queue_id, RX_RING_SIZE, rte_eth_dev_socket_id(portid), NULL, mbuf_pool);
        if (ret < 0)
            rte_exit(EXIT_FAILURE, "RX queue setup failed: err=%d, port=%u, queue=%d\n", ret, portid, queue_id);
    }
}

// 在不同的核心上处理不同队列的数据包
#pragma omp parallel num_threads(num_queues)
{
    
    
    int queue_id = omp_get_thread_num();
    for (;;) {
    
    
        struct rte_mbuf *bufs[BURST_SIZE];
        uint16_t nb_rx = rte_eth_rx_burst(portid, queue_id, bufs, BURST_SIZE);
        if (nb_rx > 0) {
    
    
            // 处理接收到的数据包
            process_packets(bufs, nb_rx);

            // 发送数据包
            rte_eth_tx_burst(portid, 0, bufs, nb_rx);
        }
    }
}

2. Hardware Acceleration

DPDK supports tight integration with specific hardware platforms to take full advantage of hardware acceleration features. These hardware functions include RSS (receiving side diffusion), TSO (large packet transmission), LRO (large packet reception) and so on of the network card. The performance and efficiency of packet processing can be further improved by enabling hardware acceleration.

// 配置网卡的RSS功能
struct rte_eth_conf port_conf;
port_conf.rxmode = {
    
    
    .mq_mode = ETH_MQ_RX_RSS,
    .max_rx_pkt_len = ETHER_MAX_LEN,
};
port_conf.rx_adv_conf.rss_conf = {
    
    
    .rss_key = rss_hash_key,
    .rss_key_len = RSS_HASH_KEY_LEN,
    .rss_hf = ETH_RSS_IP | ETH_RSS_TCP | ETH_RSS_UDP,
};

ret = rte_eth_dev_configure(portid, num_queues, num_queues, &port_conf);
if (ret < 0)
    rte_exit(EXIT_FAILURE, "Cannot configure device: err=%d, port=%u\n", ret, portid);

3. NUMA support

DPDK provides support for NUMA (Non-Uniform Storage Architecture) systems, enabling DPDK applications to better utilize local memory and CPU cores in systems under the NUMA architecture, reduce data access delays, and improve performance. In order to enable NUMA support, appropriate options need to be set when compiling DPDK.

make config T=x86_64-native-linuxapp-gcc CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES=y
make

4. More advanced features

DPDK also provides many other advanced features, such as dynamic adjustment of queue size, buffer pool size and MTU (Maximum Transmission Unit), etc., to further optimize application performance. In practical applications, these features can be selected and configured according to specific requirements.

5. Performance Optimization Tips

In order to give full play to the performance advantages of DPDK, some optimization techniques can also be adopted:

Use large pages: Use large pages (hugepage) to reduce the number of accesses to the TLB (translation back buffer) and improve memory access efficiency.
Pre-allocate memory: When starting the application, try to pre-allocate all the required memory to avoid frequent memory allocation during runtime.
Avoid shared data structures: In a multi-core environment, try to avoid multiple cores accessing shared data structures to reduce lock competition.
Use lock-free data structures: Consider using lock-free data structures to improve the performance of concurrent access by multiple threads.
Disable interrupts: In DPDK applications, interrupts are usually not needed, and interrupts can be disabled at startup to reduce the overhead of interrupt processing.

The above are just some simple performance optimization suggestions, and the actual optimization method depends on the characteristics and requirements of the application.