spdk source Interpretation 1

[Section switched https://www.sdnlab.com/21087.html ]

SPDK Application Programming Framework

SPDK (Storage performance development kit, http://spdk.io ) is sponsored by Intel, used to accelerate the use of NVMe SSD application software as a back-end storage acceleration library. The core of the software library is user mode, asynchronous, polling of the NVMe driven . Compared kernel (such as the Linux Kernel) of NVMe drive, which can greatly reduce the delay NVMe Command (Latency), while improving the IOPS single CPU core , so as to form a cost-effective solution, for example using a vhost solution SPDK It may be applied HCI (Hyper Converged Infrastructure) accelerated virtual machine NVMe I / O.

In order to achieve the above object, merely to provide some function or operation of the source language NVMe user mode driver is not enough. If used incorrectly in some scenarios, not only can not play NVMe user mode driver performance, even lead to a program error. Although the underlying function NVMe some instructions, but in order to play the better the performance of the underlying NVMe, SPDK provides a programming framework (SPDK Application Framework), used to guide software developers based SPDK user mode and user mode driver NVMe block layer (User space Bdev) configured efficient storage applications. Users have two options:

(1) directly SPDK application programming framework implements application logic;
(2) using the idea SPDK programming framework, application programming logic transformation to the user a better fit SPDK NVMe driving states.

Overall, SPDK application framework can be divided into the following sections:

(1) Management of the CPU core and thread;
Efficient communication between (2) the thread;
Processing model (3) I / O and the data path (data path) of the lock-free mechanism.

CPU core and thread the pipe management

SPDK a big aim is to use minimal CPU cores and threads to complete most tasks. To this end, SPDK at initialization program (currently call spdk_app_start function) which limited the use of nuclear binding of the CPU can be configured in a configuration file or a named line, for example, use the command line -c 0x5 refers to the use core0 and core2 to start program. Can be locked by CPU core affinity binding function of CPU usage, and run a thread on each core, the thread is referred SPDK in the Reactor (as shown in Figure 1). Currently SPDK library environment (ENV) still use the default DPDK of EAL library management. In summary, a function performed Reactor Thread (_spdk_reactor_run), comprising a body of the function while (1) {} function-until Reactor's state is changed, for example, by (spdk_app_stop call). For efficient, the above-described cycle will have some mechanism for the corresponding CPU resources (such as sleep). Most of the time such a mechanism can cause CPU usage to 100% of cases, this point and DPDK compare like with like.

In other words, suppose an application programming framework using SPDK use two CPU core, it will start a Reactor thread on each core. In this way, the user how to perform their function? To solve this problem, SPDK provides a mechanism for the Poller , i.e. dispensing user-defined functions. SPDK provided Poller divided into two types:

(1) Timer-based Poller;
(2) Non-timers Poller.

Reactor thread SPDK the corresponding data structure (struct spdk_reactor) corresponding to a list of the maintenance mechanism Poller. For example, a linked list maintenance timer Poller, a linked list maintenance Poller non-timers, and provide Poller registration and destroying function . Reactor of the while loop, it will stop check the status of these Poller, the corresponding call, the user can thus function corresponding call. Because only a single CPU Reactor thread, so no mechanism to lock some of the same Reactor thread to protect the resource. Of course, the thread is located on a different CPU core still need the necessary communications. To solve this problem, SPDK encapsulates the asynchronous message passing between threads (Async Messaging Passing) a.

High pass between threads Letter

SPDK abandon the use of traditional locking way to communicate between threads, since such programs less efficient. In order to make only a single thread execution resources they manage, SPDK provides Event (event call) mechanism. The essence of the mechanism is that each data structure corresponding Reactor (struct spdk_reactor) maintains an Event event ring (rings). This ring is more than a single producer and consumer (MPSC: Multiple producer Single Consumer) model , that is, each Reactor thread can receive messages from any other event Reactor thread (including the current Reactor Thread) for processing. Event ring SPDK currently in default implementation depends on the mechanism DPDK, there should be a linear locking mechanism, but compared to the use of mechanisms for inter-thread synchronization lock is much more efficient.

There is no doubt, while Event ring processing functions are performed Reactor (_spdk_reactor_run) process. Event data structure of each event (struct spdk_event) actually includes a function to be performed, with the corresponding parameters and the core to be executed. Briefly, a Reactor A Reactor B to another communication is actually needed Reactor B performs the function F (X) (X is the corresponding parameter).

Based on the above mechanisms, SPDK to achieve a more efficient inter-thread communication mechanism. Specific examples can refer SPDK NVMe-oF target inside some implementations, the main code is (lib / nvmf) directory.

I / O processing model and lock-free data path of

SPDK main I / O processing model Run-to-completion, refers to run until completed. Mentioned above, when using SPDK application framework, a CPU core has only one thread, the thread can perform many Poller (including non-timer and timer). Run-to-completion aim is to make the best finish a thread to perform all tasks. Obvious, SPDK programming framework to meet this need. If you do not use SPDK application programming framework, you need to pay attention to their own programmer this matter. For example, using SPDK NVMe user mode driver to access a corresponding I / O QPair read and write, provided SPDK asynchronous read and write functions (spdk_nvme_ns_cmd_read), while checking the completion of the function (spdk_nvme_qpair_process_completions). These functions should call a thread to complete, should not cross-thread processing.

SPDK the I / O path also be employed without the lock mechanism. When a plurality of user mode thread operation agree SPDK block device (bdev) when, SPDK provides a conceptual I / O channel (i.e., mapping relationship between a thread and the device) . Different thread should operate the same device has a different I / O channel, each I / O channel uses its own independent resources on the I / O path can avoid competition for resources, thereby removing the lock mechanism.

Often see asked questions

01 SPDK released several versions every year? Release number is kind of how?

A: 4 versions. Release version uses the month of the year plus the way: YY.MM (MM which belongs to the set {1,4,7,11}). Publish annually a total of four versions, respectively, in January, April, July and released in November. SPDK upcoming version 18.04.

02 SPDK open source projects and project DPDK What is the relationship?

A : SPDK first project, code-named WaikikiBeach, full name DPDK For Storage, open since 2015, changed SPDK. SPDK provides a library environment abstraction (located lib / env directory), the main CPU for managing resources used by an application SPDK storage, memory and other PCIe device resources, wherein DPDK is SPDK default environment library. Each time SPDK released a new version will use the latest stable version DPDK released. For example, SPDK 18.04 will use DPDK18.02 version.

What are some 03 SPDK typical use scenario is?

A : For now, SPDK is not a general-purpose storage applications adapted solutions. Because the kernel driver into user mode, resulting in user mode implementation requires a user mode based on the complete software-driven I / O stack. File system is without a doubt one of the important topic, and the kernel file system (eg linux EXT4 and btrfs, etc.) can no longer be used directly. Although SPDK provides a very simple "file system" blobfs / blostore, but does not support the POSIX interface. For this reason, if the application you want to use the file system to migrate directly to the SPDK user mode "File System", need to do some code porting. For example, without using a POSIX interface in favor of asynchronous read and write manner similar to the AIO. SPDK communities have been efforts in that direction, now SPDK used in the following application scenarios better: (1) providing block device interface to back-end storage applications such as iSCSI Target and NVMe-oF Target; (2) the virtual machine the I / O (virtio) acceleration, the main support QEMU under Linux / KVM as scene managed by the hypervisor of the virtual machine, using vhost interactive protocol efficient vhost user mode target shared memory channel (e.g. vhost SCSI / blk / NVMe target), thus speeding up the virtual machine virto SCSI / blk kernel Native NVMe protocols and I / O drivers. The main principle is to reduce the number of (for example, interrupt, and VM_EXIT) VM interruption event, and shortens the host OS of the I / O stack; (3) SPDK accelerate database storage engine. By implementing the abstract class file in RocksDB, SPDK the blobfs / blobstore currently Rocksdb and integrated, for accelerating the engine in use RocksDB NVMe SSD. The essence of the operation is completely bypass kernel file system based SPDK user mode I / O stack. In addition, reference SPDK support for Rocksdb, and can also integrate other database storage engine with SPDK blobfs / blobstore.