DAOS operates through two tightly integrated planes. The data plane handles heavy transport operations, while the control plane is responsible for process orchestration and storage management, simplifying the operation of the data plane.

module interface

The I/O engine supports a module interface that allows server-side code to be loaded on demand. Each module is actually a library, dynamically loaded by the I/O engine via dlopen. The interface between the module and the I/O engine dss_module is defined in a data structure.

Each module shall specify:

module name
daos_module_id module identifier in
bitmask
A module initialization and destruction function

Additionally, modules can optionally be configured with:

Configuration and cleanup functions called after the entire stack is up and running
CART RPC handler
dRPC handler

Threading model integrated with Argobot

The I/O engine is a multi-threaded process that uses Argobot for non-blocking processing.

By default, each Target creates a main xstream and no offload xstreams. The actual number of offload xstream can be configured through daos_engine command line parameters. Additionally, an additional xstream is created to handle incoming metadata requests. Each xstream is bound to a specific CPU core. main xstream receives incoming requests for Target from clients and other servers. A specific ULT (User Level Thread) helps with network and NVMe I/O operations.

Thread-local Storage (TLS)

The private storage allocated by each xstream can dss_tls_get() be accessed through functions.

When registering, each module can specify a module key whose data structure size will be allocated by each xstream in TLS.

dss_module_key_get() The purpose of the function is to return the data structure of the specific registration module key.

Incast Variable Integration

DAOS uses IVs (incast variables) to share values and state between servers under a single IV namespace (organized as a tree). The root node of the tree is called the IV leader, and the server can be a leaf node or a non-leaf node.

Each server maintains its own IV cache. During fetch, if the local cache cannot fulfill the request, it forwards the request to its parent cache until it reaches the root cache (IV leader). For an update operation, the server first updates its local cache, then forwards to its parent cache until it reaches the root cache, and then propagates the change to other servers.

The IV namespace is per-Pool, created during Pool connection, and destroyed during Pool disconnection.

To use IV, each user needs to register himself under the IV namespace to obtain an identifier, and then the user will use this ID to obtain or update his own IV value under the IV namespace.

dRPC server

The I/O engine includes a dRPC server that listens for activity on a given Unix Domain Socket.

For more details on the basics of dRPC and the low-level API in Go and C, see the dRPC Documentation .

The dRPC server periodically polls for incoming client connections and requests. struct drpc_progress_context It can handle multiple client connections simultaneously through an object that manages the struct drpc object listening on the Socket as well as any active client connections.

The server runs cyclically in xstream 0's own ULT (User Level Thread). The dRPC Socket has been set to be non-blocking and use polling with no timeout. This allows the server to run in the ULT instead of its own xstream, where traffic is expected to be relatively low.

dRPC process

drpc_progress Represents one iteration of the dRPC server loop. Its workflow is as follows:

Timeout polls concurrently on the listening Socket and any open client connections.
If you see any activity on client connections:
1. If data has been entered: call to drpc_recv process the entered data.
2. If the client has disconnected or the connection is destroyed: release struct drpc the object and remove it drpc_progress_context from .
If any activity is found on the listener:
1. If a new connection comes in: call drpc_accept and add the new struct drpc object to drpc_progress_context the list of client connections in .
2. If there was an error: will be -DER_MISC returned to the caller. The error is logged in the I/O engine, but does not break the dRPC server loop. Getting an error on a listener is unexpected.
If no activity is seen, it will be -DER_TIMEDOUT returned to the caller. This is purely for debugging purposes, in practice the I/O engine ignores this error code because lack of activity is not actually an error.

dRPC handler registration

A single DAOS module can process dRPC messages by registering one or more dRPC module ID handlers.

Registering a handler is simple. In dss_server_module the field of sm_drpc_handlers , statically allocate an array of struct dss_drpc_handler with the last item of zero to indicate the end of the list. Setting the field to NULL indicates that there are no handlers to register. When the I/O engine loads the DAOS module, it automatically registers all dRPC handlers.

Notice:

The dRPC module ID is different from the DAOS module ID.
This is because a given DAOS module may need to register multiple dRPC module IDs, depending on the functionality covered by the DAOS module.
dRPC module IDs must be system-wide unique and are listed in a central header file `src/include/daos/drpc_modules.h.

dRPC servers use functions drpc_hdlr_process_msg to process incoming messages. This function checks the module ID of the incoming message, searching for a handler.

If a handler is found, it is executed, and returns Drpc_Response.
If not found, it will generate its own Drpc_Response, indicating that the module ID is not registered.

Related Information

GitHub: https://github.com/storagezhang

Email: [email protected]

DAOS: https://github.com/daos-stack/daos

This article is translated from https://github.com/daos-stack/daos/blob/master/src/engine/README.md

DAOS Distributed Asynchronous Object Storage｜Data Plane