Triton Tutorial - Storage Proxy

insert image description here

Triton Series Tutorials:

The repository agent extends Triton with new functionality that runs when a model is loaded or unloaded. You can introduce your own code to perform authentication, decryption, conversion, or similar operations when loading the model.

Beta: The repository proxy API is beta quality and subject to non-backwards compatible changes for one or more releases.

The repository proxy communicates with Triton using the repository proxy API. The checksum_repository_agent GitHub repository provides an example repository agent for validating file checksums before loading a model.

Use repository proxy

A model can use one or more repository agents by specifying them in the ModelRepositoryAgents section of the model configuration. Each repository proxy can have parameters specific to that proxy, which are specified in the model configuration to control the behavior of the proxy. To learn about the parameters available for a given broker, see that broker's documentation.

Multiple delegates can be specified for the same model and they will be called sequentially when loading or unloading the model. The following sample model configuration content shows how to specify two proxy “agent0”sums “agent1”so that they are called in that order with the given parameters.

model_repository_agents
{
    
    
  agents [
    {
    
    
      name: "agent0",
      parameters [
        {
    
    
          key: "key0",
          value: "value0"
        },
        {
    
    
          key: "key1",
          value: "value1"
        }
      ]
    },
    {
    
    
      name: "agent1",
      parameters [
        {
    
    
          key: "keyx",
          value: "valuex"
        }
      ]
    }
  ]
}

Implement a repository proxy

A repository proxy must be implemented as a shared library, and the name of the shared library must be libtritonrepoagent_<repo-agent-name>.so. Shared libraries should hide all symbols except those required by the repository proxy API. See the CMakeList.txt of the checksum example for an example of how to use ldscript to expose only the necessary symbols .

Shared libraries will be dynamically loaded by Triton when needed. For a repository proxy named A, the shared library must be installed as <repository_agent_directory>/A/libtritonrepoagent_A.so. By default <repository_agent_directory>yes /opt/tritonserver/repoagents. –repoagent-directoryFlags can be used to override defaults.

Your repository proxy must implement the repository proxy API, as described tritonrepoagent.hin .

Triton follows these steps when loading a model:

Load the model's configuration file ( config.pbtxt) and extract ModelRepositoryAgentsthe settings . The repository proxy config.pbtxtsettings config.pbtxt in the initial file are used throughout the loading process, even if the repository proxy modifies the file.
For each repository proxy specified:
- Initializes the corresponding repository proxy, loading shared libraries if necessary. Model loading will fail if the shared library is not available or fails to initialize.
- Use TRITONREPOAGENT_ACTION_LOADthe action to call the repository proxy's TRITONREPOAGENT_ModelActionfunction. As input, the agent can access the model's repository as a cloud storage location or a local file system location.
- A repository proxy can return success to indicate that no changes were made to the repository, it can return failure to indicate that model loading should fail, or it can create a new repository for the model (for example, by decrypting the input repository) and return success to indicate that it should Use a new repository.
- If the agent returns success, Triton will continue to the next agent. If an agent returns a failure, Triton will skip calling any other agents.
If all proxies return success, Triton tries to load the model using the final model repository.
For each repository proxy TRITONREPOAGENT_ACTION_LOADinvoked , in reverse order:
- Triton calls the repository proxy's TRITONREPOAGENT_ModelActionfunction , TRITONREPOAGENT_ACTION_LOAD_COMPLETEthe action if the model loads successfully, and the model load fails TRITONREPOAGENT_ACTION_LOAD_FAIL.

Triton follows these steps when unloading a model:

Triton uses the repository proxy settings from the initial config.pbtxtfile , even if one or more proxies modified its contents during loading.
The sequence is the same for each repository proxy TRITONREPOAGENT_ACTION_LOADinvoked :
- Triton uses the action TRITONREPOAGENT_ACTION_UNLOADto invoke the repository proxy's TRITONREPOAGENT_ModelActionfunction.
Triton unloads the model.
For each repository proxy invoked with TRITONREPOAGENT_ACTION_UNLOAD , in reverse order:
- Triton uses the action TRITONREPOAGENT_ACTION_UNLOAD_COMPLETEto invoke the repository proxy's TRITONREPOAGENT_ModelActionfunction.