1. The overall framework of MMDepoly
Model deployment refers to deploying the algorithm model generated by machine learning training to various cloud, edge, and end devices, and making it run efficiently, so that the algorithm model can be actually applied to various tasks in real life, so that Realize the intelligent transformation of AI+.
Currently, we have supported 5 algorithm libraries and 5 back-end reasoning engines, covering a variety of application scenarios:
Algorithm library supported by MMDeploy:
• Detection (MMDetection)
• Segmentation (MMSegmentation)
• Classification (MMClassification)
• Editing (MMEditing)
• Text Recognition (MMOCR)
Backend reasoning engines supported by MMDeploy:
• ONNX Runtime
• TensorRT
• OpenPPL.NN
• ncnn
• OpenVINO
From the perspective of specific module composition, MMDeploy includes two core elements: Model Converter (Model Converter) and Application Development Kit (SDK).
2.1 Model Converter
Model Converter ( Model Converter ) is responsible for converting the PyTorch model of each algorithm library into an inference backend model, and further packaging it as an SDK model.
模型转换器的具体步骤为：
1）把 PyTorch 转换成 ONNX 模型；
2）对 ONNX 模型进行优化；
3）把 ONNX 模型转换成后端推理引擎支持的模型格式;
4）（可选）把模型转换中的 meta 信息和后端模型打包成 SDK 模型。
Compatibility is the most difficult bottleneck in traditional deployment pipelines. In response to these problems, MMDeploy added three major functions: module rewriting, model chunking and custom operators in the model converter
Module Rewriting - Efficient Code Replacement
Aiming at the problem that some Python codes cannot be directly converted into ONNX, MMDeploy uses the rewriting mechanism to implement code replacement at three granularities, such as functions, modules, and symbol tables, effectively adapting to ONNX.
Model partitioning - precise removal of redundancy
To solve the problem that the logic of some models is too complex and cannot be supported in the backend, MMDeploy uses the model block mechanism, which can accurately cut off the difficult-to-convert parts of the model like a scalpel, divide the original model into multiple sub-models, and convert them separately . These removed logics will be implemented in the SDK.
Custom Operators - Extend Engine Capabilities
OpenMMLab implements some new operators which are not supported in ONNX or the backend. In response to this problem, MMDeploy implements custom operators on multiple backends, expanding the expressive capabilities of the inference engine.
2.2 Application Development Kit SDK
The SDK provides a set of C APIs for each vision task. At present, interfaces for several tasks such as classification, detection, segmentation, super-resolution, text detection, and text recognition have been opened. The SDK fully considers the usability and friendliness of the interface. Each set of interfaces consists only of functions such as 'create handle', 'application handle', 'destroy data' and 'destroy handle'. Simple usage and easy integration.
2. Prerequisites
For end-to-end model deployment, MMDeploy requires Python 3.6+ and PyTorch 1.8+.
conda create --name mmdeploy python=3.8 -y
conda activate mmdeploy
conda install pytorch=={
pytorch_version} torchvision=={
torchvision_version} cudatoolkit={
cudatoolkit_version} -c pytorch -c conda-forge
pip3 install torch==1.8.2+cu102 torchvision==0.9.2+cu102 torchaudio===0.8.2 -f https://download.pytorch.org/whl/lts/1.8/torch_lts.html -i http://mirrors.aliyun.com/pypi/simple/ --trusted-host mirrors.aliyun.com
# CUDA 11.6
conda install pytorch==1.12.0 torchvision==0.13.0 torchaudio==0.12.0 cudatoolkit=11.6 -c pytorch -c conda-forge
3. Installation
Step 0. Install MMCV
pip install -U openmim
mim install mmengine
mim install "mmcv>=2.0.0rc2" -i http://mirrors.aliyun.com/pypi/simple/ --trusted-host mirrors.aliyun.com
Step 1. Install MMDeploy and inference engine
We recommend using MMDeploy precompiled packages as our best practice. Currently supports model converter and sdk inference pypi package, add link description here to provide sdk c/cpp library. You can download them according to your target platform and device.
The supported platform and device matrix is as follows:
NOTE: If the MMDeploy pre-built package does not match your target platform or device, please build MMDeploy from source.
1. Taking the latest precompiled package as an example, the installation method is as follows:
Linux-x86_64
# 安装 ONNX Runtime, ONNX, OpenCV
pip install onnxruntime onnx opencv-python
pip install onnxruntime==1.8.1 -i https://pypi.tuna.tsinghua.edu.cn/simple
# 1. install MMDeploy model converter
pip install mmdeploy==1.1.0
# 2. install MMDeploy sdk inference
# you can install one to install according whether you need gpu inference
# 2.1 support onnxruntime
pip install mmdeploy-runtime==1.1.0
# 2.2 support onnxruntime-gpu, tensorrt
pip install mmdeploy-runtime-gpu==1.1.0
# 3. install inference engine
# 3.1 install TensorRT
# !!! If you want to convert a tensorrt model or inference with tensorrt,
# download TensorRT-8.2.3.0 CUDA 11.x tar package from NVIDIA, and extract it to the current directory
https://developer.nvidia.com/nvidia-tensorrt-download
https://developer.nvidia.com/nvidia-tensorrt-8x-download
pip install TensorRT-8.2.3.0/python/tensorrt-8.2.3.0-cp38-none-linux_x86_64.whl
pip install pycuda
export TENSORRT_DIR=$(pwd)/TensorRT-8.2.3.0
export LD_LIBRARY_PATH=${TENSORRT_DIR}/lib:$LD_LIBRARY_PATH
# !!! Moreover, download cuDNN 8.2.1 CUDA 11.x tar package from NVIDIA, and extract it to the current directory
export CUDNN_DIR=$(pwd)/cuda
export LD_LIBRARY_PATH=$CUDNN_DIR/lib64:$LD_LIBRARY_PATH
# 3.2 install ONNX Runtime
# you can install one to install according whether you need gpu inference
# 3.2.1 onnxruntime
pip install onnxruntime==1.8.1 -i https://pypi.tuna.tsinghua.edu.cn/simple
wget https://github.com/microsoft/onnxruntime/releases/download/v1.8.1/onnxruntime-linux-x64-1.8.1.tgz
tar -zxvf onnxruntime-linux-x64-1.8.1.tgz
export ONNXRUNTIME_DIR=$(pwd)/onnxruntime-linux-x64-1.8.1
export LD_LIBRARY_PATH=$ONNXRUNTIME_DIR/lib:$LD_LIBRARY_PATH
# 3.2.2 onnxruntime-gpu
pip install onnxruntime-gpu==1.8.1
wget https://github.com/microsoft/onnxruntime/releases/download/v1.8.1/onnxruntime-linux-x64-gpu-1.8.1.tgz
tar -zxvf onnxruntime-linux-x64-gpu-1.8.1.tgz
export ONNXRUNTIME_DIR=$(pwd)/onnxruntime-linux-x64-gpu-1.8.1
export LD_LIBRARY_PATH=$ONNXRUNTIME_DIR/lib:$LD_LIBRARY_PATH
2. Build from source code
Teach you how to use MMDeploy on ubuntu
Download
git clone -b main [email protected]:open-mmlab/mmdeploy.git --recursive
Note:
If fetching submodules fails, you can fetch submodules manually by following these instructions:
cd mmdeploy
git clone [email protected]:NVIDIA/cub.git third_party/cub
cd third_party/cub
git checkout c3cceac115
# go back to third_party directory and git clone pybind11
cd ..
git clone [email protected]:pybind/pybind11.git pybind11
cd pybind11
git checkout 70a58c5
cd ..
git clone [email protected]:gabime/spdlog.git spdlog
cd spdlog
git checkout 9e8e52c048
If it fails when git clone via SSH, you can try the HTTPS protocol like this:
git clone -b main https://github.com/open-mmlab/mmdeploy.git --recursive
Build
Please visit the link below to learn how to build MMDeploy according to the target platform.
linux-x86_64.md
windows.md
Windows-x86_64
4. How to write the configuration
This tutorial describes how to write configurations for model conversion and deployment. Deployment configuration includes onnx config, codebase config, backend config.
1. How to write onnx configuration
Onnx configuration describes how to export models from pytorch to onnx.
onnx 配置参数的描述
type：配置字典的类型。默认为onnx.
export_params：如果指定，将导出所有参数。如果您想导出未经训练的模型，请将其设置为 False。
keep_initializers_as_inputs：如果为 True，则导出图中的所有初始值设定项（通常对应于参数）也将作为输入添加到图中。如果为 False，则初始值设定项不会添加为图形的输入，并且仅将非参数输入添加为输入。
opset_version：Opset_version 默认为 11。
save_file：输出onnx文件。
input_names：分配给图形输入节点的名称。
output_names：分配给图的输出节点的名称。
input_shape：模型输入张量的高度和宽度。
onnx_config = dict(
type='onnx',
export_params=True,
keep_initializers_as_inputs=False,
opset_version=11,
save_file='end2end.onnx',
input_names=['input'],
output_names=['output'],
input_shape=None)
If you need to use dynamic axes
If you need dynamic shapes for input and output, you need to add dynamic_axes dict in onnx configuration.
dynamic_axes: describe the dimension information of input and output
dynamic_axes={
'input': {
0: 'batch',
2: 'height',
3: 'width'
},
'dets': {
0: 'batch',
1: 'num_dets',
},
'labels': {
0: 'batch',
1: 'num_dets',
},
}
2. How to write code base configuration
The repository configuration section contains information such as repository type and task type.
Description of code base configuration parameters
type: the code base of the model, including mmpretrain, mmdet, mmseg, mmocr, mmagic.
task: the task type of the model, refer to the task list in all code bases
codebase_config = dict(type='mmpretrain', task='Classification')
3. How to write the backend configuration
The backend configuration is mainly used to specify the backend of the model running, and provide the information required for the model to run on the backend, refer to ONNX Runtime, TensorRT, ncnn, PPLNN.
type: model background, including onnxruntime,,,,,. ncnnpplnntensorrtopenvino
backend_config = dict(
type='tensorrt',
common_config=dict(
fp16_mode=False, max_workspace_size=1 << 30),
model_inputs=[
dict(
input_shapes=dict(
input=dict(
min_shape=[1, 3, 512, 1024],
opt_shape=[1, 3, 1024, 2048],
max_shape=[1, 3, 2048, 2048])))
])
4. A complete example of mmpretrain on TensorRT
Here we provide a complete deployment configuration of mmpretrain on TensorRT.
codebase_config = dict(type='mmpretrain', task='Classification')
backend_config = dict(
type='tensorrt',
common_config=dict(
fp16_mode=False,
max_workspace_size=1 << 30),
model_inputs=[
dict(
input_shapes=dict(
input=dict(
min_shape=[1, 3, 224, 224],
opt_shape=[4, 3, 224, 224],
max_shape=[64, 3, 224, 224])))])
onnx_config = dict(
type='onnx',
dynamic_axes={
'input': {
0: 'batch',
2: 'height',
3: 'width'
},
'output': {
0: 'batch'
}
},
export_params=True,
keep_initializers_as_inputs=False,
opset_version=11,
save_file='end2end.onnx',
input_names=['input'],
output_names=['output'],
input_shape=[224, 224])
5. Naming rules for our deployment configuration
There is a specific naming convention for the filename of the deployment configuration file.
(task name) (backend name) (dynamic or static).py
task name: the task type of the model.
backend name: The name of the backend. Note that if you use the quantization function, you need to specify the quantization type. Like tensorrt-int8.
dynamic or static: Dynamic or static export. Please note that if the backend requires explicit shape information, you need to add a description of the input size and height x width format. Like dynamic-512x1024-2048x2048, which means the minimum input shape is 512x1024 and the maximum input shape is 2048x2048.
detection_tensorrt-int8_dynamic-320x320-1344x1344.py
6. How to write the model configuration
According to the code base of the model, write the model configuration file. The configuration file of the model is used to initialize the model, refer to MMPretrain, MMDetection, MMSegmentation, MMOCR, MMagic.
5. Conversion model
Once installed, you can enjoy your model deployment journey starting with converting your PyTorch model to your current model by running tools/deploy.py.
Based on the above settings, we provide an example of converting Faster R-CNN in MMDetection to TensorRT as follows:
# clone mmdeploy to get the deployment config. `--recursive` is not necessary
git clone -b main https://github.com/open-mmlab/mmdeploy.git
# clone mmdetection repo. We have to use the config file to build PyTorch nn module
git clone -b 3.x https://github.com/open-mmlab/mmdetection.git
cd mmdetection
mim install -v -e .
cd ..
# download Faster R-CNN checkpoint
wget -P checkpoints https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
# run the command to start model conversion
python mmdeploy/tools/deploy.py \
mmdeploy/configs/mmdet/detection/detection_tensorrt_dynamic-320x320-1344x1344.py \
mmdetection/configs/faster_rcnn/faster-rcnn_r50_fpn_1x_coco.py \
checkpoints/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth \
mmdetection/demo/demo.jpg \
--work-dir mmdeploy_model/faster-rcnn \
--device cuda \
--dump-info
The converted model and its meta information will be found in the path specified by --work-dir. They make up the MMDeploy model, which can be fed into the MMDeploy SDK for model inference.
For more details on model conversion, you can read how_to_convert_model. If you want to customize the transformation pipeline, you can follow this tutorial to edit the configuration file.
This tutorial briefly describes how to use the MMDeploy tool to export an OpenMMlab model to a specific backend. notes:
Supported backends are ONNXRuntime, TensorRT, ncnn, PPLNN, OpenVINO.
The supported code libraries are MMPretrain, MMDetection, MMSegmentation, MMOCR, MMagic.
How to convert models from Pytorch to other backends
prerequisites
Install and build your target backend. You can refer to ONNXRuntime-install, TensorRT-install, ncnn-install, PPLNN-install, OpenVINO-install for more information.
Install and build your object code library. You can refer to MMPretrain-install, MMDetection-install, MMSegmentation-install, MMOCR-install, MMagic-install.
usage
python ./tools/deploy.py \
${DEPLOY_CFG_PATH} \
${MODEL_CFG_PATH} \
${MODEL_CHECKPOINT_PATH} \
${INPUT_IMG} \
--test-img ${TEST_IMG} \
--work-dir ${WORK_DIR} \
--calib-dataset-cfg ${CALIB_DATA_CFG} \
--device ${DEVICE} \
--log-level INFO \
--show \
--dump-info
Description of all parameters
**deploy_cfg: **mmdeploy's deployment configuration for the model, including the type of inference framework, whether it is quantized, whether the input shape is dynamic, etc. There may be a reference relationship between configuration files, an example is mmdeploy/mmpretrain/classification_ncnn_static.py.
**model_cfg: **The model configuration of the algorithm library, such as mmpretrain/configs/vision_transformer/vit-base-p32_ft-64xb64_in1k-384.py, regardless of the path of mmdeploy.
**checkpoint: **Torch model path. mmcv.FileClient can start with http/https, see implementation for details.
**img:** The path of the image or point cloud file used for testing during model conversion.
**–test-img:** The path to the image file used to test the model. If not specified, it will be set to None.
**–work-dir:** The path to the working directory for saving logs and models.
--calib-dataset-cfg: Only valid in int8 mode. Configuration for calibration. If not specified, it will be set to None and use the "val" dataset in the model configuration for calibration.
--device: The device used for model conversion. If not specified, it will be set to cpu. For trt, use cuda:0 format.
–log-level: Set log levels 'CRITICAL', 'FATAL', 'ERROR', 'WARN', 'WARNING', 'INFO', 'DEBUG', 'NOTSET'. If not specified, it will be set to INFO.
–show: Whether to display the detection output.
–dump-info: Whether to output SDK information.
How to find the deployment configuration corresponding to the PyTorch model
Find the model's codebase folder in configs/. For converting a yolov3 model, you need to check configs/mmdet folder.
Find the model's task folder in configs/codebase_folder/. For a yolov3 model, you need to check configs/mmdet/detection folder.
Find the deployment config file in configs/codebase_folder/task_folder/. For deploying a yolov3 model to the onnx backend, you could use configs/mmdet/detection/detection_onnxruntime_dynamic.py.
Example
python ./tools/deploy.py \
configs/mmdet/detection/detection_tensorrt_dynamic-320x320-1344x1344.py \
$PATH_TO_MMDET/configs/yolo/yolov3_d53_8xb8-ms-608-273e_coco.py \
$PATH_TO_MMDET/checkpoints/yolo/yolov3_d53_mstrain-608_273e_coco_20210518_115020-a2c3acb8.pth \
$PATH_TO_MMDET/demo/demo.jpg \
--work-dir work_dir \
--show \
--device cuda:0
5. Reasoning model
After the model is converted, we can infer not only through the Model Converter, but also through the Inference SDK.
(1) Inference through the model converter
Model Converter provides a unified API called inference_model to do this work, making all inference backend APIs transparent to users. Take the previously converted Faster R-CNN tensor model as an example,
from mmdeploy.apis import inference_model
result = inference_model(
model_cfg='mmdetection/configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py',
deploy_cfg='mmdeploy/configs/mmdet/detection/detection_tensorrt_dynamic-320x320-1344x1344.py',
backend_files=['mmdeploy_model/faster-rcnn/end2end.engine'],
img='mmdetection/demo/demo.jpg',
device='cuda:0')
'backend_files' in this API refers to backend engine file path, which MUST be put in a list, since some inference engines like OpenVINO and ncnn separate the network structure and its weights into two files.
(2) Reasoning through the SDK
You can directly run the MMDeploy demo program in the precompiled package to obtain inference results.
wget https://github.com/open-mmlab/mmdeploy/releases/download/v1.1.0/mmdeploy-1.1.0-linux-x86_64-cuda11.3.tar.gz
tar xf mmdeploy-1.1.0-linux-x86_64-cuda11.3
cd mmdeploy-1.1.0-linux-x86_64-cuda11.3
# run python demo
python example/python/object_detection.py cuda ../mmdeploy_model/faster-rcnn ../mmdetection/demo/demo.jpg
# run C/C++ demo
# build the demo according to the README.md in the folder.
./bin/object_detection cuda ../mmdeploy_model/faster-rcnn ../mmdetection/demo/demo.jpg
In the above command, the input model is SDK Model path. It is NOT engine file path but actually the path passed to --work-dir. It not only includes engine files but also meta information like 'deploy.json' and 'pipeline.json'.
In the next section, we provide an example of deploying the above converted Faster R-CNN model using different FFI (Foreign Function Interface) of the SDK.
Python API
from mmdeploy_runtime import Detector
import cv2
img = cv2.imread('mmdetection/demo/demo.jpg')
# create a detector
detector = Detector(model_path='mmdeploy_models/faster-rcnn', device_name='cuda', device_id=0)
# run the inference
bboxes, labels, _ = detector(img)
# Filter the result according to threshold
indices = [i for i in range(len(bboxes))]
for index, bbox, label_id in zip(indices, bboxes, labels):
[left, top, right, bottom], score = bbox[0:4].astype(int), bbox[4]
if score < 0.3:
continue
cv2.rectangle(img, (left, top), (right, bottom), (0, 255, 0))
cv2.imwrite('output_detection.png', img)
image_classification.py
# Copyright (c) OpenMMLab. All rights reserved.
import argparse
import cv2
from mmdeploy_runtime import Classifier
def parse_args():
parser = argparse.ArgumentParser(
description='show how to use sdk python api')
parser.add_argument('device_name', help='name of device, cuda or cpu')
parser.add_argument(
'model_path',
help='path of mmdeploy SDK model dumped by model converter')
parser.add_argument('image_path', help='path of an image')
args = parser.parse_args()
return args
def main():
args = parse_args()
img = cv2.imread(args.image_path)
classifier = Classifier(
model_path=args.model_path, device_name=args.device_name, device_id=0)
result = classifier(img)
for label_id, score in result:
print(label_id, score)
if __name__ == '__main__':
main()
object_detection.py
# Copyright (c) OpenMMLab. All rights reserved.
import argparse
import math
import cv2
from mmdeploy_runtime import Detector
def parse_args():
parser = argparse.ArgumentParser(
description='show how to use sdk python api')
parser.add_argument('device_name', help='name of device, cuda or cpu')
parser.add_argument(
'model_path',
help='path of mmdeploy SDK model dumped by model converter')
parser.add_argument('image_path', help='path of an image')
args = parser.parse_args()
return args
def main():
args = parse_args()
img = cv2.imread(args.image_path)
detector = Detector(
model_path=args.model_path, device_name=args.device_name, device_id=0)
bboxes, labels, masks = detector(img)
indices = [i for i in range(len(bboxes))]
for index, bbox, label_id in zip(indices, bboxes, labels):
[left, top, right, bottom], score = bbox[0:4].astype(int), bbox[4]
if score < 0.3:
continue
cv2.rectangle(img, (left, top), (right, bottom), (0, 255, 0))
if masks[index].size:
mask = masks[index]
blue, green, red = cv2.split(img)
x0 = int(max(math.floor(bbox[0]) - 1, 0))
y0 = int(max(math.floor(bbox[1]) - 1, 0))
mask_img = blue[y0:y0 + mask.shape[0], x0:x0 + mask.shape[1]]
cv2.bitwise_or(mask, mask_img, mask_img)
img = cv2.merge([blue, green, red])
cv2.imwrite('output_detection.png', img)
if __name__ == '__main__':
main()
image_segmentation.py
# Copyright (c) OpenMMLab. All rights reserved.
import argparse
import cv2
import numpy as np
from mmdeploy_runtime import Segmentor
def parse_args():
parser = argparse.ArgumentParser(
description='show how to use sdk python api')
parser.add_argument('device_name', help='name of device, cuda or cpu')
parser.add_argument(
'model_path',
help='path of mmdeploy SDK model dumped by model converter')
parser.add_argument('image_path', help='path of an image')
args = parser.parse_args()
return args
def get_palette(num_classes=256):
state = np.random.get_state()
# random color
np.random.seed(42)
palette = np.random.randint(0, 256, size=(num_classes, 3))
np.random.set_state(state)
return [tuple(c) for c in palette]
def main():
args = parse_args()
img = cv2.imread(args.image_path)
segmentor = Segmentor(
model_path=args.model_path, device_name=args.device_name, device_id=0)
seg = segmentor(img)
if seg.dtype == np.float32:
seg = np.argmax(seg, axis=0)
palette = get_palette()
color_seg = np.zeros((seg.shape[0], seg.shape[1], 3), dtype=np.uint8)
for label, color in enumerate(palette):
color_seg[seg == label, :] = color
# convert to BGR
color_seg = color_seg[..., ::-1]
img = img * 0.5 + color_seg * 0.5
img = img.astype(np.uint8)
cv2.imwrite('output_segmentation.png', img)
if __name__ == '__main__':
main()
C++ API
Using the SDK C++ API should follow the following pattern,
Now we apply this process to the Faster R-CNN model above.
#include <cstdlib>
#include <opencv2/opencv.hpp>
#include "mmdeploy/detector.hpp"
int main() {
const char* device_name = "cuda";
int device_id = 0;
std::string model_path = "mmdeploy_model/faster-rcnn";
std::string image_path = "mmdetection/demo/demo.jpg";
// 1. load model
mmdeploy::Model model(model_path);
// 2. create predictor
mmdeploy::Detector detector(model, mmdeploy::Device{
device_name, device_id});
// 3. read image
cv::Mat img = cv::imread(image_path);
// 4. inference
auto dets = detector.Apply(img);
// 5. deal with the result. Here we choose to visualize it
for (int i = 0; i < dets.size(); ++i) {
const auto& box = dets[i].bbox;
fprintf(stdout, "box %d, left=%.2f, top=%.2f, right=%.2f, bottom=%.2f, label=%d, score=%.4f\n",
i, box.left, box.top, box.right, box.bottom, dets[i].label_id, dets[i].score);
if (bboxes[i].score < 0.3) {
continue;
}
cv::rectangle(img, cv::Point{
(int)box.left, (int)box.top},
cv::Point{
(int)box.right, (int)box.bottom}, cv::Scalar{
0, 255, 0});
}
cv::imwrite("output_detection.png", img);
return 0;
}
When you build this example, try adding the MMDeploy package in your CMake project as shown below. Then pass it to -DMMDeploy_DIRcmake, where cmake indicates the path MMDeployConfig.cmake. You can find it in a prebuilt package.
When you build this example, try to add MMDeploy package in your CMake project as following. Then pass -DMMDeploy_DIR to cmake, which indicates the path where MMDeployConfig.cmake locates. You can find it in the prebuilt package.
find_package(MMDeploy REQUIRED)
target_link_libraries(${name} PRIVATE mmdeploy ${OpenCV_LIBS})
For more SDK C++ API usage, read these samples
. For the remaining C, C#, and Java API usage, read the C demo, C# demo, and Java demo, respectively. We'll talk more about them in the next release.
Speed up preprocessing (experimental)
If you want to integrate preprocessing to speed up, please refer to this document to add a link description
In some cases, MMDeploy provides the ability to fuse transformations for acceleration.
When using the SDK for inference, you can edit pipeline.json to enable the fuse option.
To bring the ability of fuse transformation to MMDeploy, you can refer to the use of CVFusion.
6. Evaluation model
You can test the performance of deployed models using tool/test.py. For example
python ${MMDEPLOY_DIR}/tools/test.py \
${MMDEPLOY_DIR}/configs/detection/detection_tensorrt_dynamic-320x320-1344x1344.py \
${MMDET_DIR}/configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py \
--model ${BACKEND_MODEL_FILES} \
--metrics ${METRICS} \
--device cuda:0
You can read How to Evaluate Models for more details. add link description
Useful Tools
In addition to deploy.py, there are other useful tools tools/ in this directory.