Article directory
https://github.com/DeployAI/nndeploy
https://nndeploy-zh.readthedocs.io/zh/latest/introduction/index.html
Through the above two official links, you should have an understanding of the basic usage of nndeploy.
The following is using nndeploy to run an unet type network for image denoising.
For convenience, I directly modified the source code and recompiled a demo. As long as everyone understands it, you can organize and improve it yourself and create a new demo, because I did not create a new deno here, but directly modified the relevant code of yolov5.
1. Modify directly in the source code demo
Source code If you want to run yolo5 which is a target detection model, you can pass the following command:
onnxruntime:115ms
./install/lib /demo_nndeploy_detect --name NNDEPLOY_YOLOV5 --inference_type kInferenceTypeOnnxRuntime --device_type kDeviceTypeCodeX86:0 --model_type kModelTypeOnnx --is_path --model_value …/…/yolov5s.onnx --input_type kInputTypeImage --input_path …/…/sample.jpg --output_path …/…/sample_output.jpg
openvino:57ms
./install/lib/demo_nndeploy_detect --name NNDEPLOY_YOLOV5 --inference_type kInferenceTypeOpenVino --device_type kDeviceTypeCodeX86:0 --model_type kModelTypeOnnx --is_path --model_value …/…/yolov5s.onnx --input_type kInputTypeImage --input_path …/…/sample.jpg --output_path …/…/sample_output.jpg
mnn:78ms
./install/lib/demo_nndeploy_detect --name NNDEPLOY_YOLOV5 --inference_type kInferenceTypeMnn --device_type kDeviceTypeCodeX86:0 --model_type kModelTypeMnn --is_path --model_value …/…/yolov5s.mnn --input_type kInputTypeImage --input_path …/…/sample.jpg --output_path …/…/sample_output.jpg
tensorrt: 17ms
./install/lib/demo_nndeploy_detect --name NNDEPLOY_YOLOV5 --inference_type kInferenceTypeTensorRt --device_type kDeviceTypeCodeCuda:0 --model_type kModelTypeOnnx --is_path --model_value …/…/yolov5s.onnx --input_type kInputTypeImage --input_path …/…/sample.jpg --output_path …/…/sample_output.jpg
Then I directly modify the source code and compile it to run unet denoise model through the following command.
tensorrt
./install/lib/demo_nndeploy_detect --name NNDEPLOY_YOLOV5 --inference_type kInferenceTypeTensorRt --device_type kDeviceTypeCodeCuda:0 --model_type kModelTypeOnnx --is_path --model_value /home/tony/nndeploy/mymodel/scripts/unet8.opt.onnx --input_type kInputTypeImage --input_path …/…/1007_01_06_40_000101.png --output_path …/…/sample_output.jpg
onnxruntime
./install/lib/demo_nndeploy_detect --name NNDEPLOY_YOLOV5 --inference_type kInferenceTypeOnnxRuntime --device_type kDeviceTypeCodeX86:0 --model_type kModelTypeOnnx --is_path --model_value /home/tony/nndeploy/mymodel/scripts/unet8.opt.onnx --input_type kInputTypeImage --input_path …/…/1007_01_06_40_000101.png --output_path …/…/sample_output.jpg
openvino:
./install/lib/demo_nndeploy_detect --name NNDEPLOY_YOLOV5 --inference_type kInferenceTypeOpenVino --device_type kDeviceTypeCodeX86:0 --model_type kModelTypeOnnx --is_path --model_value /home/tony/nndeploy/mymodel/scripts/unet8.opt.onnx --input_type kInputTypeImage --input_path …/…/1007_01_06_40_000101.png --output_path …/…/sample_output.jpg
MNN:
./install/lib/demo_nndeploy_detect --name NNDEPLOY_YOLOV5 --inference_type kInferenceTypeMnn --device_type kDeviceTypeCodeX86:0 --model_type kModelTypeMnn --is_path --model_value /home/tony/nndeploy/mymodel/scripts/unet8.opt.mnn --input_type kInputTypeImage --input_path …/…/1007_01_06_40_000101.png --output_path …/…/sample_output.jpg
TNN:
./install/lib/demo_nndeploy_detect --name NNDEPLOY_YOLOV5 --inference_type kInferenceTypeTnn --device_type kDeviceTypeCodeX86:0 --model_type kModelTypeTnn --is_path --model_value /home/tony/nndeploy/mymodel/scripts/unet8.sim.tnnproto,/home/tony/nndeploy/mymodel/scripts/unet8.sim.tnnmodel --input_type kInputTypeImage --input_path …/…/1007_01_06_40_000101.png --output_path …/…/sample_output.jpg
2. How to modify it?
First understand my model, the input and output are c,h,w, 0-1, float32
First modify demo.cc:
The input and output of the graph are set: h, w, c, 0-1, float32
. The input is h, w, c float32
and the output is also h. ,w,c float32
A complete graph includes:
- The pre-processing needs to be completed from h, w, c foat32 -> c, h, w float32
2. Then infer runs model;: the input and output are both c, h, w, 0-1, float32
3. Then the post-processing needs to be completed from c, h, w, float32 -> h, w, c, float32
In fact, using a framework, you need to obtain the pointers of the model input and output, and then you can use opencv for pre- and post-processing. Many frameworks have their own pre- and post-processing (I don’t like to use it very much. It feels unclear and there is a learning cost. )
Complete code:
#include "flag.h"
#include "nndeploy/base/glic_stl_include.h"
#include "nndeploy/base/time_profiler.h"
#include "nndeploy/dag/node.h"
#include "nndeploy/device/device.h"
#include "nndeploy/model/detect/yolo/yolo.h"
using namespace nndeploy;
cv::Mat drawBox(cv::Mat &cv_mat, model::DetectResult &result) {
// float w_ratio = float(cv_mat.cols) / float(640);
// float h_ratio = float(cv_mat.rows) / float(640);
float w_ratio = float(cv_mat.cols);
float h_ratio = float(cv_mat.rows);
const int CNUM = 80;
cv::RNG rng(0xFFFFFFFF);
cv::Scalar_<int> randColor[CNUM];
for (int i = 0; i < CNUM; i++)
rng.fill(randColor[i], cv::RNG::UNIFORM, 0, 256);
int i = -1;
for (auto bbox : result.bboxs_) {
std::array<float, 4> box;
box[0] = bbox.bbox_[0]; // 640.0;
box[2] = bbox.bbox_[2]; // 640.0;
box[1] = bbox.bbox_[1]; // 640.0;
box[3] = bbox.bbox_[3]; // 640.0;
box[0] *= w_ratio;
box[2] *= w_ratio;
box[1] *= h_ratio;
box[3] *= h_ratio;
int width = box[2] - box[0];
int height = box[3] - box[1];
int id = bbox.label_id_;
NNDEPLOY_LOGE("box[0]:%f, box[1]:%f, width :%d, height :%d\n", box[0],
box[1], width, height);
cv::Point p = cv::Point(box[0], box[1]);
cv::Rect rect = cv::Rect(box[0], box[1], width, height);
cv::rectangle(cv_mat, rect, randColor[id]);
std::string text = " ID:" + std::to_string(id);
cv::putText(cv_mat, text, p, cv::FONT_HERSHEY_PLAIN, 1, randColor[id]);
}
return cv_mat;
}
//
int main(int argc, char *argv[]) {
gflags::ParseCommandLineNonHelpFlags(&argc, &argv, true);
if (demo::FLAGS_usage) {
demo::showUsage();
return -1;
}
// 检测模型的有向无环图graph名称，例如:
// NNDEPLOY_YOLOV5/NNDEPLOY_YOLOV6/NNDEPLOY_YOLOV8
std::string name = demo::getName();
// 推理后端类型，例如:
// kInferenceTypeOpenVino / kInferenceTypeTensorRt / kInferenceTypeOnnxRuntime
base::InferenceType inference_type = demo::getInferenceType();
// 推理设备类型，例如:
// kDeviceTypeCodeX86:0/kDeviceTypeCodeCuda:0/...
base::DeviceType device_type = demo::getDeviceType();
// 模型类型，例如:
// kModelTypeOnnx/kModelTypeMnn/...
base::ModelType model_type = demo::getModelType();
// 模型是否是路径
bool is_path = demo::isPath();
// 模型路径或者模型字符串
std::vector<std::string> model_value = demo::getModelValue();
// 有向无环图graph的输入边packert
dag::Edge input("detect_in");
// 有向无环图graph的输出边packert
dag::Edge output("detect_out");
// 创建检测模型有向无环图graph
dag::Graph *graph =
dag::createGraph(name, inference_type, device_type, &input, &output,
model_type, is_path, model_value);
if (graph == nullptr) {
NNDEPLOY_LOGE("graph is nullptr");
return -1;
}
// 初始化有向无环图graph
NNDEPLOY_TIME_POINT_START("graph->init()");
base::Status status = graph->init();
if (status != base::kStatusCodeOk) {
NNDEPLOY_LOGE("graph init failed");
return -1;
}
NNDEPLOY_TIME_POINT_END("graph->init()");
// 有向无环图graph的输入图片路径
std::string input_path = demo::getInputPath();
// opencv读图
cv::Mat input_mat = cv::imread(input_path);
int img_h = input_mat.rows;
int img_w = input_mat.cols;
input_mat.convertTo(input_mat, CV_32FC3, 1.0/255);
// 将图片写入有向无环图graph输入边
input.set(input_mat);
// 定义有向无环图graph的输出结果
cv::Mat result(img_h, img_w, CV_32FC3);
//model::DetectResult result;
// 将输出结果写入有向无环图graph输出边
output.set(result);
// 有向无环图Graphz运行
NNDEPLOY_TIME_POINT_START("graph->run()");
status = graph->run();
if (status != base::kStatusCodeOk) {
NNDEPLOY_LOGE("graph run failed");
return -1;
}
NNDEPLOY_TIME_POINT_END("graph->run()");
//drawBox(input_mat, result);
std::string ouput_path = demo::getOutputPath();
result.convertTo(result, CV_8UC3, 255);
//cv::imwrite("ret.png", output);
cv::imwrite(ouput_path, result);
// 有向无环图graphz反初始化
NNDEPLOY_TIME_POINT_START("graph->deinit()");
status = graph->deinit();
if (status != base::kStatusCodeOk) {
NNDEPLOY_LOGE("graph deinit failed");
return -1;
}
NNDEPLOY_TIME_POINT_END("graph->deinit()");
NNDEPLOY_TIME_PROFILER_PRINT("detetct time profiler");
// 有向无环图graphz销毁
delete graph;
NNDEPLOY_LOGE("hello world!\n");
return 0;
}
3. Modify graph
A complete graph includes:
- The pre-processing needs to be completed from h, w, c foat32 -> c, h, w float32
2. Then infer runs model;: the input and output are both c, h, w, 0-1, float32
3. Then the post-processing needs to be completed from c, h, w, float32 -> h, w, c, float32
dag::Graph* createYoloV5Graph(const std::string& name,
base::InferenceType inference_type,
base::DeviceType device_type, dag::Edge* input,
dag::Edge* output, base::ModelType model_type,
bool is_path,
std::vectorstd::string model_value)
中修改前后处理函数即可。
Pre-processing, infer, post-processing is a graph, which is the complete graph in the demo.
The input and output in the demo are the input and output of the complete graph.
Then pre-processing, infer, and post-processing also have their own input and output internally, so don't get confused.
For example, the model infer input and output are the results of c,h,w, float32.
The post-processing input is the data of c,h,w float32, and the output is converted to h, w,c float32 data (corresponding to the above cv::Mat result(img_h, img_w, CV_32FC3);)
Then we modify the post-processing function to
base::Status YoloPostProcess::runV5V6() {
// data, img_data 就是输入和输出的指针，将图像从c,h,w转为h，w,c
device::Tensor* tensor = inputs_[0]->getTensor();
float* data = (float*)tensor->getPtr();
int batch = tensor->getBatch();
int channel = tensor->getChannel();
int height = tensor->getHeight();
int width = tensor->getWidth();
NNDEPLOY_LOGE("batch:%d, channel:%d, height:%d, width:%d. (%f,%f,%f))\n", batch, channel, height, width, data[0], data[1], data[2]);
cv::Mat* dst = outputs_[0]->getCvMat();
NNDEPLOY_LOGE("mat channel:%d, height:%d, width:%d.\n", dst->channels(), dst->rows, dst->cols);
auto* img_data = (float*)dst->data;
for (int h = 0; h < height; h++)
{
for (int w = 0; w < width; w++)
{
for (int c = 0; c < 3; c++)
{
int in_index = h * width * 3 + w * 3 + c;
int out_index = c * width * height + h * width + w;
// if (w < 10)
// if(h < 10)
// printf("%.2f,", data[out_index]);
img_data[in_index] = data[out_index];
}
}
// if(h < 10)
// printf("\n");
}
return base::kStatusCodeOk;
}
The same goes for pre-processing.
4. Summary
After the pre- and post-processing is completed, recompile, and the obtained ./install/lib/demo_nndeploy_detect is to process UNET type input and output. It cannot process the target detection network. The input of target detection is uint8 image, and the output is It is a series of target frames, and the corresponding pre- and post-processing are different.
A 1080P image can run 30-60ms on the denoising model I trained (not quantized) based on openvino on AMD's Ryzen Embedded.
Comparison before and after noise reduction: The most important thing about noise reduction is that it does not lose details and improves clarity.
