Deploying PaddlePadle-YOLOE model based on OpenVINO—3.C++ deployment implementation

1. C++ project configuration

  The current project deployment kit uses OpenVINO ^TM , and the image processing uses OpenCV, so two additional dependencies of OpenVINO ^{TM and OpenCV need to be configured here.}

 The project uses OpenVINO ^TM version 2022.2, and the OpenCV version uses 4.5.5. For specific installation methods, please refer to the following three links:

【OpenVINO】OpenVINO 2022.1 installation tutorial (Windows)

[OpenVINO] OpenVINO 2022.1 update 2022.2 tutorial

OpenCV C++ installation and configuration

2. Model reasoning class Predictor

2.1 Core of reasoning structure

// @brief 推理核心结构体
typedef struct openvino_core {
    ov::Core core; // core对象
    std::shared_ptr<ov::Model> model_ptr; // 读取模型指针
    ov::CompiledModel compiled_model; // 模型加载到设备对象
    ov::InferRequest infer_request; // 推理请求对象
} CoreStruct;

 In order to facilitate the use of the OpenVINO ^TM reasoning tool, several important member variables of the suite are encapsulated into the reasoning core structure here, so that the model can be transferred between different methods.

2.2 Predictor class structure

class Predictor {
public:
    // 构造函数
    Predictor(std::string& model_path, std::string& device_name);
    // 析构函数
    ~Predictor() { delete p; }
    // 获取节点张量
    ov::Tensor get_tensor(std::string node_name);
    // 填充图片数据
    void fill_tensor_data_image(ov::Tensor& input_tensor, const cv::Mat& input_image);
    void fill_tensor_data_image(ov::Tensor& input_tensor, const std::vector<cv::Mat> input_image);
    // 模型推理
    void infer();
    // 获取模型输出
    std::vector<float> get_output_data(std::string output_node_name);
private:
    CoreStruct* p;
};

 Since we only perform inference on the PP-YOLOE model here, there is no need to build too complex inference classes, which mainly include

• Constructor Predictor(model_path, device_name): The main function is to initialize the inference core, including three steps: reading the local model, loading the model to the device, and creating an inference channel;
• Get the node tensor get_tensor(node_name): The main implementation function is to get the tensor of the specified node, mainly used to get the input node tensor;
• Filling input data fill_tensor_data_image(input_tensor, input_image): mainly implements adding inference data to the model, supports adding single image data and adding multiple images under multi-channel inference;
• Model inference infer(): Model inference function calculates the model based on the read model and loaded inference data;
• Read model output get_output_data(output_node_name): Read the output of the inference model.

 For the implementation of the appeal method, you can refer to the source code and will not be explained in detail here.

3. Image data processing class ImageProcess

class ImageProcess {
public:
	// 预处理图片
	cv::Mat image_normalize(cv::Mat& sourse_mat, cv::Size& size);
	// 处理推理结果
	cv::Mat yoloe_result_process(cv::Mat& sourse_mat, std::vector<float>& vector_box, std::vector<float>& vector_conf);
	// 读取lable文件
	void read_class_names(std::string path_name);
	// 设置缩放比例
	void set_scale_factor(double scale);
private:
	// 缩放比例
	double scale_factor;
	// lable容器
	std::vector<std::string> class_names;
};

 A data processing class is set up here mainly to process the input and output data of the model. It mainly does not include buying a few methods first:

• Preprocessing images image_normalize(sourse_mat, size);: Preprocessing image data, including the following processing steps: 1. Convert RGB 2. Scale images 3. Image normalization;
• Processing PP-YOLOE results yoloe_result_process(sourse_mat, vector_box, vector_conf): Since we have trimmed the model and removed the post-processing and non-maximum value suppression that comes with the model, the output data of the model is more complicated. The method of encapsulation here is mainly to realize the data read from the model Process according to the specified requirements and draw it on the result picture;
• Read label file read_class_names(path_name): read local label.txt file;
• Set the scaling ratio set_scale_factor(scale): Set the scaling ratio between the original image and the model input, which is used to reply to the model prediction box.

4. Model reasoning implementation

4.1 Define relevant information

// 模型路径
//std::string model_path = "../model/ppyoloe_plus_crn_s_80e_coco.onnx";
std::string model_path = "../model/ir/ppyoloe_plus_crn_s_80e_coco.xml";
// 设备名称
std::string device_name = "CPU";
// 输入节点
std::string input__node_name = "image";
// 输出节点名
std::string output_box_node_name = "tmp_16";
std::string output_conf_node_name = "concat_14.tmp_0";

// 测试图片
std::string image_path = "../image/demo_3.jpg";
// 类别文件
std::string lable_path = "../model/lable.txt";

 First define the relevant information. The ONNX model and IR model can be used for reasoning here; the model input node is image, and the output node is:

tmp_16The prediction box output node and concat_14.tmp_0confidence output can be directly read using the OpenVINO ^{TM tool. The name of the model node is directly defined and used here.}

4.2 Preprocessing inference data

// 创建数据处理类
ImageProcess image_pro;
// 读取类别文件
image_pro.read_class_names(lable_path);
// 图片预处理
cv::Mat image = cv::imread(image_path);
cv::Size input_size(640, 640);
// 将图片放到方形背景中
int length = image.rows > image.cols ? image.rows : image.cols;
cv::Mat input_mat = cv::Mat::zeros(length, length, CV_8UC3);
cv::Rect roi(0, 0, image.cols, image.rows);
image.copyTo(input_mat(roi));

// 设置缩放比例
image_pro.set_scale_factor((double)length / 640.0);

// 归一化处理
cv::Mat input_data = image_pro.image_normalize(input_mat, input_size);

 The inference data preprocessing here is mainly ImageProcessimplemented by the method defined in the class. The input of the PP-YOLOE model is 3×640×640, and the shape is a square. Therefore, the inference data is placed in a larger square background to prevent image Shape deforms when scaled.

4.3 Model reasoning

// 创建推理通道
Predictor predictor(model_path, device_name);
// 加载模型推理数据
ov::Tensor input_tensor = predictor.get_tensor(input__node_name);
predictor.fill_tensor_data_image(input_tensor, input_data);
// 模型推理
predictor.infer();

 Here, it is implemented by calling the Predictor class defined above, initializing the Predictor class, reading the model into the memory, and loading it into the specified device, and creating an inference channel; the next step is to load the preprocessed inference data to the model; finally, the model reasoning.

4.4 Processing inference results

// 读取推理结果
std::vector<float> result_boxes = predictor.get_output_data(output_box_node_name);
std::vector<float> result_conf = predictor.get_output_data(output_conf_node_name);
// 处理推理结果
cv::Mat result_image = image_pro.yoloe_result_process(image, result_boxes, result_conf);

 The PP-TYOLOE model has two output nodes after we cut it, one is responsible for outputting the prediction frame, and the other is responsible for outputting the confidence value. After reading the prediction result, call the inference result processing method for processing.

5. Model inference effect

 Using this model, two pictures were predicted respectively. The prediction results can be viewed in the two pictures below. The second picture is the flying paddle test picture, which can be compared with the identification results of the flying paddle reasoning suite.