Opencv C++ deploys YOLOv3v4

1. Reasoning about C++ source code

0.Source code link

https://learnopencv.com/deep-learning-based-object-detection-using-yolov3-with-opencv-python-c/

1.Initialization parameters

//初始化参数
float confThreshold = 0.5; // 置信度阈值
float nmsThreshold = 0.4;  // 非极大值抑制阈值
int inpWidth = 416;        //网络输入尺寸宽度
int inpHeight = 416;       //网络输入尺寸高度

2. Load model and category labels

//加载类别名称
string classesFile = "coco.names";
ifstream ifs(classesFile.c_str());
string line;
while (getline(ifs, line)) classes.push_back(line);

//给模型喂养配置文件与权重文件
String modelConfiguration = "yolov3.cfg";
String modelWeights = "yolov3.weights";

//加载网络模型
Net net = readNetFromDarknet(modelConfiguration, modelWeights);
net.setPreferableBackend(DNN_BACKEND_OPENCV);
net.setPreferableTarget(DNN_TARGET_CPU);

3. Read the input image

outputFile = "yolo_out_cpp.avi";
if (parser.has("image"))
    {
        //打开图片文件
        str = parser.get<String>("image");
        ifstream ifile(str);
        if (!ifile) throw("error");
        cap.open(str);
        str.replace(str.end()-4, str.end(), "_yolo_out_cpp.jpg");
        outputFile = str;
    }
    else if (parser.has("video"))
    {
        //打开视频文件
        str = parser.get<String>("video");
        ifstream ifile(str);
        if (!ifile) throw("error");
        cap.open(str);
        str.replace(str.end()-4, str.end(), "_yolo_out_cpp.avi");
        outputFile = str;
    }
    //打开网络摄像机
    else cap.open(parser.get<int>("device"));

    catch(...) {
        cout << "Could not open the input image/video stream" << endl;
        return 0;
    }
    
    //初始化视频存储器，存储输出视频
    if (!parser.has("image")) {
        video.open(outputFile, VideoWriter::fourcc('M','J','P','G'), 28, Size(cap.get(CAP_PROP_FRAME_WIDTH), cap.get(CAP_PROP_FRAME_HEIGHT)));
    }

4. Process each frame

//处理每一帧
while (waitKey(1) < 0)
{
    //从视频获取图像
    cap >> frame;

    // Stop the program if reached end of video
    if (frame.empty()) {
        cout << "Done processing !!!" << endl;
        cout << "Output file is stored as " << outputFile << endl;
        waitKey(3000);
        break;
    }
    //从原图像创建一个4D的blob
    blobFromImage(frame, blob, 1/255.0, cv::Size(inpWidth, inpHeight), Scalar(0,0,0), true, false);
        
    //为网络设置输入
    net.setInput(blob);
    
    //运行前向推理的过程，得到输出层的输出
    vector<Mat> outs;
    net.forward(outs, getOutputsNames(net));
        
    //移除低置信度的输出框
    postprocess(frame, outs);
        
    //展示效率信息。函数getPerfProfile返回每一层的推理时间
    vector<double> layersTimes;
    double freq = getTickFrequency() / 1000;
    double t = net.getPerfProfile(layersTimes) / freq;
    string label = format("Inference time for a frame : %.2f ms", t);
    putText(frame, label, Point(0, 15), FONT_HERSHEY_SIMPLEX, 0.5, Scalar(0, 0, 255));
        
    //保存带有检测框的一帧图像
    Mat detectedFrame;
    frame.convertTo(detectedFrame, CV_8U);
    if (parser.has("image")) imwrite(outputFile, detectedFrame);
    else video.write(detectedFrame);
        
    imshow(kWinName, frame);        
}

2. forward function

//运行前向推理的过程，得到输出层的输出
vector<Mat> outs;
net.forward(outs, getOutputsNames(net));

The output of the function getOutputsNames()is the name of the output layer of three different scales in the network (predicting large, medium and small objects respectively).

net.forward()The function gets the output result based on the output layer name. The output content outsis an Matarray, each of which Matis a 5+nc dimension vector group (n rows and 5+nc column matrix, nc is the number of categories). outsEach element of the array is a 5+nc dimensional vector (there are three elements in total, all are Mat). The 5+nc dimensional vector is the output of each output layer in the form of a blob binary object.

The output of the YOLOv4 network is a rectangular box, each rectangular box is represented by a vector, and all rectangular boxes form a vector group. The length of each vector is the number of categories + 5 parameters . The first four of these five parameters are the position of the rectangular frame on the image, center_x, center_y, width, height (all are proportions, ranging from 0-1) , the fifth parameter is the confidence that the rectangular box may contain objects . Starting from the sixth parameter of the vector, it represents the confidence of each category of the object in the rectangular box .

Reference link: https://www.freesion.com/article/72531365556/

3.net.setPreferableBackend and net.setPreferableTarget

1.setPreferableBackend() function parameters

DNN_BACKEND_INFERENCE_ENGINE;
DNN_BACKEND_OPENCV;
DNN_BACKEND_HALIDE;
DNN_BACKEND_CUDA;
DNN_BACKEND_DEFAULT;

Among them, if OpenCV is compiled with Intel's Inference Engine library, then DNN_BACKEND_DEFAULT is DNN_BACKEND_INFERENCE_ENGINE, otherwise it is DNN_BACKEND_OPENCV.

2.setPreferableTarget() function parameters

List of supported combinations backend / target:
|                        | DNN_BACKEND_OPENCV | DNN_BACKEND_INFERENCE_ENGINE | DNN_BACKEND_HALIDE |  DNN_BACKEND_CUDA |
|------------------------|--------------------|------------------------------|--------------------|-------------------|
| DNN_TARGET_CPU         |                  + |                            + |                  + |                   |
| DNN_TARGET_OPENCL      |                  + |                            + |                  + |                   |
| DNN_TARGET_OPENCL_FP16 |                  + |                            + |                    |                   |
| DNN_TARGET_MYRIAD      |                    |                            + |                    |                   |
| DNN_TARGET_FPGA        |                    |                            + |                    |                   |
| DNN_TARGET_CUDA        |                    |                              |                    |                 + |
| DNN_TARGET_CUDA_FP16   |                    |                              |                    |                 + |

Reference link: https://blog.csdn.net/weixin_43996161/article/details/113994433