自动驾驶多任务框架Hybridnets——同时处理车辆检测、可驾驶区域分割、车道线分割模型部署（C++/Python）

一、多感知任务

在移动机器人的感知系统，包括自动驾驶汽车和无人机，会使用多种传感器来获取关键信息，从而实现对环境的感知和物体检测。这些传感器包括相机、激光雷达、雷达、惯性测量单元（IMU）、全球导航卫星系统（GNSS）等。在自动驾驶和移动机器人的感知系统里面，相机的功能主要是周围的物体，这些感知任务包括任务包括物体检测和分割。
物体检测：物体检测是指通过处理传感器数据来检测环境中的物体，如汽车、行人、骑自行车的人、红绿灯等。YOLO系列网络是常用于实时物体检测的解决方案。这些网络可以实时识别多个物体，并提供它们的位置和边界框。

实例分割：实例分割是一项更复杂的任务，它不仅可以检测物体，还可以将它们分割成属于不同实例的部分。这对于跟踪和识别不同的车辆或行人非常有用。通常，Mask R-CNN等网络用于实例分割。

语义分割：语义分割是将图像中的每个像素分配到特定的语义类别的任务，例如将道路、建筑物、车辆等分割成不同的区域。U-Net和全卷积网络（FCN）等CNN架构通常用于语义分割。
在处理这些任务时，有两种主要方法可以考虑：
单一模型：开发一个单一的神经网络模型，可以同时处理多个感知任务，如检测、分割。这种方法可以提高计算效率，但需要大量的训练数据和复杂的网络结构。
多模型集成：使用多个独立的模型来处理不同的感知任务，每个模型专注于一个任务。这种方法可以更容易地处理不同任务之间的数据不平衡，并且可以根据任务的重要性对不同模型进行资源分配。

二、Hybridnets多任务框架

Hybridnets一种多任务处理的端到端多感知网络，它提出了几种关键的优化方法以提高精度。

1. 引入了基于加权双向特征网络的高效分割头和边界框/类别预测网络。
2. 提出了加权双向特征网络中各层的自动定制锚定方法。
3. 提出了一种有效的训练损失函数和训练策略，以平衡和优化网络性能。

基于这些优化，Hybridnets开发了一种端到端感知网络，用于执行多任务处理，包括交通目标检测、可驾驶区域分割和车道线检测，将其称为混合网。混合网在伯克利DeepDrive数据集上表现出色，平均精度达到了77.3%，平均交并比为31.6%，并且参数数量仅为1283万，浮点操作数为156亿。此外，它能够实时执行视觉感知任务，因此是一个实用而准确的多任务处理解决方案。

算法源码：https://github.com/datvuthanh/HybridNets.git

三、使用C++进行模型部署

把训练好的模型转成onnx格式的模型，然后使用OpenCV 4.6 和contrib dnn模块进行推理，使用的IDE是Vs 2019。
首先定义一人模型推理的类，这里使用为了方便演示，只用CPU进行推理。

struct Net_config
{
    
    
	float confThreshold; 
	float nmsThreshold; 
	std::string modelpath;
	std::string anchorpath;
};

class HybridNets
{
    
    
public:
	HybridNets(Net_config config);
	cv::Mat detect(cv::Mat frame); 
	~HybridNets(); 
private:
	int inpWidth;
	int inpHeight;
	std::vector<std::string> det_class_names = {
    
     "car" };
	std::vector<std::string> seg_class_names = {
    
     "Background", "Lane", "Line" };
	int det_num_class;
	int seg_numclass;

	float confThreshold;
	float nmsThreshold;
	cv::dnn::Net net;
	float* anchors = nullptr;
	const float mean_[3] = {
    
     0.485, 0.456, 0.406 };
	const float std_[3] = {
    
     0.229, 0.224, 0.225 };
	const bool keep_ratio = true;
	cv::Mat resize_image(cv::Mat srcimg, int* newh, int* neww, int* padh, int* padw);
	cv::Mat normalize_(cv::Mat img);
	std::vector<cv::Vec3b> class_colors = {
    
     cv::Vec3b(0,0,0), cv::Vec3b(0, 255, 0), cv::Vec3b(255, 0, 0) };
};

HybridNets::HybridNets(Net_config config)
{
    
    
	this->confThreshold = config.confThreshold;
	this->nmsThreshold = config.nmsThreshold;

	this->net = cv::dnn::readNet(config.modelpath); 
	this->det_num_class = det_class_names.size();
	this->seg_numclass = seg_class_names.size();

	size_t pos = config.modelpath.rfind("_");
	size_t pos_ = config.modelpath.rfind(".");
	int len = pos_ - pos - 1;
	std::string hxw = config.modelpath.substr(pos + 1, len);

	pos = hxw.rfind("x");
	std::string h = hxw.substr(0, pos);
	len = hxw.length() - pos;
	std::string w = hxw.substr(pos + 1, len);
	this->inpHeight = stoi(h);
	this->inpWidth = stoi(w);

	pos = config.anchorpath.rfind("_");
	pos_ = config.anchorpath.rfind(".");
	len = pos_ - pos - 1;
	std::string len_ = config.anchorpath.substr(pos + 1, len);
	len = stoi(len_);
	this->anchors = new float[len];
	FILE* fp = fopen(config.anchorpath.c_str(), "rb");
	fread(this->anchors, sizeof(float), len, fp);
	fclose(fp);
}

HybridNets::~HybridNets()
{
    
    
	delete[] anchors;
	anchors = nullptr;
}

cv::Mat HybridNets::resize_image(cv::Mat srcimg, int* newh, int* neww, int* padh, int* padw)
{
    
    
	int srch = srcimg.rows, srcw = srcimg.cols;
	*newh = this->inpHeight;
	*neww = this->inpWidth;
	cv::Mat dstimg;
	if (this->keep_ratio && srch != srcw) {
    
    
		float hw_scale = (float)srch / srcw;
		if (hw_scale > 1) {
    
    
			*newh = this->inpHeight;
			*neww = int(this->inpWidth / hw_scale);
			resize(srcimg, dstimg, cv::Size(*neww, *newh), cv::INTER_LINEAR);
			*padw = int((this->inpWidth - *neww) * 0.5);
			copyMakeBorder(dstimg, dstimg, 0, 0, *padw, this->inpWidth - *neww - *padw, cv::BORDER_CONSTANT, 114);
		}
		else {
    
    
			*newh = (int)this->inpHeight * hw_scale;
			*neww = this->inpWidth;
			resize(srcimg, dstimg, cv::Size(*neww, *newh), cv::INTER_LINEAR);
			*padh = (int)(this->inpHeight - *newh) * 0.5;
			copyMakeBorder(dstimg, dstimg, *padh, this->inpHeight - *newh - *padh, 0, 0, cv::BORDER_CONSTANT, 114);
		}
	}
	else {
    
    
		resize(srcimg, dstimg, cv::Size(*neww, *newh), cv::INTER_LINEAR);
	}
	return dstimg;
}

cv::Mat HybridNets::normalize_(cv::Mat img)
{
    
    
	std::vector<cv::Mat> bgrChannels(3);
	split(img, bgrChannels);
	for (int c = 0; c < 3; c++)
	{
    
    
		bgrChannels[c].convertTo(bgrChannels[c], CV_32FC1, 1.0 / (255.0 * std_[c]), (0.0 - mean_[c]) / std_[c]);
	}
	cv::Mat m_normalized_mat;
	merge(bgrChannels, m_normalized_mat);
	return m_normalized_mat;
}

cv::Mat HybridNets::detect(cv::Mat srcimg)
{
    
    
	int newh = 0, neww = 0, padh = 0, padw = 0;
	cv::Mat rgbimg;
	cvtColor(srcimg, rgbimg, cv::COLOR_BGR2RGB);
	cv::Mat dstimg = this->resize_image(rgbimg, &newh, &neww, &padh, &padw);
	cv::Mat normalized_mat = this->normalize_(dstimg);


	cv::Mat blob = cv::dnn::blobFromImage(normalized_mat);
	this->net.setInput(blob);
	std::vector<cv::Mat> outs;
	this->net.forward(outs, this->net.getUnconnectedOutLayersNames());

	float* classification = (float*)outs[0].data;
	float* box_regression = (float*)outs[1].data;
	float* seg = (float*)outs[2].data;

	std::vector<cv::Rect> boxes;
	std::vector<float> confidences;
	std::vector<int> classIds;
	float ratioh = (float)srcimg.rows / newh, ratiow = (float)srcimg.cols / neww;
	const int num_proposal = outs[1].size[1];  
	for (int n = 0; n < num_proposal; n++)
	{
    
    
		float conf = classification[n];
	
		if (conf > this->confThreshold)
		{
    
    
			const int row_ind = n * 4;
			float x_centers = box_regression[row_ind + 1] * this->anchors[row_ind + 2] + this->anchors[row_ind];
			float y_centers = box_regression[row_ind] * this->anchors[row_ind + 3] + this->anchors[row_ind + 1];
			float w = exp(box_regression[row_ind + 3]) * this->anchors[row_ind + 2];
			float h = exp(box_regression[row_ind + 2]) * this->anchors[row_ind + 3];

			float xmin = (x_centers - w * 0.5 - padw) * ratiow;
			float ymin = (y_centers - h * 0.5 - padh) * ratioh;
			w *= ratiow;
			h *= ratioh;
			cv::Rect box = cv::Rect(int(xmin), int(ymin), int(w), int(h));
			boxes.push_back(box);
			confidences.push_back(conf);
			classIds.push_back(0);
		}
	}
	
	std::vector<int> indices;
	cv::dnn::NMSBoxes(boxes, confidences, this->confThreshold, this->nmsThreshold, indices);

	cv::Mat outimg = srcimg.clone();
	for (size_t i = 0; i < indices.size(); ++i)
	{
    
    
		int idx = indices[i];
		cv::Rect box = boxes[idx];
		rectangle(outimg, cv::Point(box.x, box.y), cv::Point(box.x + box.width, box.y + box.height), cv::Scalar(0, 0, 255), 2);
		std::string label = cv::format("%.2f", confidences[idx]);
		label = this->det_class_names[classIds[idx]] + ":" + label;
		putText(outimg, label, cv::Point(box.x, box.y - 5), cv::FONT_HERSHEY_SIMPLEX, 0.75, cv::Scalar(0, 0, 255), 1);
	}

	const int area = this->inpHeight * this->inpWidth;
	int i = 0, j = 0, c = 0;
	for (i = 0; i < outimg.rows; i++)
	{
    
    
		for (j = 0; j < outimg.cols; j++)
		{
    
    
			const int x = int(j / ratiow) + padw;  
			const int y = int(i / ratioh) + padh;
			int max_id = -1;
			float max_conf = -10000;
			for (c = 0; c < seg_numclass; c++)
			{
    
    
				float seg_conf = seg[c * area + y * this->inpWidth + x];
				if (seg_conf > max_conf)
				{
    
    
					max_id = c;
					max_conf = seg_conf;
				}
			}
			if (max_id > 0)
			{
    
    
				outimg.at<cv::Vec3b>(i, j)[0] = this->class_colors[max_id][0];
				outimg.at<cv::Vec3b>(i, j)[1] = this->class_colors[max_id][1];
				outimg.at<cv::Vec3b>(i, j)[2] = this->class_colors[max_id][2];
			}
		}
	}
	return outimg;
}

读取视频，然后使用模型进行推理

int main()
{
    
    
	Net_config cfg = {
    
     0.3, 0.5, "models/hybridnets_768x1280.onnx", "models/anchors_736560.bin" };
	HybridNets net(cfg);

	//cv::VideoWriter outputVideo;
	cv::VideoCapture cap("test2.mp4");

	//cv::Size S = cv::Size((int)cap.get(cv::CAP_PROP_FRAME_WIDTH),
		//(int)cap.get(cv::CAP_PROP_FRAME_HEIGHT));
	//std::string out_path = "dst.ma4";
	if (cap.isOpened())
	{
    
    
		//outputVideo.open(out_path, -1, 30.0, S, true);
		cv::Mat cv_fram;
		while (1)
		{
    
    
			cap.read(cv_fram);
			if (!cv_fram.empty())
			{
    
    
				cv::Mat outimg = net.detect(cv_fram);
				//outputVideo << outimg;
				cv::imshow("视频", outimg);
			}
			
			if (cv::waitKey(100) == 27)break;
		}
	}

	cap.release();
	return 0;
}

推理效果如下：

四、使用Python进行模型部署

import cv2
import argparse
import numpy as np
import os

print(cv2.__version__)

class HybridNets():
    def __init__(self, modelpath, anchorpath, confThreshold=0.5, nmsThreshold=0.5):
        self.det_classes = ["car"]
        self.seg_classes = ["Background", "Lane", "Line"]

        self.net = cv2.dnn.readNet(modelpath)
        self.confThreshold = confThreshold
        self.nmsThreshold = nmsThreshold
        
        h, w = os.path.basename(modelpath).split('_')[-1].replace('.onnx', '').split('x')
        self.inpHeight, self.inpWidth = int(h), int(w)
        self.mean_ = np.array([0.485, 0.456, 0.406], dtype=np.float32).reshape((1, 1, 3))
        self.std_ = np.array([0.229, 0.224, 0.225], dtype=np.float32).reshape((1, 1, 3))
        self.anchors = np.load(anchorpath)  ### cx_cy_w_h
    
    def resize_image(self, srcimg, keep_ratio=True):
        padh, padw, newh, neww = 0, 0, self.inpWidth, self.inpHeight
        if keep_ratio and srcimg.shape[0] != srcimg.shape[1]:
            hw_scale = srcimg.shape[0] / srcimg.shape[1]
            if hw_scale > 1:
                newh, neww = self.inpHeight, int(self.inpWidth / hw_scale)
                img = cv2.resize(srcimg, (neww, newh), interpolation=cv2.INTER_LINEAR)
                padw = int((self.inpWidth - neww) * 0.5)
                img = cv2.copyMakeBorder(img, 0, 0, padw, self.inpWidth - neww - padw, cv2.BORDER_CONSTANT,
                                         value=(114, 114, 114))  # add border
            else:
                newh, neww = int(self.inpHeight * hw_scale), self.inpWidth
                img = cv2.resize(srcimg, (neww, newh), interpolation=cv2.INTER_LINEAR)
                padh = int((self.inpHeight - newh) * 0.5)
                img = cv2.copyMakeBorder(img, padh, self.inpHeight - newh - padh, 0, 0, cv2.BORDER_CONSTANT,
                                         value=(114, 114, 114))
        else:
            img = cv2.resize(srcimg, (self.inpWidth, self.inpHeight), interpolation=cv2.INTER_LINEAR)
        return img, newh, neww, padh, padw
    
    def detect(self, srcimg):
        img, newh, neww, padh, padw = self.resize_image(cv2.cvtColor(srcimg, cv2.COLOR_BGR2RGB))
        scale_h, scale_w = srcimg.shape[0] / newh, srcimg.shape[1] / neww
        img = (img.astype(np.float32) / 255.0 - self.mean_) / self.std_
        # Sets the input to the network
        blob = cv2.dnn.blobFromImage(img)
        self.net.setInput(blob)
        
        classification, box_regression, seg = self.net.forward(self.net.getUnconnectedOutLayersNames())

        x_centers = box_regression[..., 1] * self.anchors[..., 2] + self.anchors[..., 0]
        y_centers = box_regression[..., 0] * self.anchors[..., 3] + self.anchors[..., 1]
        w = np.exp(box_regression[..., 3]) * self.anchors[..., 2]
        h = np.exp(box_regression[..., 2]) * self.anchors[..., 3]

        xmin = x_centers - w * 0.5
        ymin = y_centers - h * 0.5
        
        bboxes_wh = np.stack([xmin, ymin, w, h], axis=2).squeeze(axis=0)
        
        confidences = np.max(classification.squeeze(axis=0), axis=1)  ####max_class_confidence
        classIds = np.argmax(classification.squeeze(axis=0), axis=1)
        mask = confidences > self.confThreshold
        bboxes_wh = bboxes_wh[mask]
        confidences = confidences[mask]
        classIds = classIds[mask]
        
        bboxes_wh -= np.array([[padw, padh, 0, 0]])  
        bboxes_wh *= np.array([[scale_w, scale_h, scale_w, scale_h]])
        
        indices = cv2.dnn.NMSBoxes(bboxes_wh.tolist(), confidences.tolist(), self.confThreshold,
                                   self.nmsThreshold).flatten().tolist()
        
        drive_area_mask = np.squeeze(seg, axis=0)[:, padh:(self.inpHeight - padh), padw:(self.inpWidth - padw)]
        seg_id = np.argmax(drive_area_mask, axis=0).astype(np.uint8)
        seg_id = cv2.resize(seg_id, (srcimg.shape[1], srcimg.shape[0]), interpolation=cv2.INTER_NEAREST)
     
        outimg = srcimg.copy()
        for ind in indices:
            x, y, w, h = bboxes_wh[ind,:].astype(int)
            cv2.rectangle(outimg, (x, y), (x + w, y + h), (0, 0, 255), thickness=2, lineType=cv2.LINE_AA)
            cv2.putText(outimg, self.det_classes[classIds[ind]]+ ":" + str(round(confidences[ind], 2)), (x, y - 5), cv2.FONT_HERSHEY_SIMPLEX, 0.75, (0, 0, 255),
                        thickness=1, lineType=cv2.LINE_AA)

        outimg[seg_id == 1] = [0, 255, 0]
        outimg[seg_id == 2] = [255, 0, 0]
        return outimg


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument('--imgpath', type=str, default='images/test.jpg', help="image path")
    parser.add_argument('--modelpath', type=str, default='models/hybridnets_768x1280.onnx')
    parser.add_argument('--anchorpath', type=str, default='models/anchors_768x1280.npy')
    parser.add_argument('--confThreshold', default=0.3, type=float, help='class confidence')
    parser.add_argument('--nmsThreshold', default=0.5, type=float, help='nms iou thresh')
    args = parser.parse_args()
    
    yolonet = HybridNets(args.modelpath, args.anchorpath, confThreshold=args.confThreshold,
                         nmsThreshold=args.nmsThreshold)
    srcimg = cv2.imread(args.imgpath)
    srcimg = yolonet.detect(srcimg)
    
    cv2.namedWindow('dst', 0)
    cv2.imshow('dst', srcimg)
    cv2.waitKey(0)
    cv2.destroyAllWindows()