9.2. tensorRT advanced (4) package series - autonomous driving case project self-driving - depth estimation

Preface

I have read the tensorRT high-performance deployment course from scratch launched by Teacher Du before, but I didn’t take notes and I forgot many things. I’ll do it again this time and take notes.

This course learns tensorRT advanced-autonomous driving case project self-driving-depth estimation

Please see the mind map below for the course syllabus

1. Depth estimation

In this section we study the analysis of depth estimation models. Our purpose is to find the onnx of depth estimation, analyze the general usage logic of onnx, and then write the most concise version of predict.py, which can be roughly divided into the following three steps:

1. Open the onnx of depth estimation and view its input and output

2. View the code, find the preprocessing of onnx, and analyze the preprocessing logic

3. For the information obtained, write predict.py and try to write it out

Let’s take a look at its onnx model, as shown in the figure below:

Figure 1 onnx model

From the exported onnx model, we can know that the input is 1x3x256x512, and there are 6 outputs

Let's analyze the image_processor/depth_engine.cpp code in the project to get the specific preprocessing work: (Please refer to the video for detailed analysis)

1. The input is 1x3x256x512, input.1

2. The output is 1x1x256x512, 2499 nodes

3. normalize.mean = 0.485f，norm = 0.229f

4. y = (x / 255.0 - mean) / norm

5. The resize part is not so complicated, just resize directly

6. In terms of color, cvtColor is required $\to$ to RGB

We can simply write a preprocessing program to verify, the code is as follows:

import onnxruntime
import cv2
import numpy as np

session = onnxruntime.InferenceSession("workspace/ldrn_kitti_resnext101_pretrained_data_grad_256x512.onnx", provider_options=["CPUExecutionProvider"])

image = cv2.imread("workspace/imgs/dashcam_00.jpg")
image = cv2.resize(image, (512, 256))
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
image_tensor = (image / 255.0)
mean = [0.485, 0.456, 0.406]
norm = [0.229, 0.224, 0.225]
image_tensor = ((image_tensor - mean) / norm).astype(np.float32)
image_tensor = image_tensor.transpose(2, 0, 1)[None]

prob = session.run(["2499"], {
    
    "input.1": image_tensor})[0]

print(prob.shape)

prob = prob[0, 0] * -5 + 255
y = int(prob.shape[0] * 0.18)
prob = prob[y:]

cv2.imwrite("depth.jpg", prob)

The output is as shown below:

Figure 2 Preprocessing verification

It can be seen that the output meets our expectations, and the output depth estimation map is as follows:

Figure 3 Depth estimation map

In addition, we can also visualize through matplotlib, the code is as follows:

import onnxruntime
import cv2
import numpy as np
import matplotlib.pyplot as plt

session = onnxruntime.InferenceSession("workspace/ldrn_kitti_resnext101_pretrained_data_grad_256x512.onnx", provider_options=["CPUExecutionProvider"])

image = cv2.imread("workspace/imgs/dashcam_00.jpg")
image = cv2.resize(image, (512, 256))
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
image_tensor = (image / 255.0)
mean = [0.485, 0.456, 0.406]
norm = [0.229, 0.224, 0.225]
image_tensor = ((image_tensor - mean) / norm).astype(np.float32)
image_tensor = image_tensor.transpose(2, 0, 1)[None]

prob = session.run(["2499"], {
    
    "input.1": image_tensor})[0]

print(prob.shape)

prob = prob[0, 0]
y = int(prob.shape[0] * 0.18)
prob = prob[y:]

plt.imsave("depth.jpg", prob, cmap='plasma_r')

The output depth estimation map is as follows:

Figure 4 Depth estimation chart (matplotlib)

Summarize

This course studied depth estimation cases in open source projects. It mainly conducted a simple analysis of onnx of the depth estimation model, clarified the preprocessing part through the analysis of the project code, and then conducted a simple verification through onnxruntime, and performed The final depth estimation results are visually displayed