PointPillars project reproduction

先赞后看，养成好习惯。有帮助的话，点波关注！我会坚持更新，感谢谢您的支持！

Requirements : Learn and reproduce PointPillars, solve various problems encountered during deployment, the original reference project is PointPillars_MultiHead_40FPS, 注释版以及添加ROS节点版本in PointPillars_MultiHead_40FPS_ROS , for your reference and study, welcome to leave a message!

Reference project :
1. PointPillars_MultiHead_40FPS
2. OpenPCDet
3. onnx2trt
4. Possible problems during onnx2trt installation
5. Spconv

insert image description here

1. Preparations

1. Download the project

PointPillars_MultiHead_40FPS: Point cloud target detection network main project
OpenPCDet工具: used to convert pth model to onnx model
onnx-tensorrt工具: for onnx to trt model conversion.

git clone https://github.com/hova88/PointPillars_MultiHead_40FPS.git --recursive
git clone https://github.com/hova88/OpenPCDet.git 
git clone https://github.com/onnx/onnx-tensorrt.git

2. Model conversion:
注意 :
The addresses of the two official onnx link files are invalid, so the OpenPCDet tool can only be used to generate onnx.
1) cbgs_pp_multihead_pfe.onnx. 2) cbgs_pp_multihead_backbone.onnx
2.1) Convert pth to onnx
Under the OpenPCDet tool project, perform the following operations:

A. Download the *.pth weight file and copy it to the OpenPCDet project
B. Configuration modification
Mainly modify the related paths of tools/onnx_utils/trans_pfe.py and tools/onnx_utils/trans_backbone_multihead.py, for example:

## 1. trans_pfe.py, 修改main文件中三处路径, 用### 标记
if __name__ == "__main__":
    from pcdet.config import cfg, cfg_from_yaml_file
    cfg_file = '/home/cui/workspace/deepLearning/OpenPCDet/tools/cfgs/nuscenes_models/cbgs_pp_multihead.yaml' ###
    filename_mh = "/home/cui/workspace/deepLearning/OpenPCDet/pp_multihead_nds5823_updated.pth" ###
    cfg_from_yaml_file(cfg_file, cfg)
    model_cfg=cfg.MODEL
    pfe , dummy_input  = build_pfe( filename_mh, cfg)
    pfe.eval().cuda()
    export_onnx_file = "/home/cui/workspace/deepLearning/OpenPCDet/cbgs_pp_multihead_pfe.onnx" ###
    torch.onnx.export(pfe,
                    dummy_input,
                    export_onnx_file,
                    opset_version=12,
                    verbose=True,
                    do_constant_folding=True) # 输出名
                    
## 2. trans_backbone_multihead.py，同样修改三处路径，形式如下：
if __name__ == "__main__":
    from pcdet.config import cfg, cfg_from_yaml_file
    cfg_file = '/home/cui/workspace/deepLearning/OpenPCDet/tools/cfgs/nuscenes_models/cbgs_pp_multihead.yaml'
    filename_mh = "/home/cui/workspace/deepLearning/OpenPCDet/pp_multihead_nds5823_updated.pth"
    cfg_from_yaml_file(cfg_file, cfg)
    model_cfg=cfg.MODEL
    pfe , dummy_input  = build_pfe( filename_mh, cfg)
    pfe.eval().cuda()
    export_onnx_file = "/home/cui/workspace/deepLearning/OpenPCDet/cbgs_pp_multihead_pfe.onnx"
    torch.onnx.export(pfe,
                    dummy_input,
                    export_onnx_file,
                    opset_version=12,
                    verbose=True,
                    do_constant_folding=True) # 输出名

C. Install the SparseConv library
. Execute different commands according to the CUDA version in the system. For details, refer to Spconv . CUDA10.2 in this article is as follows:

pip install spconv-cu102  # 本人使用cuda10.2版本

D. Install the pcdet tool

conda create -n pointpillars python=3.6   #创建pointpillars
conda activate pointpillars    #进入pointpillars虚拟环境
pip install -r requirements.txt   # 安装依赖，在OpenPCDet工程下
python setup.py develop

E. Execute the onnx generation script

cd tools/onnx_utils
python trans_pfe.py   # 在OpenPCDet目录下，生成cbgs_pp_multihead_pfe.onnx
python trans_backbone_multihead.py  # 在OpenPCDet目录下，生成cbgs_pp_multihead_backbone.onnx

Note : 执行脚本过程中有任何问题，参考第五部分问题答疑.

2.2) Convert onnx to trt model
Copy the two onnx files in the previous step to the workspace/tools/onnx-tensorrt/build directory, and then execute the conversion command:

cd workspace/tools/onnx-tensorrt/build  # 进入onnx-tensorrt工具的目录
onnx2trt cbgs_pp_multihead_pfe.onnx -o cbgs_pp_multihead_pfe.trt -b 1 -d 16
onnx2trt cbgs_pp_multihead_backbone.onnx -o cbgs_pp_multihead_backbone.trt -b 1 -d 16

A result similar to the following is generated, that is, the generation is successful!

----------------------------------------------------------------
Input filename:   cbgs_pp_multihead_backbone.onnx
ONNX IR version:  0.0.7
Opset version:    10
Producer name:    pytorch
Producer version: 1.10
Domain:           
Model version:    0
Doc string:       
----------------------------------------------------------------
Parsing model
[2022-07-29 02:47:54 WARNING] [TRT]/home/cui/workspace/tools/onnx-tensorrt/onnx2trt_utils.cpp:220: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
Building TensorRT engine, FP16 available:1
    Max batch size:     1
    Max workspace size: 1024 MiB
Writing TensorRT engine to cbgs_pp_multihead_backbone.trt
All done

3. Modification of model path
Copy 4 model files to PointPillars/model.
If there is a model file change, you need to modify the *.onnx and *.trt model paths in the bootstrap.yaml file.

## bootstrap.yaml文件内容
BoxFeature: 7
ScoreThreshold: 0.1
NmsOverlapThreshold: 0.2
UseOnnx: false 

PfeOnnx: ../model/cbgs_pp_multihead_pfe.onnx
BackboneOnnx: ../model/cbgs_pp_multihead_backbone.onnx

PfeTrt: ../model/cbgs_pp_multihead_pfe.trt
BackboneTrt: ../model/cbgs_pp_multihead_backbone.trt

ModelConfig: ../pointpillars/cfgs/cbgs_pp_multihead.yaml

InputFile: ../test/testdata/nuscenes_10sweeps_points.txt
OutputFile: ../test/testdata/demo_boxes.txt

4. Download the test point cloud data
nuscenes_10sweeps_points.txt

Two. Compile

cd PointPillars_MultiHead_40FPS
mkdir build && cd build
cmake .. && make -j8

Note : 编译过程中有任何问题，参考第五部分问题答疑.

3. Run the test

./test/test_model

Note : 运行过程中有任何问题，参考第五部分问题答疑.

4. Visualization results

python viewer.py

insert image description here

5. Troubleshooting

Q1 . CMake Error at CMakeLists.txt:1 (cmake_minimum_required):
CMake 3.17 or higher is required. You are running version 3.16.6
A1 : The cmake version in Cmakelists.txt is too high, just lower it.

cmake_minimum_required(VERSION 3.16) # 本机是3.16版本

Q2 . The Cmakelists in test report an error, CMake Error at test/CMakeLists.txt:1 (add_subdirectory): The source directory
A2 : The test/gtest in the project belongs to the submodule of the subproject, because the project has not been downloaded completely.

git clone https://github.com/hova88/PointPillars_MultiHead_40FPS.git --recursive

Q3 . NVINFER NOT FOUND
A3 : Add the address of TensorRT

set(CMAKE_PREFIX_PATH "/usr/local/TensorRT/TensorRT-7.1.3.4/lib") # 设置前缀路径

Q4 . #error – unsupported GNU version! gcc versions later than 6 are not supported!
A4 : Check the CUDA environment and find that the version is wrong, switch to the conda environment of CUDA10.2, and recompile and execute.

Q5 . fatal error: NvInfer.h: No such file or directory
A5 : The include directory is not added, and the include path is added according to the local TensorRT situation

include_directories("/usr/local/TensorRT/TensorRT-7.1.3.4/include")

Q6 . The download address of the sample data in nuscenes_10sweeps_points.txt is incorrect.
A6 . The download address is changed to: nuscenes_10sweeps_points.txt The new download address

Q7 . Using the visual script python viewer.py prompts that the data cannot be found
A7 . Need to modify the data path, in the bootstrap.yaml file, modify InputFile and OutputFile, for example

InputFile: ../test/testdata/nuscenes_10sweeps_points.txt # 将Q6中下载好的数据放入此文件夹中
OutputFile: ../test/testdata/demo_boxes.txt

Q8 . When executing the trans_pfe.py script, an error is reported SyntaxError: Non-ASCII character '\xe8' in file trans_pfe.py on line 113, but no encoding declared; A8.
Add the following code at the beginning of the trans_pfe.py file to support Chinese character

# -*- coding:utf-8 -*-

Q9 . When executing the trans_pfe.py script, an error No module named 'pcdet' is reported.
A9 . You need to install the pcdet project first according to the installation tutorial . You can also refer to 2.1) C. steps in this article.

Q10 . When executing the trans_pfe.py script, an error is reported AttributeError: module 'spconv' has no attribute 'SparseModule'
A10 . Reference:
1. github spconv issue
2. Openpcdet project code (debugging and Q&A records)
modify pcdet/models/backbones_3d/spconv_backbone .py file and pcdet/models/backbones_3d/spconv_unet.py file

## 文件初始位置引入spconv
from spconv.pytorch import ops
from spconv.pytorch.conv import (SparseConv2d, SparseConv3d, SparseConvTranspose2d,
                         SparseConvTranspose3d, SparseInverseConv2d,
                         SparseInverseConv3d, SubMConv2d, SubMConv3d)
from spconv.pytorch.core import SparseConvTensor
from spconv.pytorch.identity import Identity
from spconv.pytorch.modules import SparseModule, SparseSequential
from spconv.pytorch.ops import ConvAlgo
from spconv.pytorch.pool import SparseMaxPool2d, SparseMaxPool3d
from spconv.pytorch.tables import AddTable, ConcatTable

class SparseBasicBlock(SparseModule)  ## 去掉spconv.

Q11 . When executing the trans_pfe.py script, the error No such file or directory: 'cfgs/dataset_configs/nuscenes_dataset.yaml'
A11 . Modify the fifth line in the tools/cfgs/nuscenes_models/cbgs_pp_multihead.yaml file, _BASE_CONFIG_ is the absolute path, modify For the following effect:

_BASE_CONFIG_: /home/cui/workspace/deepLearning/OpenPCDet/tools/cfgs/dataset_configs/nuscenes_dataset.yaml

Q12 . When executing the visual script python viewer.py, the error ModuleNotFoundError: No module named 'open3d'
A12 . Install open3d

pip install open3d

Q13 . Change the gtest file to an ordinary file, which is convenient for single-step debugging to view the variables in the file. When compiling, an error is reported: unsupported GNU version! gcc versions later than 6 are not supported A13 . When compiling, the cmake command specifies
gcc g++6.0 version

cmake -DCMAKE_C_COMPILER=/usr/bin/gcc-6 -DCMAKE_CXX_COMPILER=/usr/bin/g++-6 ..