BEVFusion (Peking University & Alibaba) environment building tutorial

BEVFusion environment setup

Thesis title: BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework

Paper address: https://arxiv.org/pdf/2205.13790.pdf

代码地址：ADLab-AutoDrive/BEVFusion: Offical PyTorch implementation of “BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework” (github.com)

Preface: This is a new paper published this year. When I read it for the first time, the code was not yet open source. I found out a few days ago that it is open source, and the framework is very clear. I can familiarize myself with the fusion solution based on this work, and also Can be improved slightly.

Author's environment:

sys.platform: linux
Python: 3.8.3 | packaged by conda-forge | (default, Jun 1 2020, 17:43:00) [GCC 7.5.0]
CUDA available: True
GPU 0,1: NVIDIA GeForce RTX 3090
CUDA_HOME: /usr
NVCC: Build cuda_11.5.r11.5/compiler.30672275_0
GCC: gcc (Ubuntu 7.5.0-6ubuntu2) 7.5.0
PyTorch: 1.8.0+cu111
PyTorch compiling details: PyTorch built with:

GCC 7.3

C++ Version: 201402

Intel® Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel® 64 architecture applications

Intel® MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683)

OpenMP 201511 (a.k.a. OpenMP 4.5)

NNPACK is enabled

CPU capability usage: AVX2

CUDA Runtime 11.1

NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86

CuDNN 8.0.5

Magma 2.5.2

Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++,CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1,PERF_WITH_AVX512=1, TORCH_VERSION=1.8.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

TorchVision: 0.9.0+cu111
OpenCV: 4.6.0
MMCV: 1.4.0
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: 11.1
MMDetection: 2.11.0
MMDetection3D: 0.11.0+9d3e162

In human terms: torch1.8.0+cuda111, the graphics card is 3090, and the rest of the environment is configured according to the official documents

1. Installation

1. Clone the BEVFusion code

git clone https://github.com/ADLab-AutoDrive/BEVFusion.git

2. Create a new virtual environment

conda create -n bevfusion python=3.8.3
conda activate bevfusion #激活

3. Install torch according to the cuda version

Try to choose 1.7 or 1.8. The CUDA version should correspond to
November 22, 2022. Update: Try to choose 1.7. Bugs in 1.8 cause some problems in training. Update on
December 2, 2022: I have been too busy recently. I found that the torch of 1.7 is not available a few days ago. It is suitable for use in this project. The specific reason is related to the 3090 graphics card and the lack of related libraries (3090 supports cuda11.1 at least, and torch1.7 supports cuda11.0 at most)

pip install torch=1.8.0+cu111 torchvision=0.9.0+cu111 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html

4. Install mmcv-full

This depends on your own torch and cuda. In addition, the BEVFusion official document recommends version 1.4.0.

Find your own suitable version from the following URL

https://download.openmmlab.com/mmcv/dist/{cu_version}/{torch_version}/index.html

Among them, cu_version represents the cuda version and torch_version represents the torch version.

For example, mine is torch1.8.0+cuda111, so my address is:

https://download.openmmlab.com/mmcv/dist/cu111/torch1.8.0/index.html

Because the environment created is python3.8, so choose cp38. Don’t forget that the mmcv-full version is 1.4.0. Download the whl file and transfer it to the server, and then pip the whl file (I will not go into details about pip installation of whl, Baidu )

Insert image description here

5. Install MMDetection

The following is the environment officially recommended by BEVFusion

python=3.8.3
mmdet=2.11.0 (please install mmdet in mmdetection-2.11.0)
mmcv=1.4.0
mmdet3d=0.11.0
numpy=1.19.2
torch=1.7.0
torchvision=0.8.0

Note please install mmdet in mmdetection-2.11.0 . The author has already placed this folder in the warehouse. We enter this folder directly.
Insert image description here

Then execute:

pip install -r requirements/build.txt
pip install -v -e .  # or "python setup.py develop"(推荐执行后者)

6. Install MMDetection3D

Return to the root directory of BEVFusion and execute the following command

pip install -v -e .  # or "python setup.py develop"（推荐执行后者）

The configuration is now complete

The following is my environment

Package Version Editable project location

absl-py 1.3.0
addict 2.4.0
anyio 3.6.2
argon2-cffi 21.3.0
argon2-cffi-bindings 21.2.0
asttokens 2.1.0
attrs 22.1.0
backcall 0.2.0
beautifulsoup4 4.11.1
black 22.10.0
bleach 5.0.1
cachetools 5.2.0
certifi 2022.9.24
cffi 1.15.1
charset-normalizer 2.1.1
click 8.1.3
contourpy 1.0.6
cycler 0.11.0
Cython 0.29.32
debugpy 1.6.3
decorator 5.1.1
defusedxml 0.7.1
depthwise-conv2d-implicit-gemm 0.0.0
descartes 1.1.0
entrypoints 0.4
exceptiongroup 1.0.4
executing 1.2.0
fastjsonschema 2.16.2
filelock 3.8.0
fire 0.4.0
flake8 5.0.4
fonttools 4.38.0
google-auth 2.14.1
google-auth-oauthlib 0.4.6
grpcio 1.50.0
h5py 3.7.0
huggingface-hub 0.11.0
idna 3.4
imageio 2.22.4
importlib-metadata 5.0.0
importlib-resources 5.10.0
iniconfig 1.1.1
ipykernel 6.17.1
ipython 8.6.0
ipython-genutils 0.2.0
ipywidgets 8.0.2
jedi 0.18.1
Jinja2 3.1.2
joblib 1.2.0
jsonschema 4.17.0
jupyter 1.0.0
jupyter_client 7.4.6
jupyter-console 6.4.4
jupyter_core 5.0.0
jupyter-server 1.23.2
jupyterlab-pygments 0.2.2
jupyterlab-widgets 3.0.3
kiwisolver 1.4.4
llvmlite 0.31.0
loguru 0.6.0
lyft-dataset-sdk 0.0.8
Markdown 3.4.1
MarkupSafe 2.1.1
matplotlib 3.6.2
matplotlib-inline 0.1.6
mccabe 0.7.0
mistune 2.0.4
mmcls 0.24.1
mmcv-full 1.4.0
mmdet 2.11.0 /home/wistful/work/my_bevfusion/mmdetection-2.11.0
mmdet3d 0.11.0 /home/wistful/work/my_bevfusion
mmpycocotools 12.0.3
msgpack 1.0.4
msgpack-numpy 0.4.8
multimethod 1.9
mypy-extensions 0.4.3
nbclassic 0.4.8
nbclient 0.7.0
nbconvert 7.2.5
nbformat 5.7.0
nest-asyncio 1.5.6
networkx 2.2
ninja 1.11.1
notebook 6.5.2
notebook_shim 0.2.2
numba 0.48.0
numpy 1.23.4
nuscenes-devkit 1.1.9
oauthlib 3.2.2
opencv-python 4.6.0.66
packaging 21.3
pandas 1.4.4
pandocfilters 1.5.0
parso 0.8.3
pathspec 0.10.2
pexpect 4.8.0
pickleshare 0.7.5
Pillow 9.3.0
pip 22.3.1
pkgutil_resolve_name 1.3.10
platformdirs 2.5.4
plotly 5.11.0
pluggy 1.0.0
plyfile 0.7.4
prettytable 3.5.0
prometheus-client 0.15.0
prompt-toolkit 3.0.32
protobuf 3.20.3
psutil 5.9.4
ptyprocess 0.7.0
pure-eval 0.2.2
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycocotools 2.0.6
pycodestyle 2.9.1
pycparser 2.21
pyflakes 2.5.0
Pygments 2.13.0
pyparsing 3.0.9
pyquaternion 0.9.9
pyrsistent 0.19.2
pytest 7.2.0
python-dateutil 2.8.2
pytz 2022.6
PyWavelets 1.4.1
PyYAML 6.0
pyzmq 24.0.1
qtconsole 5.4.0
QtPy 2.3.0
requests 2.28.1
requests-oauthlib 1.3.1
rsa 4.9
scikit-image 0.19.3
scikit-learn 1.1.3
scipy 1.9.3
Send2Trash 1.8.0
setuptools 65.5.1
Shapely 1.8.5.post1
six 1.16.0
sniffio 1.3.0
soupsieve 2.3.2.post1
stack-data 0.6.1
tabulate 0.9.0
tenacity 8.1.0
tensorboard 2.11.0
tensorboard-data-server 0.6.1
tensorboard-plugin-wit 1.8.1
tensorpack 0.11
termcolor 2.1.0
terminado 0.17.0
terminaltables 3.1.10
threadpoolctl 3.1.0
tifffile 2022.10.10
timm 0.6.11
tinycss2 1.2.1
toml 0.10.2
tomli 2.0.1
torch 1.8.0+cu111
torchaudio 0.8.0
torchpack 0.3.1
torchvision 0.9.0+cu111
tornado 6.2
tqdm 4.64.1
traitlets 5.5.0
trimesh 2.35.39
typing_extensions 4.4.0
urllib3 1.26.12
wcwidth 0.2.5
webencodings 0.5.1
websocket-client 1.4.2
tool 2.2.2
wheel 0.38.4
widgetsnbextension 4.0.3
yapf 0.32.0
zipp 3.10.0

sys.platform: linux
Python: 3.8.3 | packaged by conda-forge | (default, Jun 1 2020, 17:43:00) [GCC 7.5.0]
CUDA available: True
GPU 0,1: NVIDIA GeForce RTX 3090
CUDA_HOME: /usr/local/cuda
NVCC: Build cuda_11.1.TC455_06.29069683_0
GCC: gcc (Ubuntu 7.5.0-6ubuntu2) 7.5.0
PyTorch: 1.8.0+cu111
PyTorch compiling details: PyTorch built with:

GCC 7.3

C++ Version: 201402

Intel® Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel® 64 architecture applications

Intel® MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683)

OpenMP 201511 (a.k.a. OpenMP 4.5)

NNPACK is enabled

CPU capability usage: AVX2

CUDA Runtime 11.1

NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86

CuDNN 8.0.5

Magma 2.5.2

Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++,CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1,PERF_WITH_AVX512=1, TORCH_VERSION=1.8.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

TorchVision: 0.9.0+cu111
OpenCV: 4.6.0
MMCV: 1.4.0
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: 11.1
MMDetection: 2.20.0
MMDetection3D: 0.11.0+9d3e162

2. Organize Data Sets

Note that generally speaking, when working with the mmdetection3d framework, the data set must be organized under mmdetection3d/data instead of creating a data directory in the root directory of the project. Therefore, the data set must be organized under BEVFusion/mmdetection3d/data

It is recommended to symlink the dataset root to $MMDETECTION3D/data. If your folder structure is different from the following, you may need to change the corresponding paths in config files.

I won’t go into details about the organization of the data set. It should be organized according to the official organization form.

mmdetection3d
├── mmdet3d
├── tools
├── configs
├── data
│ ├── nuscenes
│ │ ├── maps
│ │ ├── samples
│ │ ├── sweeps
│ │ ├── v1.0-test
| | ├── v1.0-trainval
│ ├── kitti
│ │ ├── ImageSets
│ │ ├── testing
│ │ │ ├── calib
│ │ │ ├── image_2
│ │ │ ├── velodyne
│ │ ├── training
│ │ │ ├── calib
│ │ │ ├── image_2
│ │ │ ├── label_2
│ │ │ ├── velodyne
│ ├── waymo
│ │ ├── waymo_format
│ │ │ ├── training
│ │ │ ├── validation
│ │ │ ├── testing
│ │ │ ├── gt.bin
│ │ ├── kitti_format
│ │ │ ├── ImageSets
│ ├── lyft
│ │ ├── v1.01-train
│ │ │ ├── v1.01-train (train_data)
│ │ │ ├── lidar (train_lidar)
│ │ │ ├── images (train_images)
│ │ │ ├── maps (train_maps)
│ │ ├── v1.01-test
│ │ │ ├── v1.01-test (test_data)
│ │ │ ├── lidar (test_lidar)
│ │ │ ├── images (test_images)
│ │ │ ├── maps (test_maps)
│ │ ├── train.txt
│ │ ├── val.txt
│ │ ├── test.txt
│ │ ├── sample_submission.csv
│ ├── scannet
│ │ ├── meta_data
│ │ ├── scans
│ │ ├── batch_load_scannet_data.py
│ │ ├── load_scannet_data.py
│ │ ├── scannet_utils.py
│ │ ├── README.md
│ ├── sunrgbd
│ │ ├── OFFICIAL_SUNRGBD
│ │ ├── matlab
│ │ ├── sunrgbd_data.py
│ │ ├── sunrgbd_utils.py
│ │ ├── README.md

(You don’t need all of the above data sets, just which organization you need)

Then use the BEVFusion/tools/create_data.py script to organize the files. Here is a screenshot of the successful organization of the nuScenes data set.

Insert image description here

3. Training and testing

1. Distributed training

# first train camera stream
./tools/dist_train.sh configs/bevfusion/cam_stream/bevf_pp_4x8_2x_nusc_cam.py 8
# then train LiDAR stream
./tools/dist_train.sh configs/bevfusion/lidar_stream/hv_pointpillars_secfpn_sbn-all_4x8_2x_nus-3d.py 8
# then train BEVFusion
./tools/dist_train.sh configs/bevfusion/bevf_pp_2x8_1x_nusc.py 8

./tools/dist_train.sh configs/bevfusion/cam_stream/bevf_pp_4x8_2x_nusc_cam.py 8Among them, 8 represents the number of GPUs

When using sh files for training and testing, you may encounter the problem of no permissions. chmod 777 tools/dist_train.shJust use to change the permissions.
Insert image description here

Among them, I will not describe the configuration file. You can use the official simple test. If you need to use your own configuration file, please refer to the mmdetection3d official documentation.

It is worth noting that when I used distributed and non-distributed methods for testing, the non-distributed method ran normally, but the distributed method still prompted that the data set could not be found. I searched for information and found out that it was due to the working path. I print(os.getcwd())printed it and found that when using non-distributed training, the working path is /home/wistful/work/my_bevfusion/mmdetection3d/toolsunder BEVFusion/mmdetection3d/tools, and when using distributed training, the working path /home/wistful/work/my_bevfusionis under the BEVFusion root directory. So I os.chdir('/home/wistful/work/my_bevfusion/mmdetection3d/tools')changed the working path using
Insert image description here

2. Non-distributed training

Just set each parameter directly in pycharm and train, no more details.

3. Test

./tools/dist_test.sh configs/bevfusion/bevf_pp_2x8_1x_nusc.py ./work_dirs/bevfusion_pp.pth 8 --eval bbox
 # 前者是配置文件，后者是GPU个数

# 如需可视化
./tools/dist_test.sh configs/bevfusion/bevf_pp_2x8_1x_nusc.py ./work_dirs/bevfusion_pp.pth 8 --eval bbox --show-dir YourVisualResultsDir --show --eval bbox #根据test.py的参数看一下

The visualization results will be saved in the directory you specify, as shown below.
Insert image description here
Import them to CloudCamare for viewing.