foreword

由于毕设还有横向的需要，在导师的推荐下开始复现这篇基于VoteNet的焊接平板识别网络PanelNet，这个网络主要解决的问题是采用VoteNet网络根据三维点云对焊接过程中的焊接平板进行识别。由于实验室的显卡是RTX3080，最低只能装CUDA11和pytorch1.7，所以在搭建VoteNet和PanelNet过程中有各种bug，也比较麻烦，特开此文记录。

1. Code download

github link:

GitHub - ikh-innovation/roboweldar-weld-seam-detection: Proposal of candidate weld seams from 3D models of welding panel configurations

2. Operation steps

1. Brief introduction

VoteNet在2019年于论文“Deep Hough Voting for 3D Object Detection in Point Clouds“中提出，由于之前的方法受2D目标检测影响很大，但是三维点云和图片原始感知到的数据形式是不同的，所以该方法回归第一性原则解决问题。PanelNet对VoteNet进行改进，用于焊接平板的识别。

Paper link

2. Environment configuration

2.1GPU environment configuration

Anaconda environment configuration (five) - anaconda installation and python environment configuration under Linux - Programmer Sought

Nvidia Driver

ubuntu18.04 install mx250 graphics card driver (super detailed) - Gray letter network (software development blog aggregation)

I would like to mention more here. For the installation of ubuntu20.04 RTX 3080 graphics card, it is recommended to select "Use NVIDIA driver metapackage from nvidia-driver-515 (proprietary)" in "Additional Drivers" in "Software and Updates" to install. It is simple and quick to install from the official website. The driver may display a black screen.

CUDA (choose CUDA11.0 here)

CUDA Toolkit 11.0 Update 2 Downloads | NVIDIA Developer

Install cuDNN8.0.5:

Uninstall cuda and cudnn on ubuntu18.04, reinstall cuda and cudnn version 10.2

The detailed installation and configuration process of CUDA and cuDNN under Ubuntu20.04, the pro-test is available (graphic)_What is this nickname called? Blog-CSDN Blog_ubuntu cuda installation

2.2 VoteNet virtual environment configuration

conda create -n ikh_weldrobot python=3.6
conda activate ikh_weldrobot

pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

The point is to install with pip, not conda! Because installing with conda will install ninja and cannot be uninstalled.

The computing power of RTX3080 is too high and does not match cuda11.0. It needs to be modified:

sudo gedit ~/.bashrc

Add the following command:

export TORCH_CUDA_ARCH_LIST="8.0"

VoteNet是PointNet类方法的延续,简单高效,直接处理点云，所以为了实现VoteNet需要首先实现PointNet2。

becausePointNet2原始搭建环境在低版本pytorch，因此在pytorch1.7复现需要修改一些函数的用法，参照下文：

pointnet2.pytorch (pointnet++) run python setup.py install related error solutions:_Zhang Wuji's Blog-CSDN Blog

Enter the pointnet2 folder under the votenet folder and run the following command:

python setup.py install
pip install -r requirements
pip install matplotlib opencv-python plyfile 'trimesh>=2.35.39,<2.35.40' 'networkx>=2.2,<2.3'
下面这个应该不需要
# pip install numpy>=1.18.5 open3d>=0.10.0.0 trimesh>=2.35.39 matplotlib>=2.2.5 plyfile>=0.7.2 scipy>=1.4.1 tensorflow==2.2.0 opencv-python>=4.3.0.36 setuptools>=49.2.0 pillow>=6.2.2

So far,VoteNet的相关依赖已经基本配置完成，运行下列代码可以测试配置是否成功。

cd ..
python models/votenet.py

The first few lines of successful output should look like this:

Dataset has not been prepared. Use a random sample.
sa1_inds tensor([[    0, 18378,  2760,  ..., 10087,  9941,  6935]], device='cuda:0',
       dtype=torch.int32)
sa1_xyz tensor([[[0.5433, 0.9419, 0.0351],
         [0.0507, 0.0205, 0.9666],
         [0.9723, 0.5971, 0.9659],
         ...,
         [0.2278, 0.0654, 0.2796],
         [0.2423, 0.6431, 0.5632],
         [0.7706, 0.7663, 0.7406]]], device='cuda:0')

Download the pre-trained model and sample point cloud in the votenet folder (/votenet/demo_files), and run the demo:

python demo.py

An error will be reported here

Traceback (most recent call last):
  File "demo.py", line 100, in <module>
    pred_map_cls = parse_predictions(end_points, eval_config_dict)
  File "/home/lin/welding_robot/ikh-innovation/roboweldar-weld-seam-detection-master/votenet/models/ap_helper.py", line 103, in parse_predictions
    if len(pc_in_box) < config_dict['min_points_2b_empty']:
KeyError: 'min_points_2b_empty'

After looking through the code, I found that the dictionary variable config_dict did not define the key min_points_2b_empty at all, so I looked through the codes of other votenets on github. The solution is as follows:

if len(pc_in_box) < config_dict['min_points_2b_empty']:
改成:
if len(pc_in_box) < 5:

In this way, it can be run. The original author may not have fixed this bug, although I am too lazy to leave issues on github. Forgive my poor English.

Point cloud segmentation results are stored indemo_files/sunrgbd_results文件夹下，新建一个python文件visualization.py用Open3d进行可视化，写入如下代码（记得修改路径，最好是绝对路径）：

import open3d as o3d
import numpy as np
pcd=o3d.io.read_point_cloud("/path/to/project/demo_files/sunrgbd_results/0000_pc.ply",format='ply')
ppoints=np.asarray(pcd.points)
pcolors=np.asarray(pcd.colors) 
o3d.visualization.draw_geometries([pcd])

After running, the following results are obtained:

2.3PanelNet test

Let's start testing the PanelNet effect, and change a bug before testing.

Open votenet/panelnet/panel_data.pythe file:

mesh, labelboxes = generate_welding_area(panels_num=np.random.randint(2, 5), hpanels_num=np.random.randint(1, 3))
改成：
pcd, mesh, labelboxes = generate_welding_area(panels_num=np.random.randint(2, 5), hpanels_num=np.random.randint(1, 3))

Run the following code in the votenet folder to segment the 3D point cloud of the welding environment:

python inference_old.py --checkpoint_dir=weights_checkpoint
# inference.py 有个变量未定义，可以自己手动输入，不得不说这个作者心很大。

The segmentation results are generated in the votenet/panelnet_inference_results folder, usingvisualization.py文件修改文件路径进行可视化（目标结果为0000_pc.ply）。

原图：

Segmentation result:

So far, the reproduction of PanelNet is basically completed, and the further operation is to download the training set and make the training set by yourself for training, which is skipped here, after all, the collection of data sets is a bit troublesome.

Summarize

This paper provides a basic introduction to the Welded Flat Panel Recognition NetworkPanelNet的配置，主要包括VoteNet和PointNet的配置，实现了在复杂环境中分割焊接平板点云。

RTX3080 reproduces the VoteNet-based welding panel recognition network PanelNet