[3D Reconstruction] [Deep Learning] [Dataset] Make your own NeuS (DTU format) data set based on COLMAP
Tip: I have recently started to conduct research on [3D reconstruction], record relevant knowledge points, and share methods that have been solved for problems encountered in learning.
Article directory
Preface
The DTU format is one of the data set formats used for NeuS network model training. This article uses the COLMAP software to show the complete process from the production of the DTU format data set to the start of model training. NeuS performs three-dimensional implicit modeling of the scene by inputting two-dimensional pictures and camera poses from different perspectives of the same scene, and uses a new first-order approximate unbiased formula, so that it can perform updated operations even without mask supervision. Accurate surface reconstruction.
Download and install colmap software
Download the COLMAP software [ Download address ]. This article uses the CUDA version under Windows:
After decompressing, double-click to open COLMAP.bat, and the following interface will appear:
The software is installed successfully.
Download LLFF source code
# 可能需要科学上网从Github上直接git下载,博主下载到neus下
git clone https://github.com/Fyusion/LLFF.git
# 激活虚拟环境
conda activate XXX
# eg: activate neus
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple scikit-image
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple imageio
Collect pictures and use colmap to obtain the camera pose
Collect images
The pictures needed to create the data set in this article are obtained by extracting frames after shooting videos with a mobile phone.
# 激活虚拟环境
conda activate XXX
# eg: activate neus
# 安装opencv
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple opencv-python
Create a new file Video_to_frame.py and execute the following code to complete frame extraction (customize the frame extraction interval).
import os
import cv2
def extract_images(video_path, output_folder):
# 获取视频文件名
video_name = os.path.splitext(os.path.basename(video_path))[0]
# 新建文件夹
output_path = os.path.join(output_folder, video_name)
if not os.path.exists(output_path):
os.makedirs(output_path)
# 打开视频
cap = cv2.VideoCapture(video_path)
# 设置帧间隔
frame_interval = int(2)
# 逐帧提取并保存满足间隔要求的帧
count = 0
while cap.isOpened():
ret, frame = cap.read()
if ret:
print(frame_interval)
if count % frame_interval == 0:
image_name = os.path.join(output_path, f"{
video_name}_{
count//frame_interval}.jpg")
cv2.imwrite(image_name, frame)
count += 1
else:
break
cap.release()
if __name__ == '__main__':
video_path = 'C:/Users/XXX/Desktop/test_video/test.mp4' # 视频文件路径
output_folder = 'C:/Users/XXX/Desktop/test_frame' # 输出文件夹路径
extract_images(video_path, output_folder)
pose calculation
Create a project: Click File -> New project to create a new project.
1. Click New, select a folder (the blogger and test pictures are placed in the same directory), and set the project name to create a new project data file.
2. Click Select, select the folder where the image is located, and click Save.
A new engineering project is created and configured.
Feature extraction and matching
1. Image feature extraction, click Processing -> Feature extraction, select SIMPLE_PINHOLE for Camera model, and use the default configuration for other configurations. After clicking Extract, image feature extraction will begin automatically.
Close the window after feature extraction is completed.
2. To match image features, click Processing -> Feature matching, use the default configuration and click Run directly to perform feature matching.
Close the window after feature matching is completed.
You can check the progress of feature extraction and matching in the Log column on the right. Please make sure there are no Erro errors during the process.
Sparse reconstruction
Click Reconstruction -> Start reconstruction to reconstruct. You can see the reconstruction process in the window. This process may last for a while.
After the reconstruction is completed, you will get the following picture. You can roughly judge whether the reconstruction is successful through the Images and Points in the lower right corner.
Note: Different matching poses and the number of pictures will cause errors in subsequent steps. The blogger will explain in detail how to solve this problem below, so continue to follow the steps for now.
Save the pose and sparse points
. Click File -> Export model to export the model. Create a new /sparse/0/ folder in the directory where the image folder is located. Select the folder to import the model into the directory.
Obtain the following file in the /sparse/0/ directory and save the image pose successfully.
Convert to LLFF data format
Enter the LLFF directory, open the imgs2poses.py file, add the following content, default='which is the absolute path of the directory where sparse is located', and change the parameter 'scenedir' to '- -scenedir'.
The name of the folder where the photos are stored must be images, otherwise an error will occur. The blogger here is test, so it needs to be changed to images.
Running imgs2poses.py code
# 注意要在imgs2poses.py所在目录执行命令
python imgs2poses.py
# 或者附带imgs2poses.py的路径
python XXXX\imgs2poses.py
# eg: python LLFF\imgs2poses.py
If default='' is not added.
# 在运行imgs2poses.py代码时,即使有默认值也必须传入路径(与scenedir参数有关)
python imgs2poses.py "XXXX/XXXX/"
The following problem occurs:
This problem is caused by the different matching poses and number of pictures mentioned by the blogger before. The blogger solves this problem here. By adding the following code around line 32 of the LLFF/llff/poses/pose_utils.py file:
#---------输出匹配到位姿的图片名---------
for i in np.argsort(names):
print(names[i],end=' ')
#---------输出匹配到位姿的图片名---------
Add the code location as shown below:
All images matching the pose are displayed.
Enter the directory where the images are saved, delete the images that do not match the pose, and then perform the "pose calculation" step again.
After solving the problem, the execution result is as shown in the figure below:
the npy file related to the image pose is generated, and the format conversion step is completed.
Convert to DTU data format
The blogger provided the code for converting LLFF data format to DTU data format .
Unzip the code to the appropriate folder. Here the blogger unzipped it to the NeuS path.
Execute the instruction to convert the LLFF data format to the DTU data format.
# XXX表示sparse所在的路径,不包含sparse。
python tools/preprocess_llff.py XXXX
#eg: python tools/preprocess_llff.py C:\Users\AYU\Desktop\test_frame
All preparations have been completed.
Summarize
Introduce the production process of DTU format data set as simply and in detail as possible and solve possible problems during the production process.