[3D Reconstruction] [Deep Learning] [Dataset] Make your own NeRF (LLFF format) dataset based on COLMAP

[3D Reconstruction] [Deep Learning] [Dataset] Make your own NeRF (LLFF format) dataset based on COLMAP

Tips: Recently, I started to conduct research on [3D reconstruction], record the relevant knowledge points, and share the solutions to the problems encountered in the study.

---

---

foreword

The LLFF format is one of the data set formats used for NeRF network model training. This article shows the complete process from the production of the LLFF format data set to the start of model training based on the COLMAP software. NeRF (Neural Radiation Field) performs three-dimensional implicit modeling of the scene by inputting two-dimensional pictures and camera poses from different perspectives of the same scene, and realizes the synthesis of scene pictures under any new perspective through the voxel rendering equation.

---

Download and install colmap software

Download the COLMAP software [ download address ], this article uses the CUDA version under Windows:

after decompression, double-click to open COLMAP.bat, the following interface appears:

the software installation is successful.

---

Download llff source code

# 可能血药科学上网从Github上直接git下载,博主下载到nerf-pytorch下
git clone https://github.com/Fyusion/LLFF.git
# 激活虚拟环境
conda activate XXX
# eg: conda activate nerf_mvs
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple scikit-image
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple imageio

---

Collect pictures and use colmap to get the camera pose

capture image

The pictures needed to make the data set in this article are obtained by drawing frames after shooting videos with mobile phones

# 激活虚拟环境
conda activate XXX
# eg: conda activate nerf_mvs
# 安装opencv
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple opencv-python

Create a new file Video_to_frame.py, and execute the following code to complete the frame extraction (customize the frame extraction interval).

import os
import cv2
def extract_images(video_path, output_folder):
    # 获取视频文件名
    video_name = os.path.splitext(os.path.basename(video_path))[0]
    # 新建文件夹
    output_path = os.path.join(output_folder, video_name)
    if not os.path.exists(output_path):
        os.makedirs(output_path)
    # 打开视频
    cap = cv2.VideoCapture(video_path)
    # 设置帧间隔
    frame_interval = int(2)
    # 逐帧提取并保存满足间隔要求的帧
    count = 0
    while cap.isOpened():
        ret, frame = cap.read()
        if ret:
            print(frame_interval)
            if count % frame_interval == 0:
                image_name = os.path.join(output_path, f"{
      
      video_name}_{
      
      count//frame_interval}.jpg")
                cv2.imwrite(image_name, frame)
            count += 1
        else:
            break
    cap.release()

if __name__ == '__main__':
    video_path = 'C:/Users/ruler9702/Desktop/test_video/test.mp4'  # 视频文件路径
    output_folder = 'C:/Users/ruler9702/Desktop/test_frame'  # 输出文件夹路径
    extract_images(video_path, output_folder)

pose calculation

Create a project: Click File -> New project to create a new project.
1. Click New, select a folder (the blogger and the test picture are placed in the same directory), set the project name to create a new project data file.

2. Click Select, select the folder where the image is located just now, and click Save.

Create a new project and configure it.

Feature extraction and matching
1. Image feature extraction, click Processing -> Feature extraction, Camera model select SIMPLE_PINHOLE, and other configurations can use the default configuration. After clicking Extract, the image feature extraction will start automatically.

Close the window after feature extraction is complete.

2. Image feature matching, click Processing -> Feature matching, and use the default configuration to directly click Run to perform feature matching.

Close the window after the feature matching is complete.

In the Log column on the right, you can view the progress of feature extraction and matching. Please make sure that there is no error reported during the process.

Sparse reconstruction
Click Reconstruction -> Start reconstruction to rebuild, you can see the reconstruction process in the window, this process may last for a while.

After the reconstruction is complete, the following figure is obtained. From the Images and Points in the lower right corner, you can roughly judge whether the reconstruction is successful.

Note: The difference in the number of matching poses and pictures will lead to an error in the subsequent steps. The blogger will explain the solution to this problem in detail below, and continue to follow the steps for the time being.

Save the pose and sparse points
Click File -> Export model to export the model, create a new /sparse/0/ folder in the directory where the folder where the image is saved, and select this folder to import the model into this directory.

The following files are obtained in the /sparse/0/ directory, and the image pose is successfully saved.

---

Convert to llff data format

Enter the LLFF directory, open the imgs2poses.py file, add the following content, default='the absolute path of the directory where sparse is located', and modify the parameter 'scenedir' to '–scenedir'.

Running the imgs2poses.py code

# 注意要在imgs2poses.py所在目录执行命令
python imgs2poses.py
# 或者附带imgs2poses.py的路径
python XXXX\imgs2poses.py
# eg: python LLFF\imgs2poses.py

If you do not add default='', and modify the parameter 'scenedir' to '–scenedir',

# 在运行imgs2poses.py代码时,即使有默认值也必须传入路径(与scenedir参数有关)
python imgs2poses.py "XXXX/XXXX/"

The following problem arises:

This problem is caused by the different number of matching poses and pictures that the blogger said before, and the blogger solves this problem here. By adding the following code around line 32 of the LLFF/llff/poses/pose_utils.py file:

    #---------输出匹配到位姿的图片名---------
    for i in np.argsort(names):
       print(names[i],end=' ')
    #---------输出匹配到位姿的图片名---------

The position of adding the code is shown in the figure below:

all pictures matching the pose are displayed.

Enter the directory where the image is saved, delete the image that does not match the pose, and then perform the "pose calculation" step again.

After solving the problem, the execution result is shown in the figure below:

the npy file related to the image pose is generated, and the format conversion step is completed.

---

Migrate working folders and set configuration files

Migrate the working folder completely to the /nerf-pytorch/data/nerf_llff_data/ directory of the nerf code.

Note: The folder where the pictures are saved should be renamed to images, otherwise an error will be reported

Copy the fern.txt file in the /nerf-pytorch/configs directory (because fern is also a data set in LLFF format), rename it to the name of your own test data, and modify the following content: All preparations have been completed

.

---

Summarize

As simple and detailed as possible, it introduces the production process of the f-LLFF format dataset and solves the possible problems in the production process.