NeRF makes its own dataset

Table of contents

Preparation

colmap gets the camera pose of the picture

 Convert the acquired pose and other data into llff format

 Train your own nerf

---

Based on COLMAP software, the complete process from the production of LLFF format data set to the start of model training is demonstrated.

Preparation

Install colmap first to obtain the camera pose of the picture

colmap installation package:

CUDA version and CUDA-less version

Link: https://pan.baidu.com/s/1GUJBgmYcr7CMDWvJLKo3iA 
Extraction code: lxwj

After downloading, click colmap.bat to open it

NeRF mainly uses two types of data sets, synthetic data sets (synthetic) and real data sets (real images), which are real data sets created in llff format.

Prepare pictures, you can shoot continuously, or you can shoot videos and then generate pictures through the following frame extraction script.

import os
import cv2
def extract_images(video_path, output_folder):
    # 获取视频文件名
    video_name = os.path.splitext(os.path.basename(video_path))[0]
    # 新建文件夹
    output_path = os.path.join(output_folder, video_name)
    if not os.path.exists(output_path):
        os.makedirs(output_path)
    # 打开视频
    cap = cv2.VideoCapture(video_path)
    # 设置帧间隔
    frame_interval = int(2)
    # 逐帧提取并保存满足间隔要求的帧
    count = 0
    while cap.isOpened():
        ret, frame = cap.read()
        if ret:
            print(frame_interval)
            if count % frame_interval == 0:
                image_name = os.path.join(output_path, f"{video_name}_{count//frame_interval}.jpg")
                cv2.imwrite(image_name, frame)
            count += 1
        else:
            break
    cap.release()

if __name__ == '__main__':
    video_path = 'C:/Users/ruler9702/Desktop/test_video/test.mp4'  # 视频文件路径
    output_folder = 'C:/Users/ruler9702/Desktop/test_frame'  # 输出文件夹路径
    extract_images(video_path, output_folder)

Create project: Click File -> New project to create a new project.

Form the pictures into a folder. I put the pictures into the plant folder, and create a new sparse folder under the project folder, and then create a 0 folder under sparse.

colmap gets the camera pose of the picture

open colmap

Click File, select New Project, continue to click new, manually enter the file name plant2 in the previously created folder, and click Save:

 Then click select to select your own picture folder

Click Save 

Next, click Processing on the right side of File, select Feature extraction, and the following interface will appear. You only need to configure the first option, and then click Extract below.

Close the window after feature extraction is completed.

To match image features, click Processing -> Feature matching, use the default configuration and click Run directly to perform feature matching.

Close the window after feature matching is completed

Click Reconstruction -> Start reconstruction to reconstruct. You can see the reconstruction process in the window. This process may last for a while.

After the reconstruction is completed, you will get the following picture. You can roughly judge whether the reconstruction is successful through the Images and Points in the lower right corner.

 

 Save poses and sparse points

Click File -> Export model to export the model, create a new /sparse/0/ folder in the directory where the folder where the image is saved is located, select the folder to import the model into the directory, and get four files.

 Convert the acquired pose and other data into llff format

 Next, convert the acquired pose and other data into llff format, you need to download the llff script, (Address: GitHub - Fyusion/LLFF: Code release for Local Light Field Fusion at SIGGRAPH 2019)

 In the script, enter python imgs2poses in termina and add the address of your project

For example:

Then you will get the poses_bounds.npy file

 Train your own nerf

Migrate the previous folder completely to the /nerf-pytorch/data/nerf_llff_data/ directory of the nerf code. Also change the previous picture folder to images

 

Copy the fern.txt file in the /nerf-pytorch/configs directory (because fern is also a data set in LLFF format), rename it to the name of your own test data, and modify the following content:

 All preparations have been completed and training is ready

If you encounter during training

Mismatch between imgs 0 and poses 55 !!!!
Traceback (most recent call last):
File "run_nerf.py", line 878, in <module>
train()
File "run_nerf.py", line 544, in train
spherify=args.spherify)
File "C:\Users\HP\Desktop\nerf-pytorch-master\load_llff.py", line 246, in load_llff_data
poses, bds, imgs = _load_data(basedir, factor=factor) # factor=8 downsamples original imgs by 8x
TypeError: cannot unpack non-iterable NoneType object

It may be because the function is incompatible. The solution is to create a new folder called images_8 under the folder and put the eight-fold downsampled image here.
 

import os
from PIL import Image

# 输入文件夹路径和输出文件夹路径
input_folder = "C:\\Users\\86130\\Desktop\\COLMAP-3.8-windows-cuda\\COLMAP-3.8-windows-cuda\\project1\\rice"
output_folder = "C:\\Users\\86130\\Desktop\\COLMAP-3.8-windows-cuda\\COLMAP-3.8-windows-cuda\\project1\\images_8"

# 创建输出文件夹
if not os.path.exists(output_folder):
    os.makedirs(output_folder)

# 获取输入文件夹中所有图片文件的列表
image_files = [f for f in os.listdir(input_folder) if os.path.isfile(os.path.join(input_folder, f))]

# 循环处理每个图片文件
for image_file in image_files:
    # 打开图像文件
    image_path = os.path.join(input_folder, image_file)
    image = Image.open(image_path)

    # 获取原始图像的宽度和高度
    width, height = image.size

    # 计算新图像的宽度和高度（原始图像的1/8）
    new_width = width // 8
    new_height = height // 8

    # 使用resize()函数对图像进行下采样
    downscaled_image = image.resize((new_width, new_height), Image.ANTIALIAS)

    # 构造输出文件路径
    output_path = os.path.join(output_folder, image_file)

    # 保存下采样后的图像
    downscaled_image.save(output_path)

    print(f"Downsampling complete: {image_file}")

print("All images downscaled successfully.")