Summary
With the popularity of implicit neural representations or Neural Radiative Fields (NeRF), editing methods that interact with implicit 3D models are urgently needed for tasks such as post-processing scene reconstruction and 3D content creation. Although previous works have explored NeRF editing from different perspectives, they are limited in editing flexibility, quality, and speed, unable to provide direct editing response and instant preview. The key challenge is to conceive a locally editable neural representation that directly reflects editing instructions and updates instantly. To bridge this gap, we propose a new implicit representation interactive editing method and system, called Seal-3D, which allows users to edit NeRF models in a pixel- , and has a wide range of NeRF-like Backbone, and preview editing effects immediately. To achieve these effects, our proposed proxy function maps editing instructions to the original space of the NeRF model , and a student-teacher training strategy with local pre-training and global fine-tuning to address these challenges. The NeRF editing system is designed to showcase various editing types. Our system can achieve compelling editing effects with an interactive speed of about 1 second.
Engineering link: https://windingwind.github.io/seal-3d/
frame
Left: 3D points and view orientations of the target space after user editing are mapped to the original source space to obtain guidance ct, σt from the teacher model for student training. Right: Student training consists of two stages: fast pre-training, which updates some parameters of the network with local losses to provide instant previews, and fine-tuning with global losses.
Experimental results
The first interactive pixel-level NeRF editing tool. We design an interactive user editing method and system, Seal-3D, that enables instant (about 1 second) preview (left) via our novel pre-training strategy. High-quality editing results can be further obtained through short-time (1~2 minutes) fine-tuning. The edit result tool we implemented the edit on (right) remains on view with the rich shadow detail such as shadows on the original surface (left).
3D content editing
in conclusion
We introduce an interactive framework for pixel-level editing of neural radiance fields, supporting instant previews. Specifically, we utilize a student-teacher distillation method to provide editing guidance, and design a two-stage training strategy to achieve instant convergence of the network to obtain rough results as a preview. Unlike previous work, our method does not require any explicit proxy (e.g. grid), which improves interactivity and user-friendliness. Our method also supports preserving shading effects on edited surfaces. One limitation is that our method does not support complex view-dependent lighting effects, such as specular reflection, and cannot change scene lighting, which can be improved by introducing intrinsic composition. Furthermore, our method does not handle reconstruction failures (such as floating artifacts) of the original NeRF network.