NeRF must-read: Instant-NGP----RTX3090 single card can play NeRF

foreword

It has only been three years since NeRF developed in 2020, and the work of Follow has shown a blowout trend. I believe that in the near future, NeRF will reshape the industry of 3D reconstruction in one fell swoop, and even rebuild our 4D world (a blow at the beginning ). Although the development time of NeRF is short, there are several works that started to show the trend of essential oils in the field of my research:

• PixelNeRF - Generalization Magic
• MipNeRF ----near and distant scene reconstruction
• NeRF in the wild - background reconstruction under light transformation
• NeuS----Rebuild Surface with NeRF
• Instant-NGP - multi-scale Hash coding to achieve efficient rendering

Today's protagonist is Instant-NGP from NVlabs

overview

How to sample points x \mathbf{x} in the spaceThe position encoding of$x\; so\; that\; NeRF\; can\; accurately\; express\; multi-scale\; 3D\; space\; is\; a\; problem\; that\; the\; NeRF\; community\; has\; been\; paying\; attention\; to.$Generally speaking, a dilemma is faced: whether to perform multiple rounds of sampling on a large model to obtain high-precision rendering results, or to obtain a rough result by quickly training a small model.
So the question is, is there athatcan achieve fast training, affordability of a single card, and high accuracy?
What? Can a small town ask for a bicycle? NVlabs sent us a bike. The key is that the Instant-NGP bike can be ridden on any road, as evidenced by the picture: the

above picture lists four tasks completed by Instant-NGP:

1. 3D rendering under high-resolution images (by analogy, it should also be possible under large-scale).
2. SDF reconstruction task, hey, 3D reconstruction can also be done, wonderful~
3. Neural Radiance Caching (it is not impossible to extend this method to the Phong lighting model).
4. Efficient rendering with NeRF.
   In general, Instant-NGP has the following two characteristics: universality:Adaptivity, efficiency:Efficiency。

method

Speaking of which, the novelty of Instant-NGP is quite simple and straightforward, mainly changing the sampling point $The\; encoding\; method\; of\; x$ can be summed up in one sentence: put$x$ is decomposed into L layers of hash codes, and each layer of hash codes represents different resolutions. In each layer of hash coding,$x$ takes the weighted average of the eigenvalues ​​of the surrounding coded points as its own eigenvalues, and superimposes the eigenvalues ​​of L-layer hash codes into the model to inferene the rendering result of the point ( σ , c ) (\sigma,\mathbf{c}$(p,c)$ , followed by a series of operations of NeRF. For specific examples, you can refer to this article by Mouhu:Instant-ngp Paper Explanation
Let me explain several key formulas of the article:

1. Determine the resolution of each layer of hash features (for example: 2 equal parts to 64 equal parts), expressed as a formula: $[N_{min},N_{max}]$ , and the exact division of each layer is determined by the following formula:
   $$\begin{gathered} N_l:=\lfloor N_{min}\cdot b^l\rfloor \\ b:=\exp (\frac{ln{N}_{max}-ln{N}_{min}}{L-1}). \end{gathered}$$

2. Next is how to put a three-dimensional space $x$ is encoded into a 1-dimensional hash space:
   $$h(x)=\underset{i=1}{\overset{d}{⨁}}{x}_i{Pi}_{i}modT,$$

3. How to bilinear interpolation, please also move: instant-ngp paper explanation
   At this point the position embedding has been done, but we can also add direction embedding, image feature embedding, etc., and name it $\xi$ . Now the position embedding of the Instant-NGP version is done.