[Read the paper] [Quick reading] Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields

Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields

0. Abstract

Problem introduction：

When training or testing images to observe scene content at different resolutions, NeRF may produce blurred or aliased rendering.

Solutions:

Mip-NeRF renders anti-aliased conical frustums (conical frustums) instead of rays.

Effect(Experiment):

• Be 7% faster than NeRF and half the size
• Reduces average error rates by 17% on the dataset presented with NeRF and by 60% on a challenging multiscale variant of that dataset that we present.
• Able to match the accuracy of a brute-force supersampled NeRF on our multiscale dataset while being 22× faster.(It’s a traditional method we may first think to solve the problem of blurred rendering caused by different resolutions.)

5. Conclusion

The process: Mip-NeRF casts cones, encodes the positions and sizes of conical frustums, and trains a single neural network that models the scene at multiple scales.

1. Introduction

Problem introduction：

When the training images observe scene content at multiple resolutions, renderings from the recovered NeRF appear excessively blurred in close-up views and contain aliasing artifacts in distant views.

The input contains different scales so there are some problems in rendering.

Traditional method:

A straightforward solution is to adopt the strategy used in offline raytracing: supersampling each pixel by marching multiple rays through its footprint.

This paper:

Take inspiration from the mipmapping approach: A mipmap represents a signal (typically an image or a texture map) at a set of different discrete downsampling scales and selects the appropriate scale to use for a ray based on the projection of the pixel footprint onto the geometry intersected by that ray.

Both the traditional method and the mip method come from the knowledge of graphics [1]. Mip comes from the Latin *multum in parvo: a small space where many things are placed. *In computer graphics, mipmapping is a texture rendering technology that speeds up rendering and reduces image aliasing. Simply put, mipmapping is to reduce the main image into a series of successively reduced small images, and save these lower-resolution small images. This strategy is called pre-filtering. The computational burden of anti-aliasing is concentrated on preprocessing: no matter how many times a texture needs to be rendered later, it only needs to be preprocessed once.

3. Method

For Mip-NeRF, the main innovation points are divided into three aspects:

• The rendering process of Mip-NeRF is based on anti-aliased conical frustums;
• Mip-NeRF proposed a new positional encoding method - IPE (Integrated Positional Encoding);
• Mip-NeRF reduces coarse and fine MLP to a single multiscale MLP (a single multiscale MLP);

3.1. Cone Tracing and Positional Encoding

3.1.1 Cone Tracing

Set phase machine center$o$Toward direction$d$, a cone is emitted outward according to the pixel point to be determined in the picture. The size of the cone is limited by the pixel size, and is defined on the pixel plane as: $\dot{r}=\frac{2}{\sqrt{12}}\ast \text{the width of the pixel}$

We can use the following formula to determine whether the query point is inside the cone (input is the boundary$[t_o,t_1]$Given radius$\dot{r}$):
$$\begin{array}{c}\mathrm{F}(\mathbf{x},\mathbf{o},\mathbf{d},\dot{r},t_0,t_1)=1\{\left(t_0<\frac{{\mathbf{d}}^{\mathrm{T}}(\mathbf{x}-\mathbf{o})}{{\Vert \mathbf{d}\Vert }_2^2}<t_1\right)\\ \wedge \left(\frac{d^T(x-o)}{\Vert d{\Vert }_2\Vert \mathbf{x}-\mathbf{o}{\Vert }_2}>\frac{1}{\sqrt{1+(\dot{r}/\Vert \mathbf{d}{\Vert }_2{)}^2}}\right)\},\end{array}$$
This formula has two restrictions, the first part is in $d$direction, immediately$\mathbf{x}-o$exist$The\; projection\; in\; the\; d$ direction cannot exceed$[t_o,t_1]$; the second part is in the angular direction, that is, it cannot exceed the angle limit of the cone.

Afterwards we discussed how to encode this circular cone. The authors said that the simplest thing they found was to find the expectation for the points of the entire circular cone.

In order to facilitate calculation and obtain a converged solution, a three-dimensional Gaussian distribution is first used to fit such a circular cone, so that we can characterize such a circular cone using only the mean value and the covariance matrix.

This Gaussian model is completely represented by 3 values: $m_t$ (average distance along the ray), $p_t^2$ (variance along the ray direction), $p_r^2$（沿射线垂直方向的方差）
$$\begin{array}{rl}{m}_t& =t_{\mu }+\frac{2t_{\mu }{t}_d^2}{3t_m^2+t_d^2},p_t^2=\frac{t_d^2}{3}-\frac{4t_d^4(12t_m^2-t_d^2)}{15(3t_m^2+t_d^2{)}^2}\\ p_r^2& ={\dot{r}}^2\left(\frac{t_m^2}{4}+\frac{5t_d^2}{12}-\frac{4t_d^4}{15(3t_m^2+t_d^2)}\right).\end{array}$$
其中，$t_{\mu }=(t_0+t_1)$，$t_{\delta }=(t_1-t_0)/2$。

In this way, given input (boundary$[t_o,t_1]$Given radius$\dot{r}$), you can find the $\mu$和$p$

This Gaussian model can then be converted from the truncated cone coordinate system into world coordinates to obtain the final multivariate Gaussian model:
$$m=O+m_t\mathbf{d},S=p_t^2(\mathbf{d}{\mathbf{d}}^{\mathrm{T}})+p_r^2(\mathbf{I}-\frac{\mathbf{d}{\mathbf{d}}^{\mathrm{T}}}{\Vert \mathbf{d}{\Vert }_2^2})$$

3.1.2 Positional Encoding

One of the Nerf functions:
$$\gamma (\mathbf{x})={\left[\sin (\mathbf{x}),\cos (\mathbf{x}),\dots ,\sin (2^{L-1}\mathbf{x}),\cos (2^{L-1}\mathbf{x})\right]}^T$$
Determine the equivalence of the equation and the equation:
$$\mathbf{P}={\left[\begin{array}{cccccccccc}1& 0& 0& 2& 0& 0& \cdots & 2^{L-1}& 0& 0\\ 0& 1& 0& 0& 2& 0& \cdots & 0& 2^{L-1}& 0\\ 0& 0& 1& 0& 0& 2& \cdots & 0& 0& 2^{L-1}\end{array}\right]}^T,\gamma \left(x\right)=\left[\begin{array}{c}\sin (\mathbf{Px})\\ \cos (\mathbf{Px})\end{array}\right].$$
Using matrix form to position encode the previous Gaussian model, the expectation and variance become:
$$m_{\gamma }=\mathrm{P}\mu ,S_{\gamma }=\mathrm{P}\Sigma {\mathrm{P}}^T.$$
Given the infinitesimal equation of sin(x) and cos(x):
$$\begin{array}{rl}{\mathrm{AND}}_{x\sim \mathcal{N}(\mu ,{\sigma }^2)}[\sin (x)]& =\sin (\mu )\exp (-{(}^1/2){\sigma }^2),\\ {\mathrm{AND}}_{x\sim \mathcal{N}(\mu ,{\sigma }^2)}[\cos (x)]& =\cos (\mu )\exp (-{(}^1/2){\sigma }^2).\end{array}$$
In the case of a slightly larger, slightly less stable environment, the Independent Peripheral Equipment (IPE). )definition:
$$\begin{array}{rl}\gamma (\mathbf{\mu },\mathbf{\Sigma })& ={AND}_{\mathbf{x}\sim \mathcal{N}({\mathbf{\mu }}_{\gamma },{\mathbf{\Sigma }}_{\gamma })}[\gamma (\mathbf{x})]\\ & =\left[\begin{array}{c}\sin ({\mathbf{\mu }}_{\gamma })\circ \exp (-(1/2)\mathrm{diag}({\mathbf{\Sigma }}_{\gamma }))\\ \cos ({\mathbf{\mu }}_{\gamma })\circ \exp (-(1/2\left)diag\right({\Sigma }_{\gamma }))\end{array}\right]\end{array}$$

Simplifications can be made when calculating:

$\mathrm{diag}({\mathbf{\Sigma }}_{\gamma })=[\mathrm{diag}(\mathbf{\Sigma }),4\mathrm{diag}(\Sigma ),\dots ,4^{L-1}\mathrm{diag}(\mathbf{\Sigma }){]}^{\mathrm{T}}$

$\mathrm{diag}(\mathbf{\Sigma })=p_t^2(\mathbf{d}\circ \mathbf{d})+p_r^2(1-\frac{\mathbf{d}\circ \mathbf{d}}{{\Vert \mathbf{d}\Vert }_2^2})$

3.2 Architecture

Mipnerf and Nerf only differ in cone tracking and IPE encoding. Let’s compare these two aspects:

3.2.1 Why Muti-resolution

In Nerf, sampling is based on points on the ray, and then the point information can be queried for rendering. In Mipnerf, a cone is used, and the sampled points are the two boundaries of the circular cone. As shown on the right side of the figure above, a circular cone is established between the first point and the second point. The picture shows an elliptical shape. This It is the distribution of three-dimensional Gaussians in this direction. This encoding and sampling method is the ability of mipnerf to learn multiple scales: explicitly encoding the scale into the input features, and then letting the network learn. This is the ability of mipnerf to utilize volume as mentioned in the picture, giving it a better anti-aliasing effect than nerf.

3.2.2 High-frequency information

For hyperparameters$The\; problem\; of\; L$ reflects mipnerf’s ability to process high-frequency information.

It can be seen from the figure that when L increases, the high-frequency information increases and the Nerf effect also drops sharply. However, mipnerf can remain unchanged. This is because as the sampling space becomes larger, the high-frequency information will be smoothed. (I don’t understand the specific principle yet), you can refer to the following animation:

This cone-shaped dynamic setting allows mipnerf to automatically filter high-frequency information when processing distant views, alleviate the aliasing phenomenon (that is, remove high-frequency components in the scene), and restore the processing of high-frequency information when processing close-up views.

3.2.3 Different sampling

In the previous Nerf, it was necessary to sample the coarse and fine layers of the network and put them into two MLPs to calculate a unified loss function. Now with the help of mipnerf's ability to handle multiple scales, only one MLP is needed to complete the adaptive process. Work.

In general, Mipnerf uses the volume representability of cone coding to complete multi-scale learning and achieve better anti-aliasing effects.

Reference

[1] Computer Graphics Seven: Texture Mapping and Mipmap Technology - Zhihu (zhihu.com)

[2] Mip-NeRF paper notes - Zhihu (zhihu.com)

[3] [Review] Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields (velog.io)

[4] Mip-NeRF: Anti-aliasing multi-scale neural radiation field ICCV2021_tzc_fly's blog-CSDN blog

[5] Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields | IEEE Conference Publication | IEEE Xplore