Classic literature reading--NICER-SLAM (RGB neural implicit dense SLAM)

0. Introduction

Neural implicit representation has become a popular representation method in SLAM in recent years, especially in dense visual SLAM. However, previous work in this direction either relied on RGB-D sensors, or required a separate monocular SLAM approach for camera tracking , and failed to produce high-precision, high-density 3D scene reconstructions. In this paper, we propose NICER-SLAM, a dense RGB SLAM system that simultaneously optimizes camera pose and layered neural implicit map representation, which also allows high-quality novel view synthesis.

To facilitate the map optimization process, we integrate additional supervisory signals, including easily accessible monocular geometric cues and optical flow, and introduce a simple deformation loss to further enforce geometric consistency. Furthermore, to further improve the performance in complex indoor scenes, we also propose a locally adaptive transformation from the signed distance function (SDF) to the density in the volume rendering equation . On synthetic and real datasets, we demonstrate strong performance in dense mapping, tracking, and novel view synthesis, even competing with recent RGB-D SLAM systems. This part of the code is not yet open source, you can look forward to a wave.

1. Article contribution

The contributions of this paper are as follows:

1. We propose NICER-SLAM, one of the first dense RGB SLAMs, which enables end-to-end optimization of tracking and mapping, and also enables high-quality synthesis of new views.

2. We introduce hierarchical neural implicit encodings for SDF, different geometric and motion regularizations, and locally adaptive SDF volume-density transformations.

3. We demonstrate strong mapping, tracking, and novel view synthesis performance on both synthetic and real datasets, even competing with recent RGBD SLAM methods.

2. System overview

We provide an overview of the NICER-SLAM pipeline in Figure 2. Given an RGB video as input, we simultaneously estimate accurate 3D scene geometry and color, and camera tracking via end-to-end optimization. Figure 2 shows the system overview of NICER-SLAM. Our method takes only RGB streams as input and outputs camera poses as well as learned hierarchical scene representations for geometry and color. To achieve end-to-end joint mapping and tracking, we render predicted color, depth, normal, and optimize based on input RGB and monocular cues . In addition, we further enforce geometric consistency through RGB deformation loss and optical flow loss. We use hierarchical neural implicit representations to represent scene geometry and appearance (Section 3). With NeRF-like differentiable volume rendering, we can present per-pixel color, depth, and normal values ​​(Section 4), which will be used for end-to-end joint optimization of camera pose, scene geometry, and color (Section 5 Festival). Finally, we discuss some design choices in the system (Section 6)

3. Hierarchical neural implicit representation (similar to Nice slam)

We first introduce our optimizable hierarchical scene representation approach, which combines multi-level grid features with an MLP decoder for SDF and color prediction.
Coarse-level geometry representation : The goal of coarse-level geometry representation is to efficiently model coarse scene geometry (objects that capture geometric details) and scene layout (e.g. walls, floors), even with only partial observation data. For this we use a resolution of $32\times 32\times A\; dense\; voxel\; grid\; of\; 32$ represents the normalized scene and retains 32 features in each voxel. For any point x ∈ R 3in space$x\in {\mathbb{R}}^3$ , we use a small MLP$f^{coarse}$ with a 64-dimensional hidden layer to obtain its base SDF value$scoarse\in R$ and geometric features$z^{coarse}\in {\mathbb{R}}^{32}$ , as shown in the following formula:

where$\gamma$ corresponds to a fixed positional encoding [29, 54], mapping coordinates to higher dimensions. Following the method of [71, 69, 68], we set the level of positional encoding to 6. ${Phi}^{coarse}\left(x\right)$represents the feature grid${Phi}^{coarse}$ at pointxxTrilinear interpolation at$x\; .$

Fine-level geometric representation : While coarse geometric shapes can be obtained with our coarse-level shape representation, it is important to capture high-frequency geometric details in the scene . To achieve this, we use a multi-resolution feature grid and an MLP decoder [5, 31, 76, 53] to model high-frequency geometric details as residual SDF values . Specifically, we use a multi-resolution dense feature grid Φ { finel } 1 L {Φ^\{fine}_l \}^L_1Phi{ finel​}1L​, the resolution is $R_l$. These resolutions are sampled in geometric space [31] to incorporate features of different frequencies:

where, $R_{min}$and $R_{max}$correspond to the lowest and highest resolutions, respectively . Here we set $R_{min}=32$，$R_{max}=128$ , a total of$L=8$ levels. The feature dimension of each level is 4.
Now, for a point$x$ modeling the residual SDF values, we extract and concatenate the trilinearly interpolated features at each level and feed them into a 3 hidden layer MLP of size 64$f^{fine}$ :

where$z^{fine}\in {\mathbb{R}}^{32}$ is$The\; geometric\; features\; of\; x$ at the fine level. Base SDF values ​​scarses^{coarse}by coarse layer$s^{coarse}$ and fine layer residual SDF$∆s$， x x The final predicted SDF value of$x$$\hat{s}$ is simply the sum of both:

Color representation: In addition to3D geometry information, we also predict color values ​​so that our mapping and camera tracking can also be optimized with color loss. Also, as another application, we can render images from new perspectives on the fly. Inspired by [31], we use another multi-resolution feature grid$\{F_l^{color}{\}}_1^L$and a decoder fcolorf^{color} parameterized by a 2-layer MLP of size 64$f^{color}$ to encode the color. The number of layers of the feature mesh is now$L=16$ , and the feature dimension of each layer is 2. Min and max resolution are now R min = 16 R_{min} = 16respectively$R_{min}=16$ and$R_{max}=2048$ . We predict the color value of each point as:

where,$\hat{n}$ corresponds to s ^ \hat{s}from equation (4)$\hat{s}$ Calculated pointxxNormal at$x$$\gamma \left(v\right)$ is the viewing direction, which has a position encoding of level 4, following [68, 71].

4. Volume rendering (more important arguments)

Following recent work based on implicit methods for 3D reconstruction [38, 68, 71, 59] and dense visual SLAM [51, 76], we use differentiable volume rendering of the scene representation optimized from Section 3.1. Specifically, to render a pixel, we $o$ along its normalized line of sight direction$v$ , will ray$r$ is projected onto the pixel. Then sample N points along this ray, expressed as$x_i=o+t_{i}v$ , their predicted SDF and color values ​​are${\hat{s}}_i$和${\hat{c}}_i$. For volume rendering, we follow [68] by taking the SDF ${\hat{s}}_i$Convert to density value $p_i$:

where, $b\in R$ is the parameter controlling the conversion from SDF to bulk density. As in [29], the current rayrrThe color of$r$$\hat{C}$ is calculated as:

where,$T_i$和$a_i$Correspondingly along the ray rrSampling pointii$of\; r$The transmittance and alpha value of$i$$d_i$is the distance between adjacent sampling points. Similarly, we can also calculate the current ray rrThe depth D ^ \hat{D}of the surface intersected by$r$$\hat{D}$ and normal$\hat{N}$ , as follows:

This paper presents a method for local adaptive transformation. ** βin formula (6)$The\; \beta$ parameter models the smoothness** near the surface of the object. As the network becomes more certain about objectsurfaces,$The\; value\; of\; \beta$ decreases gradually. Therefore, this optimization scheme enables faster and sharper reconstructions. In VolSDF [68], they put$\beta$ is modeled as a single global parameter. This modeling approach essentially assumes the same degree of optimization in different scene regions, which is sufficient for small scenes. However, in complex indoor scenes, the globally optimized$The\; \beta$ value is suboptimal (see section 4.2 for ablation studies). Therefore, we propose a locally adaptive method that incorporates$The\; \beta$ values ​​are localized so that the SDF-density transformation in Equation (6) is also locally adaptive. Specifically, we maintain a voxel counter throughout the scene and count the number of point samples within each voxel during the mapping process. We empirically choose a voxel size of 643 (see Section 4.2 for ablation studies). Next, we heuristically design a method for counting$T_p$to $Transformation\; of\; \beta$ values:β

under the global input setting$The\; transformation\; was\; obtained\; by\; plotting\; the\; decrease\; of\; \beta$ with voxel count and fitting the curve. We empirically found that an exponential curve was the best fit.

5. End-to-end joint mapping and tracking (this part is also relatively necessary)

From RGB temporal input only, it is difficult to simultaneously optimize 3D scene geometry and color as well as camera pose due to the high degree of ambiguity, especially for large complex scenes with many textureless and sparsely covered regions. Therefore, to achieve end-to-end joint mapping and tracking under our neural scene representation, we propose the following loss functions, including geometric and prior constraints, single-view and multi-view constraints, and global and local constraints.

RGB rendering loss : Equation (7) connects the 3D neural scene representation with 2D observations, so we can use a simple RGB reconstruction loss to optimize the scene representation:

where, $R$ represents randomly sampled pixels/rays in each iteration,$C$ is the input pixel color value.

RGB warping loss : To further enforce geometric consistency from only color input, we also add a simple per-pixel warping loss. For mmPixels in$m$$r_m$, we first render its depth value using equation (8) and project it into 3D space, then use the $The\; intrinsic\; and\; extrinsic\; parameters\; of\; n$ frames are projected into another frame. Nearby keyframesnnA projected pixel rm → n in$n$$r_{m\to n}Expressed\; as$ r_{m→n}$. Then define the deformation loss as:

where $K_m$Indicates the current frame $list\; of\; keyframes\; for\; m$ , excluding frame$m$ itself. We will be in framennPixels projected outside the image boundary of$n\; are\; masked\; out.$Note that, unlike [11], we observe that simply performing pixel warping is more efficient than using path warping for randomly sampled pixels, with no performance drop.

Optical Flow Loss : Both RGB rendering and warping losses are point-based terms that are susceptible to local minima. Therefore, we add a loss based on optical flow estimation, which respects region smoothness priors and helps resolve ambiguities. Assumed frame mmThe sampled pixels in$m$$r_m$, the corresponding projected pixel at frame $n$ is$r_n$, then the optical flow loss can be added as follows:

where $GM(r_{m\to n})$ denote the optical flow estimated from GMFlow [66].

Monocular Depth Loss : Given an RGB input, geometric cues (such as depth or normal) can be obtained by off-the-shelf monocular depth estimators [12]. Inspired by [71], we also include this information in the optimization to guide neural implicit surface reconstruction. More specifically, in order to expect depth D ^ \hat{D}
in our rendering$\hat{D}$ and monocular depth$D$To enforce deep consistency between$\stackrel{ˉ\; ,\; we\; use\; the\; following\; loss\; [43,\; 71]:}{}$

…For details, please refer to Gu Yueju