Monocular Indoor 3D Scene Reconstruction——Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image

Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image

1. Project homepage: https://yinyunie.github.io/Total3D/
2. Paper link: https://arxiv.org/pdf/2002.12212.pdf
3. Open source code: https://github.com/yinyunie/Total3DUnderstanding
4. Related Institutions: 2020, Bournemouth University, Hong Kong Chinese, Shenzhen Big Data Center, Xiamen University

Abstract

Summary core content:

1. Definition of indoor semantic reconstruction: Simultaneous scene understanding and object reconstruction.
2. The contribution of this paper: Based on a single image, an end-to-end joint optimization of indoor layout reconstruction, target 3Dbox extraction, and target grid reconstruction is proposed.
3. Datasets: SUN RGB-D, Pix3D.

paper innovation

1. For the first time, it proposes the joint optimization of tasks such as indoor layout reconstruction, target frame detection, and target grid reconstruction based on a single image in complex indoor scenes. Experiments show that the multi-task optimization strategy makes each sub-task complement each other, so that each task reaches the best level in the industry.
2. In the target mesh reconstruction task, a new density-based mesh topology modifier is proposed. According to the local point set density, the edges of the mesh are gradually pruned. This method solves the reconstruction of indoor objects in complex background very well.
3. We fully consider the attention mechanism, as well as the interrelationships between objects. In indoor 3D object detection tasks, the pose of an object has latent, multifaceted interrelationships with surrounding objects. Our proposed strategy extracts latent features, which can better determine the position and pose of the target and improve the 3D detection task.

the whole frame

The entire network structure consists of the following parts:

1. 2D target detection: Input a single scene graph, the network is Faster-RCNN, and the output is a 2D target detection frame.
2. Layout Estimate Network (LEN): The input is a single scene graph, the output is the camera pose, and the Bounding Box parameters of the scene layout.
3. 3D Object Detection Network (ODN): The input includes (1) the target image in the 2D detection frame in the scene graph; (2) geometric features (geometry features), calculated according to the 2D detection frame. The output is an estimate of the relevant parameters of the 3D Bounding Box.
4. Mesh Generation Network (MGN): The input includes (1) a target image of a 2D target detection frame; (2) one-hot encoding of a 2D detection target category; (3) a template ball. The output is the target Mesh grid.
5. Fusion results: Embed the results of all modules together, optimize them uniformly, and finally obtain the reconstruction results of the entire scene. First, by zooming, the Mesh reconstructed by MGN is placed into the Bounding Box estimated by ODN, and then converted to the world coordinate system by the camera pose estimated by LEN.

core module

1. Camera and World System Setting
   
   As shown in the figure above, the world coordinate system coincides with the origin of the camera coordinate system, and the y-axis is perpendicular to the ground. Aligns the x-axis of the world coordinate system with the forward direction of the camera around the y-axis, so the camera's yaw angle can be removed. Then, the pose of the camera relative to the world coordinate system can be pitch $\beta$，roll$Define\; \gamma$ , in particular:
   $$R(b,c)=\left[\begin{array}{ccc}cos\left(b\right)& -cos\left(c\right)sin\left(b\right)& sin\left(b\right)sin\left(c\right)\\ sin\left(b\right)& -cos\left(\beta \right)cos\left(\gamma \right)& -cos\left(b\right)sin\left(c\right)\\ 0& -sin(c& cos(c\end{array}\right]$$

2. Parametric expression of labels
   In the world coordinate system, 3DBox can be defined by the 3D center $C\in R^3$ , size$s\in R^3$ , morning angle$i\in [-\pi ,\pi ]$ expression. For indoor targets, the center$C$ consists of the 2D projection center of the image plane$c\in R^2$ , and the distance$d\in R$ expression. The internal parameter matrix K ∈ R 3 K \in R^3of the given camera$K\in R^3$， C C The expression formula of$C$
   $$C=R^{-1}(b,c)\cdot d\cdot \frac{K^{-1}[c,1{]}^T}{||K^{-1}[c,1{]}^T|{|}_2}\text{\hspace{1em}( 1 )}$$
   Among them, the 2D projection center$c$ can be further expressed as$c^b+\delta$，$c^b$ is the center of the 2D bounding box,$d\in R^2$ represents the offset, which needs to be learned by the network. From 2D detection results$From\; I$ to 3D Bounding Box, the expression of the network is:
   $$F(I|\delta ,d,b,c,s,i)\in R^{As\; mentioned\; in\; the\; 3\times 8}$$
   review, the Bounding Box parameters estimated by the ODN network are as follows:
   $$(d,d,s,\theta )$$
   The parameters estimated by the LEN network include the camera pose and the Bounding Box of the indoor layout, which are expressed as follows:
   $$R(b,\gamma )and(C,s^l,i^l)$$
   The diagram of parametric expression is as follows:
   

3. Object Detection Network (ODN)
   
   The figure above shows the entire structure of the ODN network. The basic process is as follows:

   Input: 2D detection results, geometric features (Geometry Feature)
   (1) Use ResNet-34 to extract the image features (Apperence Feature) in each detection frame.
   (2) Encode the detection frame results and their relative positions together as geometric features as one of the inputs.
   (3) Use the object relation module to calculate the relational feature (relational feature) between each detection target and other targets.
   (4) According to the similarity between image features and geometric shapes, carry out weighted summation, which is called Attention Sum.
   (5) Add Relational Feature and Target features in the way of Element-Wise.
   (6) Use a two-layer MLP network to regress the parameters of each box $(d,d,s,\theta )$ .
   For indoor reconstruction, the object relation module expresses important meanings in the physical world: objects with close distances or objects with high appearance similarity tend to have stronger relationships.

   The specific structure is shown in the table below:
   

4. Layout Estimation Network (LEN)
   The input of the LEN network is a single scene graph, and the output is the camera pose $R(b,\gamma )$，3D Box $(C,s^l,i^l)$. The network structure is the same as ODN, but the relational feature is removed. The specific network structure is as follows:
   

5. The above picture of Mesh Generation for Indoor Objects (MGN)
   
   is the overall structure of the MGN network. The specific processing logic is as follows:

   (1) Network input: (a) the target image in the 2D detection frame; (b) the one-hot code (Category Code) of the target category in the 2D detection frame; (c) the template sphere (Template Sphere), 2562 vertices unit ball.
   (2) Use ResNet-18 to extract image features (Apperence Feature).
   (3) The one-hot encoding of the target category can provide certain shape prior information, which is helpful for faster fitting to the 3D shape of the target.
   (4) Connect the Cattegory code and the Apperence feature, and input it to the decoder network AtlasNet together with the Template sphere to get the rough shape of the target.
   (5) Edge Classifier: The structure is the same as the shape decoder (AtlasNet), and the last layer is replaced by a full connection. The input is the image feature and the deformed target Mesh, and the output $f(\ast )$ (can remove redundant meshes).
   (6) Boundary Refinement: refine the boundary of Mesh and output the final Mesh.

   The specific network structure is as follows:
   
   Density vs Distance : During the mesh reconstruction process, the connection objects of the edges need to be modified. The previous method TMN modifies the topological relationship of the mesh based on a fixed distance. The authors argue that topology modification should be based on local geometric properties. To this end, a modified Mesh based on ground-truth local density (local density) adaptation is proposed. Let $p_i\in R^3$ is a point on the reconstructed mesh,$q_i\in R^3$ is$p_i$Neighbor points on the ground-truth (see MGN network structure diagram). Therefore, we design a binary classifier $f(\ast )$ predicts$p_i$Define the ground-truth function of the system:
   $$f(p_i)=\{\begin{array}{ll}False& ||p_i-q_i|{|}_2>D(q_i)\\ True& \text{otherwise}\end{array}\text{\hspace{1em}(2)}$$
   其中，$D\left(q_i\right)=\max \underset{q_m,q_n\in N(q_i)}{\min }||q_m-q_n|{|}_2,m\ne n$，$N(q_i)$ display$q_i$The set of neighborhood points on the ground-truth, $D(q_i)$ represents the local density.

loss function

1. Individual loss
   ODN and LEN: use classification and regression loss $L^{cls,reg}=L^{cls}+l_r{L}^{reg}$ a n ODN AND LEN STRUCTURES$(i,i^l,b,c,d,s,s^l$ )。$C,Delta$ uses L2 loss.
   MGN: Use Chamfer loss$L_c$, the edge loss is $L_e$, boundary loss $L_b$, the cross-entropy loss $L_{ce}$。
2. Joint loss
   $$L=\sum _{x\in \{\delta ,d,s,\theta \}}{l}_x{L}_x+\sum _{y\in \{\beta ,\gamma ,C,s^l,i^l\}}{l}_x{L}_x+\sum _{z\in \{c,e,b,ce\}}{l}_z{L}_z+l_{co}{L}_{co}+l_g{L}_g\text{\hspace{1em}( 4 )}$$
   Among them, the first three items represent the respective losses of ODN, LEN, and MGN. The last two terms represent the joint loss term,$L_{co}$Guarantee the consistency of the predicted LEN, the world coordinate system of ODN Boxes and ground-truth. $L_g$In order to constrain the alignment of the reconstructed Mesh and the scene point cloud. Since the actual scene of the indoor scene is usually rough and partially occluded, the Chamfer distance is not used to define $L_g$, the global loss of joint training is as follows:
   $$L_g=\frac{1}{N}\sum _{i=1}^N\frac{1}{|S_i|}\sum _{q\in S_i}\underset{p\in M_i}{\min }||p-q|{|}_2^2\text{\hspace{1em}(3)}$$

experiment settings

1. Dataset
   SUN RGB-D : 10335 indoor pictures, marked with 3D layout, 2D/3D bounding box, sparse point cloud.
   Pix3D : 395 furniture models, 9 categories, aligned with 10069 images, used for training Mesh reconstruction.
2. Evaluation Indicator
   Layout: Use IoU evaluation.
   Camera Pose: MAE error.
   Object Detection: AP values ​​for all classes.
   Single Mesh Generation: Charmfer distance, which evaluates the similarity of two point cloud sets in space. Scene Mesh: L g L_g
   in equation (4)$L_g$.
3. Training strategy
   First, independently train the LEN and ODN networks on SUN RGB-D, and train the MGN network on Pix3D. Then, Pix3D and SUN RGB-D are used to supervise the generative training of Mesh, and all networks are jointly trained.
4. Reasoning With
   a single 2080Ti, a scene needs about 1.2s reasoning time.

Experimental results

1. Quantified results for 3D object detection
   
2. Mesh reconstruction results
   

---

---

Holistic 3D Scene Understanding from a Single Image with Implicit Representation

Abstract

Summary core content:

1. The purpose of the paper: 3D scene understanding based on a single image can predict the object shapes, object poses, and scene layout in a single image.
2. Challenges: As we all know, restoring 3D information from a single image is an ill-posed problem; indoor objects face serious problems such as occlusion, adhesion, and aggregation.
3. Solution: (1) Use the latest deep implicit representation to solve; (2) Propose an image-based local implicit network structure to improve the estimation of the target shape; (3) Propose a new implicit graph neural network ( SGCN), which corrects the estimation of 3D pose and scene layout. (4) Propose a new [physical violation loss] loss to solve the information estimation of overlapping targets.

---

Pre-knowledge

Graph Convolutional Network (GCN)

Intuitive definition : Convolution operation is performed on the graph structure to extract features. According to the extracted convolution features, node classification (node ​​classification), graph classification (graph classification), and edge prediction (link prediction) are performed.
The basic structure of the graph: node (node), edge (edge), the edge is also divided into directed or undirected.
 
Basic principle : Assume that there is a batch of graph data, in which there are N nodes (nodes), each node has its own characteristics, assuming that there are a total of D characteristics, we set the characteristics of these nodes to form an N×D-dimensional matrix X, The relationship between each node will also form an N×N-dimensional matrix A, also known as the adjacency matrix (adjacency matrix). X and A are the inputs of the modulo GCN model.
Simple legend : As shown in the figure below, input C, go through several hidden layers (graph convolutional layer, similar to fully connected layers), output layer F (node ​​features are updated), and the activation function can be ReLU, etc. GCN inputs a graph, and through several layers, the feature of each Node of GCN changes from X to Z, but no matter how many layers go through, the connection relationship between Nodes is shared, that is, the adjacency matrix.
 
The propagation formula of GCN : $$H^{(l+1)}=d\left(D^{-1/2}{A}^{\ast }{D}^{1/2}{H}^{(l)}{W}^{\left(l\right)}\right)$$
The characteristic matrix X in the figure below is equivalent to H in the formula. In the formula,$A^{\ast }=A+I$，$I$ is the identity matrix, which is equivalent to the self-connection of the adjacency matrix of the undirected graph G. In this way, during aggregation, information from other nodes can be aggregated, and information from its own node can also be aggregated. $D$ is$A^{\ast }$ degree matrix,$H$ is the feature of each layer,$W$ is the weight parameter of each layer.
 

paper innovation

The contribution of the thesis is reflected in four aspects, as follows:

1. Based on a single image, design a [two-stage] 3D scene understanding system. With the help of deep implicit representation [deep implicit representation], the target shape, target pose, and scene layout can be predicted at the same time, and the latter two items can be optimized at the same time.
2. A local implicit shape embedding network is proposed, which can extract accurate latent shape information of objects.
3. A GCN-based scene context understanding network is proposed, which can refine the initial estimate.
4. Design a new loss function [physical violation loss] to solve the problem of predicting target crossing.

network structure

The above figure is the overall network structure of this article: including the Initial Estimation Stage and the Refinement Stage

1. 2D Detector: Input a 2D image and use Faster-RCNN for 2D target detection.
2. Object Detection Network (ODN): Input the geometric features (Geometry Feature) generated by the image in the 2D detection frame and the 2D detection frame, and estimate the initial 3D Bounding Boxes parameters of the target.
3. Local Implicit Embedding Network (LIEN): Input the one-hot code of the image in the 2D detection frame and the detection target category, extract the implicit local shape information, and obtain the latent code of the target.
4. Layout Estimation Network (LEN): Input scene image, estimate 3D Layout Bounding Box parameters and camera pose.
5. Scene Graph Convolutional Network (SGCN): Graph convolutional network, based on the context information in the scene, refines various results of the initial estimate.
6. LDIF: Decoding module, which decodes the latently encoded features of LIEN into concrete 3D geometry.

Scene Graph Convolutional Network（SGCN）

This module is one of the author's innovations. With the help of graph convolutional network (GCN), various features in the initial stage are fused and corrected to obtain the optimal scene reconstruction result. The following figure is the specific structure of SGCN. The input is the characteristics of each module in the initial stage, which is converted into various node features (as shown in the figure below, Layout Feature, Relation Feature, Object Feature), and then through [MLPs] respectively The features are converted into expressions of the same length (as shown in the figure below, Layout Representation, Relation Representation, Object Representation, the length is 512).
 

The construction process of Scene Graph Convolutional Network ( SGCN ) is as follows:

1. Construct the entire scene as a graph $G.$ _ The graph has three types of nodes (Nodes): object nodes, scene layout nodes, and relationship nodes between them. The initial graph is an undirected edge graph, which ensures the mutual flow of information between the target (object) and the layout (layout). Then, add a relation node between each object/layout pair.
2. For different types of nodes, the input features are carefully designed according to the different input sources. For each node, the features are flattened and concaenated into a vector, which is then transformed into a vector of the same length using MLP, and finally embedded into a node's expression vector.
3. Layout Node : The feature composition includes the basic image features extracted by LEN, the parameterized output of the layout bounding box, the camera pose, and the internal reference connection of the camera (as the camera prior knowledge).
4. Object Node : The feature composition includes the basic image features extracted by ODN, the analytic code of LIEN, and the one-hot coding features of the 2D Detector prediction frame category (introducing semantic information).
5. Relationship Node : (1) The characteristics of the relationship node between two objects (objec–object) include the geometry feature generated by the 2D frame and the 3D frame information of the target. (2) The relationship node features of the object and layout (object–layout) are initialized to fixed values, and then learned by the SGCN network to obtain a reasonable relationship expression.
6. Assuming that the graph contains N targets and 1 layout, then object-layout nodes and relationship nodes can be expressed as two matrices $Z^o\in R^{d\times (N+1)}and{Z}^r\in R^{d\times (N+1{)}^{2.}}$ Define independent message passing weights for different [source-destination] types. Assuming that the type of source node is a and the type of destination node is b, we define linear transformation and adjacency matrix (adjacent matrix) as$W^{ab}，a^{ab}$.
7. Node update strategy : Assuming that the source object (or layout) is s, the destination object (or layout) is d, and the relationship is r, then the expression update strategy of object node and layout node is as formula (1), and the update expression of relationship node is formula ( 2):
    
8. message passing x4 : After four message passes, independent [MLPs] decode object node Representation into relevant bounding box parameters ( $d,d,s,\theta$ ), and the layout node decodes to ($C,s^l,i^l$ ) and camera pose$R(b,c)$ .

Local Implicit Embedding Network

1. LIEN network structure
   As shown in the figure below, the input of the network is the 2D target detection result, and the one-hot encoding of the detection frame category (providing a certain shape prior), and the output is $32\times 42$ eigenvectors.
   
2. Latent code and Analytic code?
   According to the network structure, the LIEN network is to obtain the latent encoding of the target shape information. So what features should be integrated into the nodes, which is more conducive to the SGCN network to effectively extract the context information of the scene target? The paper proposes to use local implicit representation features. As shown in the figure above, LIEN obtains a (32x42) feature matrix, where 32 represents 32 3D elements, and 42 includes 10 Gaussian function parameters (analytic code) and 32-dimensional latent variables (latent code). The Gaussian parameters describe the scaling constant, center point, radius, and Euler angle of each Gaussian function, obviously they contain the structural information of the 3D geometry. Therefore, using the analytic code as one of the features of the object node also provides SGCN with the structural information of the local target.

loss function

1. Loss function of a single model
   Train LIEN and LDIF separately, the loss function is as follows,
   $$L_p=l_{ns}{L}_{ns}+l_{us}{L}_{us}\text{\hspace{1em}(3)}$$
   其中，$L_{ns},L_{us}$Indicates sampling points close to the target surface and uniform sampling points.
   LEN loss:
   $$L_{LEN}=\sum _{y\in \{\beta ,\gamma ,C,s^l,i^l\}}{l}_y{L}_y\text{\hspace{1em}( 4 )}$$
   ODN function:
   $$L_{ODN}=\sum _{x\in \{\delta ,d,s,\theta \}}{l}_x{L}_x\text{\hspace{1em}(5)}$$
2. Properties
   $$L_j=L_{LEN}+L_{ODN}+l_{co}{L}_{co}+l_{phy}{L}_{phy}\text{\hspace{1em}( 6 )}$$
   Among them,$L_{phy}$is a new loss proposed to solve the problem of prediction targets intersecting each other. The specific loss formula is as follows:
   $$L_{phy}=\frac{1}{N}\sum _{i=1}^N\frac{1}{|S_i|}\sum _{x\in S_i}||relu(0.5-sig(\alpha LDI{F}_i(x)))||\text{\hspace{1em}(7)}$$
   其中，$LDI{F}_i\left(x\right)$ will target$i$ 's coordinate point$x$ is converted to an LDIF value, and the predicted value of LDIF is obtained through the sigmoid function. ReLU means that only the intersection point is punished. penalty intersection$p$ as shown in the figure below,
   

experiment settings

1. For the data set
   Pix3D, SUN RGB-D, please refer to the previous analysis of Total3D for details.
2. Evaluation indicators
   refer to the previous Total3D.
3. Experimental details
   Directly use the output of Total3D 2D object detection, LEN, ODN have the same structure as Total3D. LIEN is trained on Pix3D together with LDIF. SGCN is trained on SUN RGB-D. The training process is still to train each module independently first, and then jointly train. When training SGCN independently, there is no need to lose $L_{phy}$.

Experimental evaluation

1. SUN RGB-D Quantization
   
2. Mesh reconstruction results