[Paper Notes] SVDM: Single-View Diffusion Model for Pseudo-Stereo 3D Object Detection

Original link: https://arxiv.org/abs/2307.02270

1 Introduction

  Current methods for generating pseudo-sensor representations from monocular cameras rely on pre-trained depth estimation networks. These methods require depth labels to train the depth estimation network, and pseudo-stereo methods synthesize stereo images through forward deformation of images, which can lead to pixel artifacts, distortions, and holes in occluded areas. Furthermore, feature-level pseudo-stereogram generation is difficult to apply directly and has limited adaptability.
  So how to bypass depth estimation and design a perspective generator at the image level? Compared with GAN, the diffusion model has a simpler structure, fewer hyperparameters and simpler training steps, but there is currently no research on pseudo-view generation for 3D target detection.
  This paper designs a single view diffusion model (SVDM) for pseudo-view synthesis. SVDM assumes that the left view image is known, replaces Gaussian noise with left image pixels, and gradually diffuses right image pixels to the entire image. Due to the subtle parallax of stereoscopic images, good results can be produced using only a few steps. SVDM does not use depth ground truth and can be trained end-to-end.

3. Method

3.1 Preparation knowledge

3.1.a Stereo 3D detector

  It can be divided into 3 categories: models that only require stereo image training (such as Stereo R-CNN), models that require additional depth ground truth training (YOLOStereo3D), and models based on volume grids (such as LIGA-Stereo).

3.1.b Denoising diffusion probability model (DDPM)

  For details , see Introduction to Diffusion Model . The goal of DDPM is to optimize the lower confidence bound (ELBO). Most conditional diffusion models retain the diffusion process and add the condition yyDetermine the function of$y$$${\mathbb{E}}_{t,x_0,}[\Vert ϵ-{ϵ}_{}(x_t,y,t){\Vert }_2^2]$$   But since$p(x_t|y)$ does not appear explicitly in the training target, and it is difficult to ensure that the diffusion model can learn the desired conditional distribution.

3.2 Single view diffusion model

  This model treats the new view generation task as an image-to-image (I2I) conversion task based on a diffusion model. The method in this article is shown in the figure below, which includes three diffusion model methods: Gaussian noise operator, view image operator and one-step generation.

3.2.a Gaussian noise operator

  In order to learn the transformation between two view domains, according to BBDM, this paper uses the Brownian bridge diffusion process instead of the DDPM method.
  The Brownian bridge process is a continuous-time stochastic model in which the probability distribution in the diffusion process is conditioned on the starting and ending states. Remember the starting state is $x_0\sim q_{data}(x_0)$ , the terminal state is$x_T$, then the state distribution of the Brownian bridge diffusion process is $$q_{BB}(x_t|x_0,y)=\mathcal{N}(x_t;(1-m_t)x_0+m_{t}y,d_{t}I)$$其中$m_t=t/T$，$d_t$is the variance. In order to avoid excessive variance that makes training impossible, the following variance scheduling is used: $$d_t=s[1-((1-m_t{)}^2+m_t^2)]=2s(m_t-m_t^2)$$ where$s$ controls the diversity of samples and defaults to 1.
  The forward process is as follows: when$t=0$ ,$m_t=0$ , the mean value at this time is$x_0$, the variance is 0; when $t=When\; T$ ,$m_t=1$ , the mean value at this time is$y$ , the variance is 0. The intermediate process is calculated as follows:$$x_t=(1-m_t)x_0+m_{t}y+\sqrt{d_t}ϵ$$ increment$ϵ\sim \mathcal{N}(0,i)$ . Use$t-1Replace$ ttin the above formula$t$ , subtract the two equations to get the transition probability:$$q_{BB}(x_t|x_{t-1},y)=\mathcal{N}(x_t;\frac{1-m_t}{1-m_{t-1}}{x}_{t-1}+(m_t-\frac{1-m_t}{1-m_{t-1}}{m}_{t-1})y,d_{t|t-1}I)$$其中$$d_{t|t-1}=d_t-d_{t-1}\frac{(1-m_t{)}^2}{(1-m_{t-1}{)}^2}$$  The reverse process starts from the known view and gradually obtains the distribution of the target view. i.e. based on $x_t$Predict $x_{t-1}$: $$p_{}(x_{t-1}|x_t,y)=N(x_{t-1};m_{}(x_t,t),{\tilde{d}}_{t}I)$$其中$m_{}(x_t,t)$ is the mean value of the prediction noise, estimated by the neural network based on the maximum likelihood criterion. ${\tilde{d}}_t$is the variance of the noise at each step, the analytical form is ${\tilde{d}}_t=\frac{d_{t|t-1}{d}_{t-1}}{d_t}$.
  The complete training and inference process is as follows:

BBDM training algorithm

1. Sampling data pair $x_0\sim q(x_0),y\sim q(y)$；
2. Uniform sampling time t ∈ { 1 , 2 , ⋯ , T } t\in\{1,2,\cdots,T\}t∈{ 1,2,⋯,T}；
3. Sampling Gaussian noise $ϵ\sim \mathcal{N}(0,I)$；
4. Forward diffusion: $x_t=(1-m_t)x_0+m_{t}y+\sqrt{d_t}ϵ$；
5. Define $\Vert m_t(y-x_0)+\sqrt{d_t}ϵ-{ϵ}_{}(x_t,t){\Vert }^2$ gradient.

BBDM sampling algorithm:

1. Sampling condition input $x_T=y\sim q(y)$；
2. From $t=Starting\; from\; T$ , proceed with the following process until$t=1$ :
     Calculation$z\sim \mathcal{N}(0,I)$
     计算$x_{t-1}=c_{xt}{x}_t+c_{yt}y-c_{ϵt}{ϵ}_{}(x_t,t)+\sqrt{{\tilde{d}}_t}z$；
3. $t=1$时，calculatex$x_0=c_x{x}_1+c_{y}y-c_{ϵ}{ϵ}_{}(x_1,1)$。

3.2.b View image manipulator

  The Brownian bridge diffusion model introduces additional hyperparameters. This paper proposes a method based on view image operators, which treats the target image as special noise and iteratively converts the target image into the source image. Given the initial state $x_0$and target state $y$ , intermediate state$x_t$It can be written as: $$x_t=\sqrt{a_t}{x}_0+\sqrt{1-a_t}y$$ is different from the conventional process of adding noise. What is added here is a new view image that gradually increases the weight. The sampling process is as follows:

1. Input source image $x_T$；
2. From $t=Starting\; from\; T$ , proceed with the following process until$t=0$：
     $x_0\le f(x_t,t)$
     $x_{t-1}=x_t-D(x_0,t)+D(x_0,t-1)$

(Regarding the sampling algorithm of this method, the symbols used in the original article should be problematic and lack explanation. Here we can only guess $s$ and$i$ should actually be$t$)

  $a_t$The schedule is as follows: $$a_t=\frac{f(t)}{f(0)},f(t)=\cos (\frac{t/T+s}{1+s}\cdot \frac{}{2}{)}^2$$ Compared to linear scheduling, cosine scheduling adds target views more slowly.

3.2.c Accelerated sampling and one-step generation

  Since diffusion probability models usually require a large number of step sampling, in order to speed up the inference process, this paper proposes two methods: one is to add a high-order solver to guide DPM sampling, and the other is to introduce a one-step generation method.
  Accelerated sampling : Similar to the basic idea of ​​DDIM, BBDM can also maintain the same edge distribution as the Markov inference process while using a non-Markov process.
  Given { 1 , 2 , ⋯ , T } \{1,2,\cdots,T\}{ 1,2,⋯,The length of T } isSSSubsequence of$S$ { T 1 , T 2 , ⋯ , TS } \{T_1,T_2,\cdots,T_S\}{ T1​,T2​,⋯,TS​} , the inference process can be determined by the subset of latent variables{ x T 1 , x T 2 , ⋯ , x TS } \{x_{T_1},x_{T_2},\cdots,x_{T_S}\}{ xT1​​,xT2​​,⋯,xTS​​}定义：$$q_{BB}(x_{T_{s-1}}|x_{T_s},x_0,y)=\mathcal{N}((1-m_{T_{s-1}})x_0+m_{T_{s-1}}+\frac{\sqrt{d_{T_{s-1}}-p_{T_s}^2}}{\sqrt{d_{T_s}}}(x_{T_s}-(1-m_{T_s})x_0-m_{T_s}y),p_{T_s}^{2}I)$$
   One-step generation: The goal is to perform one-step generation without sacrificing the advantages of iterative refinement. These advantages include the ability to balance computation and quality, and the ability to edit zero-shot data. The method is based on the continuous-time diffusion model probabilistic flow ordinary differential equation (ODE), whose trajectory smoothly transforms from the data distribution to a tractable noise distribution. Use a model that learns to map points on any step to the starting point of a trajectory so that the model is self-consistent (i.e., points on the same trajectory will be mapped to the same starting point).
  The consistency model can combine the random noise vector (the end point of the ODE trajectory,$x_T$) into data samples (starting point of ODE trajectory, $x_0$). By connecting the output of the consistency model in multiple steps, more calculations can be used to improve sample quality and perform zero-sample data editing, thereby maintaining the advantages of iterative refinement.

3.3 Network structure

  According to the implicit diffusion model (LDM), SVDM performs generative learning in the latent space rather than the original pixel space to reduce computation.
  LDM uses pre-trained VAE encoder $E$ maps the image$v\in {\mathbb{R}}^{3\times H\times W}$ is encoded as an implicit embedding$z=E\left(v\right)\in R^{c\times h\times w}$ . Its forward process gradually moves toward$Add\; noise\; to\; z$ , and reverse the denoising process to predict$z$ . Finally, LDM uses the pre-trained VAE decoder$D$ decodingzz$z$ , get a high-resolution image$v=D\left(z\right)$ . The encoder and decoder of VAE remain fixed during training and inference, and since$h<H,w<W$ , diffusion in low-resolution latent space is more efficient than diffusion in pixel space. The method in this article is similar, given the slave domainAAImage IA I_Asampled in$A$$I_A$, first extract the latent features $L_A$, and then perform the SVDM process to convert $L_A$Map to the corresponding,domain BBImplicit expression LA in$B$$L_{A\to B}$. Finally, the pre-trained VQGAN decoder is used to generate image $I_{A\to B}$.
  The SVDM model connects two images along the channel dimension, and uses the standard U-Net structure and Conv-NeXt residual block for up and down sampling to achieve a large receptive field and obtain contextual information. In addition, attention blocks are also introduced at different resolutions because global interactions can significantly improve the reconstruction quality.

3.4 Loss function

  The loss function contains 3 items: RGB L1 loss, RGB SSIM loss and perceptual loss.

3.4.a RGB L1 loss and SSIM loss

  The L1 loss and SSIM loss are as follows: $$\begin{gathered} L_L=\frac{1}{3HW\_\_}\sum |{\hat{I}}_{tgt}-I_{tgt}| \\ {\mathcal{L}}_{yes}=1-SSIM({\hat{I}}_{tgt},I_{tgt}) \end{gathered}$$ where${\hat{I}}_{tgt}$与$I_{tgt}$are the predicted values ​​and true values ​​of the pixel channels respectively.

3.4.b Perceived loss

  Building on past work, the perceptual loss ensures that the reconstruction is constrained to the image manifold by enforcing local authenticity and avoids the blur introduced by relying solely on RGB losses. $$L_{latent}=\frac{1}{2}\sum _{j=1}^J[(u_j^2+p_j^2)-1-\log p_j^2]$$

4. Experiment

4.4 View synthesis results based on a single image

  Quantitative results : The method in this article can surpass SotA in terms of PSNR index, but the SSIM and LPIPS indicators are slightly lower than SotA.
  Qualitative results : Visualizations show that our method generates more realistic images with smaller distortions and artifacts. This demonstrates the ability of our method to model the geometry and texture of complex scenes.

4.5 3D target detection results

  Quantitative results : Experiments show that SVDM can outperform most advanced methods when using BBDM. Using the view diffusion method can further improve performance, which shows that the view structure has better generalization ability in 3D target detection.
  In addition, although it cannot completely surpass SotA, SVDM has better performance in the detection of difficult objects. The reason for poor performance on simple objects may be limited constraints. Both background and obstacles inevitably interfere with new view generation. The ConvNeXt-UNet structure can alleviate this problem to a certain extent, but it is not perfect.

4.3 Ablation studies

  3D detection results of pedestrians and bicycles : Due to the small number of samples, the detection of pedestrians and bicycles is more difficult than the detection of cars. But the method in this article can surpass SotA on almost all difficulties.

5. Conclusion and future prospects

  Currently, one drawback of SVDM is that it cannot be trained end-to-end.