Reading classic literature--SST-Calib (Synchronized spatiotemporal parameter calibration method of lidar and camera)

0. Introduction

With information from multiple input modalities, sensor fusion-based algorithms often outperform single modalities. Cameras and lidar with complementary semantic and depth information are typical sensor configurations in complex driving environments. However, for most camera and lidar fusion algorithms, sensor calibration will greatly affect performance. Specifically, detection algorithms often require precise geometric relationships between multiple sensors as input, and the timestamps of sensors are synchronized by default. The article "SST-Calib: Simultaneous Spatial-Temporal Parameter Calibration between LIDAR and Camera" proposes a segmentation-based framework to jointly estimate camera lidar Geometric and temporal parameters in kit calibration.

1. Main contributions

This work proposes a joint spatiotemporal calibration framework between lidar and cameras on autonomous driving platforms. The input to the proposed framework is a sequence of camera and lidar frames. Here, each sensor modality is processed through an arbitrary semantic segmentation network, which can be selected based on the available training data. Secondly, the segmented LIDAR point cloud is projected onto the semantic image and a newly designed bidirectional alignment loss is calculated for geometric parameter regression. Not limited to point-to-pixel loss, we also downsample semantic pixels with point-to-point loss. To estimate the time delay between,we estimate the visual odometry from two consecutive,images and predict the offset point cloud for matching. The contributions of this article are as follows:

1. Proposed a joint spatiotemporal calibration algorithm for the lidar camera sensor suite;

2. The bidirectional loss is designed to obtain more robust performance in geometric parameter regression;

3. Combine time parameters with visual odometry to estimate the time delay between sensors.

2. Main methods

The workflow of the proposed calibration method is shown in Figure 1. The calibration process includes a static spatial parameter calibration module forspace initial guessand a joint space-time fordual parameter estimation Parameter calibration module.

The input of the proposed algorithm is a point cloud scan$P_k\in {\mathbb{R}}^{3\times N_p}$, and two consecutive RGB images { I k + δ , I k + δ − 1 } ∈ Z N h × N w × 3 \{I_{k+δ }, I_{k+δ−1}\} ∈ \mathbb{Z}^{N_h×N_w×3} { Ik+δ​,Ik+δ−1​}∈WITHNh​×Nw​×3. Among them$N_p$ is the number of points in the scan, $N_h$sum$N_w$ is the size of the image. The goal of the algorithm is to estimate the 6 degrees of freedom of the geometric relationship { R , t } \{R, t\} { R,t}（其中$R\in R^{3\times 3}，t\in R^3$）sum$P_k$与$I_{k+\delta }$Time delay between $d\in \mathbb{R}$。

To achieve this goal, we first process arbitrary semantic segmentation algorithm $P_k$ sum $I_{k+\delta }$ to obtain semantic mask$P_{m,k}$和$I_{m,k+\delta }$. Then, use the initial external parameter guessesroughly measured or sampled { R i n i t , t i n i t } \{R_{init}, t_{init}\} { Rinit​,tinit​} Japanese intellectual knowledge$K\in R^{3\times 3}$, convert the lidar point cloud Projected onto the camera image plane. By finding point-to-pixel and pixel-to-point nearest neighbors, calculating the Euclidean distance between them, which is the loss function of the optimization algorithm.

The first optimization iteration (static spatial parameter calibration module) will be performed on frames where the vehicle speed is almost 0. Static spatial parameter calibration gives an initial estimate ofrotation and translation { R ^ s t a t i c , t ^ s t a t i c } \{\hat{R}^{static },\hat{t}^{static}\} { R^static,t^stati c}. This estimate will be used as an initial guess and regularization reference for joint spatiotemporal parameter calibration.

Secondly, for dynamic scenes, we estimate from visual odometry $I_{k+\delta }$sum$I_{k+\delta -1}$Time information between , this odometry will predict the speed between two camera frames${\widehat{in}}_k\in {\mathbb{R}}^3$. Zairi,$P_k$sum$I_{k+\delta }$The translational offset between can be expressed as$t_{\delta ,k}={\widehat{in}}_k\cdot \delta$. My general${\widehat{in}}_k$As part of the optimization, and estimate $\hat{d}$和 { R ^ , t ^ } \{\hat{R}, \hat{t}\} { R^,t^}。

Figure 1: Workflow of the proposed calibration method

2.1 Semantic segmentation

With ready-made semantic segmentation modules, the proposed method can be applied to any dataset with semantic labels. In this paper, we use SqueezeSegV3 [26] and SDC-net [27] for semantic segmentation of point clouds and images respectively. Considering the frequent occurrence of vehicles in urban environments, in this work, we only use vehicle categories for semantic segmentation. Applying these semantic segmentation modules to the input, we get the semantic mask $P_{m,k}$，$I_{m,k+\delta }$。

2.2 Point cloud projection

To calculate the semantic loss, we first compare the points$p_{i,m,k}\in P_{m,k}（p_{i,m,k}\in R^{The\; semantic\; mask\; of\; 3})$ is projected onto the two-dimensional image plane. According to the classic camera model [28], we can achieve projection in the following way

Zareri，$p{u}_{i,m,k}$和$p{v}_{i,m,k}$ is the projection point${\tilde{p}}_{i,m,k}\in R^{The\; image\; coordinates\; of\; 2}$.

2.3 Two-way loss (key content)

让${\tilde{p}}_{1,m,k}\dots {\tilde{p}}_{n_p,m,k}$becomes a set of projected LIDAR points within the camera's field of view. Now for the projected point ${\tilde{p}}_{i,m,k}$，裹$q_{j,m,k+\delta }\in I_{m,k+\delta }$ becomes the nearest neighbor pixel of the same category. Then, the $Unidirectional\; point-to-pixel\; (point-to-image)\; semantic\; alignment\; loss\; on\; k$ frames:

Here, the loss is calculated based on each projection point. Figure 2a shows the process of point-to-pixel loss calculation. As shown in [10], by minimizing this loss function, we can make the projected point cloud overlap well with pixels with the same semantic label. Therefore, minimizing this loss function allows us to get the correct${\widehat{AND}}_{static}=\{{\hat{R}}_{static},{\hat{t}}_{static}\}$Estimated. However,when the initial guess of the extrinsic parameter matrix is ​​significantly different from the true value, nearest neighbor matching does not necessarily give appropriate matching results for most pairs, and the information of some important pixels will Discarded. Therefore, minimizing the one-way loss will get stuck in inappropriate local minima.

Figure 2: Bidirectional projection demonstration: here, blue circles correspond to projection points and orange squares represent image pixels. Yellow squares highlight downsampled pixels.

In order to avoid the loss of information, we propose a bidirectional loss, which also utilizes pixel-to-point (image-to-point) nearest neighbor matching (Figure 2b). Considering that there are too many pixels in one image to match in real time, we downsample the pixels for pixel-to-point matching. Let { q ~ 1 , m , k + δ … q ~ n i , m , k + δ } ⊂ I m , k + δ \{\tilde{q}_{1,m ,k+δ}…\tilde{q}_{n_i,m,k+δ}\} ⊂ I_{m,k+δ} { q~​1,m,k+δ​…q~​ni​,m,k+δ​}⊂Im,k+δ​ is a set of downsampled pixels. Now for pixel${\tilde{q}}_{i,m,k+\delta }，{\tilde{p}}_{j,m,k}\in P_{m,k}$ is the nearest neighbor projection point. Then, No. $The\; pixel-to-point\; semantic\; alignment\; loss\; on\; k$ frame can be calculated as follows:

Here, the loss is calculated for each sampled pixel. Then, No. $The\; bidirectional\; semantic\; alignment\; loss\; of\; l$ iterations can be expressed as follows,

…For details, please refer toGuyueju