Summary
LiDAR and cameras are key sensors for multimodal perception systems. However, the extrinsic transformation matrix will drift over time. In order to solve this problem, this paper proposes a method based on line features to perform online camera LiDAR self-calibration. Line features are chosen to constrain the extrinsic parameter matrix because they are ubiquitous. First, line features are extracted from point clouds and images. Then the adaptive optimization method is used to obtain the accurate extrinsic parameter matrix. The paper shows that line features are robust geometric features that can be extracted from point clouds and images, and then used for external parameter calibration. This article evaluates this algorithm on the KITTI data set, and experiments show that its calibration accuracy is excellent. In several hundred frames of online experiments, this method achieved an accuracy of 0.2°. This article also demonstrated its strong adaptability in different scenarios.
Main contributions
1) Propose a method for online calibration of six degrees of freedom external parameters, which uses general line features to reduce drift errors without requiring manual point selection and special targets, so it can be applied to any given scenario.
2) Propose a point cloud line extraction method to obtain line features online from point clouds.
3) Introduce adaptive optimization methods and result confidence assessment methods to ensure that the optimization is in the right direction and to calculate calibration results more efficiently.
method
The algorithm is divided into three steps. First, pre-process the point cloud and image data. Then line features are extracted from images and point clouds, and refine operations are performed through feature filtering. Finally, the point cloud line features are projected into the pixel frame by adding perturbation to the initial external parameters, and then the scores obtained each time are calculated and optimized.
A Problem Construction and Hypotheses
This article uses Euler angles and translations to represent the relative external parameters of LiDAR and cameras. This article records the point cloud as , and the image pixel value as
(grayscale image pixels 0-255, white is 255, black is 0, so the closer to the edge point, the greater the value, the higher the score below), It represents the point i and pixel ij values of the t-th frame data. This article optimizes the 6-DOF external parameters by projecting each point to pixel coordinates, and then adding a grayscale value to each pixel to calculate the score of the current parameter. The objective function is as follows
Among them, it represents horizontal features and
vertical features, and
the point is the point of these two features. The coefficient
represents the weight of the two features. This article sets it to 0.65 in order to increase the effect of horizontal constraints.
Indicates the size of the sliding window. The score of the t-th frame is
obtained by averaging the previous frames. (Here, the weight of the image can be the gray value of the corresponding pixel. The larger the gray value, the more likely it is an edge; and the weight of the LiDAR point cloud can be its confidence as an edge, such as the distance between this edge point and the nearest point. , the greater the distance, the more likely it is an edge, and the greater the corresponding weight)
The basic assumption of this article is that when the calibration parameters are correct, the line features of the point cloud will have a higher response to the line features of the image. Moreover, this article also assumes that the internal parameters of LiDAR and camera have been correctly calibrated and the data have been synchronized.
B Image processing
First convert RGB to grayscale and use [ Lsd: a line segment detector ] to extract line features. A distance transformation model is then applied to these grayscale images. The line features in Fig3(c) are more orderly, the gray value change requirements (hyperparameters required for edge extraction) are smaller, and a larger search step can be tolerated, which can avoid the optimization from falling into local extremes.
C LiDAR processing
LiDAR mainly uses distance continuity to obtain edge line features. This article uses the local mapping method to put three frames of point clouds together, so that more points can be obtained in one frame. Use the NDT method to calculate the relative motion of the three frames. Fig4 shows the edge point extraction effect of multi-frame point cloud and single-frame point cloud, indicating that multi-frame point cloud can improve the effect of line feature extraction, especially when the LiDAR line beam is low.
The dense point cloud is then converted to an image format, and each pixel saves the distance information of the corresponding point cloud (Range Image). By comparing the distance between the current point and adjacent points, more accurate line features can be extracted, and outlier points that are close to adjacent points are eliminated. Different from [ Automatic online calibration of cameras and lasers], we use multiple wire harnesses instead of one beam. This allows horizontal features to be extracted, and line features can be used to minimize horizontal and vertical errors. Horizontal and vertical line features are saved in different point clouds. Plane luck rarely occurs, so it is ignored here, which can also reduce the amount of calculation.
D feature filter
The line features extracted by the above method are disordered, so two methods are used to extract external points. Because the point cloud is converted into a range image, a convolution kernel is used to filter out points whose distance from eight adjacent points is greater than a certain threshold. This method can filter out external points and ground points. The remaining points can be treated as line features.
A clustering method is then used to filter out line feature points with few neighboring points.
These two steps can extract more orderly point cloud line features and obtain better results in the subsequent optimization.
Line features in images will filter out features with less than 8 pixels.
E adaptive optimization
The amount of calculation and calculation accuracy should be considered when optimizing. Before optimization, the line features of LiDAR are projected into the image, and then the proportion of LiDAR points projected into the grayscale area (???) is calculated.
As shown in Fig6, in order to calculate the accuracy, two search steps are taken to get more accurate results. First, in order to avoid the search falling into the local optimal solution, a rough search method with wider image lines, smaller grayscale changes, and relatively larger compensation is adopted to quickly find the area containing the optimal solution. Then finer image line features, larger grayscale changes, and smaller search step sizes are adopted to obtain more accurate calibration results. The switching of scale and grayscale changes in the two parts will occur after the proportion of the point cloud projected into the grayscale area (proportion, how to calculate???) exceeds a certain threshold.
For computational efficiency, this paper adopts an adaptive method to keep the optimization in the right direction. In [Automatic online calibration of cameras and laser], they operate on 729 different values to obtain the function score. This method is not efficient because some steps are redundant. This paper uses a search method to optimize the objective function. It will compare the current score with the adjacent 728 scores. During this process, the Japanese peace search program finds that the external parameter has a higher score, will pause the current search step, and then start a new search at a higher score position. The search process will stop after reaching the maximum number of iterations or finding the best score, which can increase computational efficiency. Also use the sliding window method. In this article, the three-frame point cloud is used to prevent the optimization search from going in the wrong direction or falling into a local optimal solution. Therefore, the final optimization parameters should be optimal in all frames of the sliding window.
Overall, the two steps can produce more robust and accurate results, and the termination strategy can also speed up calibration. Algorithms such as Algorithm1.
