Exchange group | Enter "sensor group/skateboard chassis group", please add WeChat ID: xsh041388
Exchange group | Enter the "Automotive Basic Software Group", please add WeChat account: Faye_chloe
Remarks: group name + real name, company, position
Author | Xi Shaohua
In the first article "Explaining the Processing Process of Lidar Point Cloud Data" and the second article "Problems and Countermeasures Encountered in Lidar Point Cloud Processing" , the author analyzed the point cloud processing of Lidar respectively. Typical problems and countermeasures encountered in the process and point cloud processing. In this article, the author will focus on analyzing the point cloud registration technology of laser point cloud in the positioning process.
Due to the limitation of the field of view, LiDAR can only obtain point cloud images within a limited field of view when collecting point cloud data in real time. In order to obtain the point cloud data of the 3D scene, the perception algorithm personnel need to convert the collected two frames of different point clouds into a unified coordinate system under the known initial attitude information, and stitch together the overlapping parts of the point clouds—— This is the point cloud registration technology.
The following will analyze what point cloud registration is, the technical methods of point cloud registration, the problems and countermeasures of point cloud registration one by one.
01
Application of point cloud registration in automatic driving
Point cloud registration was mainly used in building information modeling (BIM) in the construction industry, mining in the mining industry, etc., and there are three main roles in the field of autonomous driving, namely 3D map construction, high-precision map positioning , pose estimation .
First, 3D map construction. When building a high-precision map, the automatic driving system performs point cloud registration through the adjacent frame point clouds collected by the lidar, and unifies the point clouds collected from different locations into one coordinate system, and then constructs a three-dimensional high-resolution map. Accuracy map.
Second, high-precision map positioning. When a self-driving vehicle is driving, the vehicle needs to achieve centimeter-level precise positioning. For example, an autonomous vehicle needs to estimate its precise location on a map and its distance from the curb of the road. The automatic driving system uses point cloud registration technology to match the point cloud data collected in real time with the data of the high-precision map to provide accurate positioning for the automatic driving vehicle.
Third, pose estimation. The automatic driving system estimates the relative attitude information of the vehicle through the point cloud registration technology, and then it is beneficial to make decision planning for the vehicle.
02
The working principle and technical method of point cloud registration
2.1 How point cloud registration works
The working principle of point cloud registration is that due to the limitation of various factors such as the environment, the point cloud collected in a single point cloud collection process can only cover a part of the target surface. In order to obtain a complete target point cloud information, it is necessary to scan the target object multiple times, and perform the rigid body transformation of the coordinate system on the obtained 3D point cloud data, and convert the local point cloud data on the target object into the same coordinate system.
In layman's terms, the key to point cloud registration is how to find the correspondence between the initial point cloud and the target point cloud, and then match the original point cloud and the target point cloud through this correspondence, and calculate their features The similarity is finally unified into one coordinate system.
Figure: 3D mapping effect of point cloud
(Data source: Internet)
Point cloud registration can usually be divided into two steps, namely coarse registration and fine registration .
Coarse registration , that is, the initial registration of point clouds, refers to aligning two point clouds with different positions as much as possible through the initial value of a rotation and translation matrix. The mainstream methods of coarse registration include RANSAC, 4PCS, etc.
After rough registration, the overlapping parts of the two point clouds can be roughly aligned, but the accuracy is far from meeting the positioning requirements of autonomous vehicles, and further fine registration is required.
Fine registration refers to the further calculation of the approximate rotation and translation matrices of the two point clouds on the basis of the initial registration. The mainstream methods of fine registration include ICP, NDT, deep learning , etc.
2.2 Technical method of point cloud registration
Some specific methods of point cloud registration are mentioned above. Since the technical level and technical solutions of various autonomous driving companies are different, they will adopt different point cloud registration methods-some companies will use rough registration or fine registration. Only one method is adopted in the registration process, and some companies will adopt a combination of methods. For example, in the precise allocation process, some companies will adopt the method of ICP + deep learning. This chapter will discuss in detail the above several point cloud registration methods.
2.2.1 Technical method of coarse registration
(1) RANSAC (RANdom SAmple Consensus, random sampling consistent)
Principle of the method: the algorithm randomly selects some samples from a given sample set and estimates a mathematical model, and brings the rest of the samples into the mathematical model for verification. If there are enough sample errors within a given range, the The mathematical model is optimal, otherwise continue to loop this step.
The RANSAC algorithm is introduced into the field of 3D point cloud registration. Its essence is to continuously sample random samples of the source point cloud and obtain the corresponding transformation model, then test each random transformation model, and continue to cycle through the process until the selected The optimal transformed model is used as the final result.
Specific steps:
1) Downsampling and filtering are performed on the point cloud to reduce the calculation amount of the point cloud.
2) Based on the point cloud data processed by downsampling and filtering, feature extraction is performed.
3) Use the RANSAC algorithm for iterative sampling to obtain an ideal transformation matrix.
4) Use the obtained transformation matrix to perform point cloud transformation operations.
Advantages: It is suitable for a large amount of point cloud data, and can achieve coarse registration of point clouds regardless of the distance between point clouds.
Disadvantages: There is a problem of instability in registration accuracy.
(2) 4PCS (4-Points Congruent Sets, congruent four-point set)
Principle of the method: This algorithm utilizes the geometric invariance in rigid body transformation (such as vector/line segment ratio, Euclidean distance between points), and according to the rigid transformation, the proportion of the line segment occupied by the intersection point remains unchanged and the Euclidean distance between points Invariant characteristics, try to find four approximately coplanar points (approximately congruent four point sets) corresponding to it in the target point cloud, so as to obtain the transformation matrix by using the least squares method, and iteratively select multiple groups of base points based on the RANSAC algorithm framework , according to the evaluation criterion of the largest common point set (LCP) to compare to get the optimal transformation.
Specific steps:
1) Find a coplanar four-point base that meets the long baseline requirements in the target point cloud set (the determination of the baseline has a lot to do with the overlap in the input parameters, the larger the overlap, the longer the baseline selection, and the long baseline can ensure the robustness of the matching nature, and the number of matches is small).
2) Extract the topological information of the coplanar four-point basis, and calculate two scaling factors between the four-point basis.
3) Calculate the four possible intersection positions, and then calculate the intersection coordinates of all medium and long baseline point pairs, compare the intersection coordinates and determine the matching set, and find the corresponding consistent and congruent four points.
4) Find all coplanar four-point sets in the point cloud, repeat the above steps to obtain congruent four-point sets, and find the optimal congruent four-point pairs.
Advantages: It is suitable for point cloud registration in scenes with small overlapping areas or large changes in overlapping areas, without pre-filtering and denoising of input data.
Disadvantages: Not suitable for engineering applications.
2.2.2 Technical method of fine registration
(1) ICP (Iterative Closest Point, nearest point iteration method)
Principle of the method: Select the closest point in the two point clouds as the corresponding point, solve the rotation and translation transformation matrix through all corresponding point pairs, and make the registration error between the two point clouds smaller and smaller by continuous iteration , until the threshold requirement or number of iterations we set in advance is met.
Specific steps:
1) Calculate the corresponding near point of each point in the source point cloud in the target point set.
2) Obtain the rigid body transformation that minimizes the average distance between the above corresponding points, and obtain the translation parameters and rotation parameters.
3) Perform space transformation on the obtained translation and rotation matrix to obtain a new transformation point set.
4) If the new transformation point set and the reference point set meet the average distance between the two point sets is less than a given threshold, or the number of iterations reaches the set maximum value, then stop the iterative calculation, otherwise the new transformation point set will be used as a new The source point cloud continues to iterate until it meets the requirements of the objective function.
advantage:
1) There is no need for segmentation and feature extraction of point cloud sets.
2) When the initial value is good, good algorithm convergence can be obtained.
shortcoming:
1) In the process of searching for corresponding points, the amount of calculation is large and the calculation speed is slow.
2) There are certain requirements for the initial position of the registration point cloud, and an unreasonable initial position will cause the algorithm to fall into local optimum.
3) When the ICP algorithm is looking for corresponding points, the model will use the point with the closest Euclidean distance between any two point clouds as the corresponding point. This assumption will generate a certain number of wrong corresponding points.
(2) NDT (Normal Distribution Transform, normal distribution point cloud algorithm)
Principle of the method: Firstly, rasterize the point cloud to be registered, divide it into grids of specified size, construct the probability distribution function of each grid through normal distribution, and then optimize and solve the optimal transformation parameters. The source point cloud probability density distribution is maximized to achieve the best match between the two point clouds.
Specific steps:
1) Divide the space into grid cells (also called voxel grids).
2) Project the reference point cloud into each grid.
3) Based on the points in the grid, calculate the parameters of the probability density function (PDF) of the normal distribution of the grid.
4) Transform each point of the second point cloud according to the transformation matrix.
5) Which grid of the reference point cloud does the point of the second scanned point cloud fall on, and calculate the corresponding probability distribution function PDF.
6) Calculate the maximum likelihood function to obtain the optimal transformation parameters.
advantage:
1) The requirements for the initial registration are not high, even if the error of the initial value is large, it can have a good effect.
2) There is no need for feature matching between point clouds, which avoids problems in feature matching, such as point cloud noise, object movement, and point cloud coincidence.
shortcoming:
1) The requirements for the size of the voxel grid are relatively high. If it is too large, the registration accuracy will be low, and if it is too small, the calculation will increase.
2) The registration accuracy is slightly lower than that of the ICP algorithm.
(3) Point cloud registration based on deep learning
In addition to being applied at the perception level, deep learning is also applied to the point cloud registration at the positioning level.
Point cloud registration based on deep learning refers to the use of deep learning model to extract the features of the original point cloud, so as to obtain the initial configuration value of the point cloud, and then further complete the fine registration according to the feature value. In recent years, common deep learning-based point cloud registration methods include PointNetLK, Deep ICP, DCP, PRNet, IDAM, RPM-Net, 3DRegNet, DGR, etc.
Compared with other traditional registration methods (ICP and NDT), the method based on deep learning can make the calculation faster, and can learn more advanced features, so as to achieve higher robustness.
According to whether the structure of the registration method is composed entirely of deep neural networks or replaces some components of non-deep learning methods with deep learning-based networks, learning-based point cloud registration methods are divided into partial deep learning methods and end-to-end deep learning methods.
The point cloud registration method of partial deep learning refers to directly replacing a component in the non-deep learning point cloud registration method with a component based on deep learning, which may bring speed or robustness to the original algorithm. . The biggest advantage of the point cloud registration method of partial deep learning lies in its greater flexibility.
The end-to-end point cloud registration method means that from the input of the point cloud to the final registration result, all are realized in a complete network. The end-to-end point cloud registration method can maximize the efficiency and intelligence of the deep learning method, and can also better utilize the parallel computing capability of the GPU and have faster computing speed.
At present, the application of point cloud registration technology in the autonomous driving industry is still in the early stage. A perception algorithm engineer of an autonomous driving company said: "The point cloud registration based on deep learning is still in the early stage, and the accuracy of its application cannot be guaranteed, and the application of traditional methods such as ICP also requires a lot of computing time."
03
Problems and Countermeasures of Point Cloud Registration
3.1 The problem and countermeasures of too many point clouds during registration
At present, the data on the car is uploaded to the cloud first and then processed. However, on the one hand, the cost of data return is very high. On the other hand, the real-time data processing cannot be achieved, which makes it difficult to guarantee the safety of autonomous vehicles. Therefore, it is necessary to complete the data processing link at the vehicle end.
Existing mass-produced vehicles have limited computing power and cannot directly process huge point cloud data, which has become a major obstacle in point cloud registration, and the main method to solve this problem is point cloud simplification .
Point cloud reduction refers to the removal of useless point clouds in the target point cloud data set. There are mainly two ways to achieve this:
1) Redundancy removal refers to the removal of data in some repeated areas after data registration, and this part of data has a large amount of data, and most of them are useless data, which has a great impact on the speed and quality of modeling.
2) Extraction and simplification means that due to the excessive density and quantity of point cloud data obtained by lidar scanning, part of the data is not very useful for later modeling, so under the premise of meeting a certain accuracy and maintaining the geometric characteristics of the measured object , this part of the data needs to be streamlined to improve the actual operation speed, modeling efficiency and model accuracy of the data.
3.2 Problems and Countermeasures of Point Cloud Initial Registration
As mentioned above, initial registration is also called coarse registration, which is the key point of point cloud registration. Generally, the initial value of registration is obtained through the motion data of IMU and vehicle wheel speedometer, but the current industry point Cloud initial registration is not efficient.
Improving the efficiency of initial registration can reduce the rotation and translation misalignment between point clouds under different viewing angles, and at the same time improve the overall success rate and efficiency of point cloud registration.
For the ICP algorithm, the effect of the initial value of the initial registration will greatly affect the accuracy of the ICP algorithm. For the NDT algorithm, although the initial value is not sensitive to the effect of the algorithm, improving the efficiency and quality of the initial registration can also improve the registration accuracy of the NDT algorithm.
So, in the field of autonomous driving, how do technicians improve the initial registration efficiency of point clouds?
One is to capture the key points of the target. Tang Qiang, Perception Algorithm Engineer of Zongmu Technology, said: "In order to improve the initial registration of point clouds, the most direct method is to extract and select a key object, such as a car, and then select several key points from this key object to make the rotation matrix and translation matrices."
The other is feature extraction, such as FPFH (Fast Point Feature Histogram). Dr. Xu Jian, head of Tudatong algorithm, said: "Different from ICP algorithm and NDT algorithm, the point cloud registration method based on feature extraction can improve the speed and accuracy of point cloud registration. It extracts some features (such as edges and corners or Lines and planes), using the spatial topology between two point clouds to match between point clouds. In addition, in practical engineering applications, online point cloud registration technologies such as data association and system deviation estimation are also very important. "
3.3 Problems and countermeasures of homologous point cloud registration
Homologous point cloud registration refers to the point cloud registration that acquires point cloud data from the same type of sensors but at different times and viewpoints.
On a self-driving vehicle, in addition to a main lidar, the vehicle end may also be equipped with some blind-filling lidars, or more than two main lidars may be used. The point clouds collected by these lidars belong to the same source point cloud data, but the point clouds collected from different lidars will encounter some problems in point cloud registration due to the collection from different time points and different perspectives.
On the one hand, the point cloud data collected by multiple lidars from different angles have different three-dimensional coordinate systems, and the coordinate systems of the generated noise are also different. When doing point cloud registration, the point cloud data needs to be unified into a coordinate system to achieve a more accurate registration effect.
Xu Jian said: "In order to unify the point clouds collected by multiple lidars into the same coordinate system, multi-target point cloud registration will bring more information than single-target point cloud registration during joint calibration. , so that the registration accuracy will be higher. For a single target, when the laser scans on the target, the point cloud at the edge of the target is incomplete, which will affect the accuracy of the registration. For multiple targets, the point cloud information collected by the lidar will be more For example, when the target is stationary, topological structure information will be formed between multiple targets, making the information of the initial registration of the point cloud richer, which is conducive to improving the overall accuracy of the registration."
On the other hand, the hardware performance indicators between different lidars may be different, which may cause the number of point clouds covered by the same target surface to be different.
Tang Qiang said: "Lidars with different performances have different angular resolution parameters, which means that the point cloud density is different, which will affect the final effect of point cloud registration. In order to adjust the difference in point cloud density, the perception algorithm personnel can Applying the point cloud simplification method to reduce the lidar parameters with high point cloud density can also be adjusted from the hardware, for example, the technician may adjust the angular resolution of the two lidars to the same level."
3.4 Problems and countermeasures of cross-source point cloud registration
Cross-source point cloud registration refers to the registration of point clouds from different types of sensors. Its advantage is to combine the respective advantages of multiple different types of sensors to provide richer 3D point cloud information for the automatic driving system. Compared with homologous point cloud registration, cross-source point cloud registration is still in the academic stage, and its application in the field of autonomous driving is also in the pre-research stage.
A technical expert from an autonomous driving solution provider said: "Cross-source point cloud registration is one of the main challenges for future point cloud registration technology."
Specifically, due to the different working principles between different types of sensors, the resolution of the point cloud, the quality of the point cloud density, the scale of the point cloud image, and the distance information of the point cloud between the sensors will be different, and these problems cause The technical implementation difficulty of cross-source point cloud registration is greatly increased.
Taking 4D imaging radar and lidar as an example, although 4D imaging radar can form point cloud data like lidar, the point cloud density of 4D imaging radar may only reach certain low-line laser radars (such as 16-line , 32 lines, etc.), but cannot achieve the point cloud density of high-line lidar (such as 128 lines, etc.). In addition, in terms of angular resolution parameters, 4D imaging radar can only approach the parameter level of 1°, while some lidars can achieve the level of 0.1°. When two sensors scan the surface of the same object, the effect of the point cloud collected by them will be different. For example, in the point cloud data at a distance in front of the vehicle, the quality of the point cloud of the lidar may be better than that of the 4D millimeter wave point cloud (the former has more energy return and more point clouds), resulting in the final point The accuracy of cloud registration decreases.
In the existing academic research, there are many methods for cross-source point cloud registration, but they are mainly applied in the fields of augmented reality and construction, and there are almost no corresponding applications in the field of automatic driving. The following figure shows some methods of cross-source point cloud registration compiled by the author. For example, CSGM transforms the registration problem into a graph matching problem, and uses the graph matching theory to overcome the problem of cross-source point cloud registration. For another example, FMR discusses the application of deep learning in cross-source point cloud registration.
Figure: Cross-source point cloud registration method
(Data source: "2021 Latest Comprehensive Review on Point Cloud Registration")
references
[1] Point cloud registration, "blood recognition" in automatic driving
https://mp.weixin.qq.com/s/xjg0UOqOsukgx1swxBKwHA
[2] The latest comprehensive overview of point cloud registration in 2021
https://mp.weixin.qq.com/s/SOtHePcis9S9hpN1oqP9Ug
[3] RANSAC algorithm idea and realization of point cloud coarse registration
https://blog.csdn.net/qq_42537872/article/details/126069525
[4] 3D point cloud registration algorithm-4PCS (4-point congruent set registration algorithm)
https://blog.csdn.net/weixin_42755079/article/details/126638986
[5] ICP point cloud registration algorithm
https://blog.csdn.net/
【6】Point cloud registration ICP&NDT
https://blog.csdn.net/weixin_49024732/article/details/124794910
[7] Learning-based point cloud registration method
https://zhuanlan.zhihu.com/p/501503130
write at the end
communicate with the author
If you want to communicate directly with the author of the article, you can directly scan the QR code on the right and add the author's own WeChat.
Note: Be sure to note your real name, company, and current position when adding WeChat
And the information about the position of interest, thank you!
About Contribution
If you are interested in contributing to "Nine Chapters Smart Driving" ("knowledge accumulation and sorting" type articles), please scan the QR code on the right and add staff WeChat.
Note: Be sure to note your real name, company, and current position when adding WeChat
And the information about the position of interest, thank you!
Quality requirements for "knowledge accumulation" manuscripts:
A: The information density is higher than most reports of most brokerages, and not lower than the average level of "Nine Chapters Smart Driving";
B: Information needs to be highly scarce, and more than 80% of the information needs to be invisible on other media. If it is based on public information, it needs to have a particularly powerful and exclusive point of view. Thank you for your understanding and support.
Recommended reading:
◆ Nine chapters - a collection of articles in 2022
◆ Dedicated to "the first city of autonomous driving" - a "moving the capital"
◆ The 4D millimeter-wave radar is clearly explained in the long text of 4D
◆ Application of deep learning algorithm in automatic driving regulation and control
◆ Challenges and dawn of wire control shifting to mass production and commercial use