Preface: This lesson mainly introduces the related knowledge of high-precision maps. Personal level is limited and there are mistakes in understanding. We welcome criticisms.
CONTENTS:
1. HD MAP OVERVIEW [High-precision map introduction]
2. OPENDRIVE PROTOCAL [Road network protocol, here refers to the high-precision map protocol]
3. CLOUD POINT MAPPING [Point cloud map]
4. V2X [Vehicle-to-everything, car Using wireless communication technology]
1 High-precision map
1.1 Unmanned driving software framework Above the
figure: perception, decision-making planning, control
Below the figure: high-precision map, positioning
Perception: like the human eye, a channel to obtain first-hand information;
Decision-making and planning: like the human brain, it decides how to go and how to avoid collisions, so as to reach the destination efficiently;
Control: Receive the results from the decision-making and planning, the control module has brakes, throttles, and steering wheels to track the trajectory given by the computer, so as to realize the ability of unmanned driving;
High-precision map: In the L4 level, it provides great support and supplement to perception, and also provides very efficient and convenient capability support for planning, and also supports positioning to reach high-precision level;
Positioning: Understand the basic modules of oneself, the above-mentioned perception, decision-making, planning, and control cannot be carried out
1.2 High-precision map
[The above two integrations are to get the following map information: point cloud + road information (obtained by fusion)]
The high-precision map is to mark the physical information, dynamic details, and static details on the map: the
yellow line indicates the sidewalk, and then the map will tell the basic information in the figure, such as the traffic lights, the length and width of the sidewalk, and also tell the pedestrians to tell. Which traffic light; where is the green belt, where is the edge of the lane, whether the lane line is soft or hard [soft can pass through, hard not, it can also be understood as a virtual and real lane line]; the green line is the road intermediate line
The radar point cloud information is the point cloud of the road section passed by the map picking vehicle [a vehicle specifically used to collect road point cloud information to make high-precision maps], and the integrated point cloud obtained by fusion is placed in the high-precision map backup information; Its function is to accurately depict the ground position, the size and direction of each coordinate point of the point cloud, which facilitates the correction when the point cloud information of the same section of road is collected during the positioning process.
1.3 HD MAP (high-precision map) VS NAVIGATION (navigation map)
high-precision maps are collected and produced by a picking vehicle; high-precision maps are lagging behind because of the long production process; high-precision maps generally store static information , There will be less dynamic information; for dynamic information, it will be updated separately.
1.3 The relationship between high-precision map and positioning.
Self-localization (self-localization): The high-precision map itself is a self-positioning system: splicing its perceived area with the map to find the same, and at the same time, determine its own High-precision map location to achieve its own positioning purpose.
Camera&Point cloud (Camera&Point cloud): [The above self-perceptive area is generally obtained through these two]
----Pre-processing pre-processing: For example, when the map is matched, find some landmark buildings; remove the frames with data problems
----Coordinate transformation: Coordinate transformation is required for different sensor data to show the environment in the same coordinate system
----Data fusion: fusion of sensor data
High precision (high precision): high-precision map and high-precision location matching, on the basis of the existing positioning, to further reduce the positioning error
1.4 High-precision map and perception (perception)
1) High range visualization: The high-precision map stores static information to realize trans-horizon perception (×× traffic lights in the picture, unmanned vehicles can’t recognize it, but The high-precision map is saved and can be passed to the unmanned vehicle, so that you can know the two traffic lights, so that you can make better and longer-term decision-making and planning)
[In the strong light environment, the data collected by the camera is directly scrapped; the effective data set of the 64-line lidar is within 100m; the millimeter-wave radar is far away, but there is a lot of noise, and the perception and processing are difficult]
[For occlusion type obstacles, perception cannot be handled]
[V2X is also a type of beyond-horizon sensing, which can transmit dynamic information to unmanned vehicles; when V2X is combined with high-precision maps, it can achieve actual beyond-horizon sensing. For example, Tucson does this in the future, known as perception Range up to 1 km]
2) ROI: The high-precision map can reduce the perceived ROI (region of interest), thereby improving the perception accuracy (the sensing area is reduced, and the algorithm is better implemented. For example, the location of the traffic light is told by the high-precision map to tell the unmanned vehicle, only the identification light The color is enough, the algorithm is relatively simple and can be realized) and speed up the perception (for example, the object data outside the lane is directly ignored according to the high-precision map feedback, and there is more computing power to process other data) [unmanned vehicles perceive at least 10hz , That is, the maximum delay is no more than 100 milliseconds, so the reduction of ROI is very important to improve the perception speed]
3) Saving computation: saving computing power (as can be seen from the analysis of the ROI function above)
1.5 In the high-precision map and planning (Planning)
map, the left side is the information provided by the traditional map, and the right side is the information provided by the high-definition map
1) Lane center (lane center line): High-precision maps can provide this information, so that unmanned vehicles do not need to waste computing power in identifying these lane solid lines and lane dotted lines, making planning faster.
--Reference line--
Boundary boundary
2) Road network (road network): For example, unmanned vehicles need to turn right, according to the lane network information provided by the high-precision map, change lanes to the right lane in advance, which makes the planned route more reasonable [why not need to perceive it? Because there are more cars next to it, the perception ability is prone to failure]
-Merging zone (merging zone, lane merging)
-Junction (intersection) [important for prediction]
3) Traffic information: provide some traffic information to unmanned vehicles, the planning is more reasonable-
Stop sign (stop sign, stop sign)
-Speed limit (speed limit, speed limit)
[The information provided by the high-precision map may not rely on perception for better path planning]
1.6 High-precision map generation
Take APOOLO HD MAP as an example.
Data collection: collect the area where the fleet needs to make a high-precision map, and any point requires a large number of repeated collections
Data processing: point cloud splicing, repeated point cloud fitting and filtering to generate a base map [the base map meets the requirements for marking on it]
[Point cloud information is the main high-precision collection method currently used, and the information is easy to process. The accuracy is also high, but the acquisition cost is very high; you can also use the camera to collect]
Element processing: mainly to identify road attributes, mainly for camera equipment, use perception capabilities to identify street signs, marking lines such as lane lines
-Element recognition [mainly for the camera, the lane lines are recognized when the image is collected, and the recognition is not started when the driver is unmanned; it will be more troublesome and time-consuming to recognize when driving],
-Point cloud classification: classify some obstacles, such as classify dynamic cars on the road, and remove them later
Element recognition and point cloud classification are to make up and cooperate in different tasks.
Manual verification: check whether the lane line recognition is correct; whether the signal lights and signs are correct; virtual roads and logical relationships are processed by adding human driving habits [for example, if there is no turning line at an intersection, manually mark (according to people’s driving habits) To fit the turning line) into the high-precision map].
1.7 APOLLO HDMAP high-precision map production
collection vehicle The structure of the high-precision map collection vehicle: camera, lidar, GPS differential positioning; NVME SSD hard disk [store the collected data, these data are collected after the end of the process], Industrial PC, main line, GPS receiver
[A collection vehicle runs for a day, it will generate 28T data]
1.8 The process of making a high-precision map from the collected data (similar to the SLAM mapping process)
[slam is a technology used in an environment where the environment is not very complicated, the light is appropriate, and there are few dynamic obstacles]
Sensor data -> visual odometer (-> loopback verification) -> post-processing optimization -> map drawing
Visual odometry visual odometry: splicing sensor data together to calculate the driving trajectory of unmanned vehicles, and then compose each frame of pictures into local environment analysis; through trajectory and local environment analysis, one picture is matched and transferred to a complete picture On the diagram; this process is equivalent to a single-phase environment processing. However, there will be drift accumulation in each frame processing
Loopback verification, simple understanding, the unmanned vehicle passes a circle, theoretically, the position obtained by the odometer is also the initial position, but the position obtained by the odometer is another position, then, match the calculated position of the odometer with the initial position. The error of the front odometer is corrected, and this process is a loopback calibration.
Loopback verification function: correct the drift error generated by the processing of each frame of the visual odometer, to achieve close agreement between the big pictures, avoid gaps and leaks [traditional slam is only done by vision, but the unmanned car In the map, RTK (high-precision positioning) will be used, and point cloud information will be assisted to do it]
Post-processing optimization: further processing optimization data, such as further correction of errors
[Through multiple corrections to make up for it, you can make a big map]
Map drawing: splicing local maps and making maps
1.9 Mainstream technology used in HDMAP AI high-precision map making process
1) Multi-sensor calibration: radar, camera, IMU, GPS, transform all sensor information into the same coordinate system for fusion [this does not count as AI technology]
2) Point cloud processing: motion compensation, 3D model -> feature extraction, point cloud fusion -> point cloud recognition
3) Visual processing: image recognition, using offline processing
4) Multi-sensor fusion: 3D->2D projection->element tracking/clustering (->filtering & completion)
[Automated production of high-precision maps is an important direction]
[The collection vehicle does not necessarily require every lane to be run again for data collection, as long as the accuracy is appropriate, it is not necessary; when running, it uses a fleet of vehicles to collect from different angles, so as to perform fitting from different angles. To find the best fit]
[In the process of map making, after many loopback checks and post-processing optimizations, a good result is achieved before manual verification is performed]
2 OPENDRIVE PROTOCAL (high-precision map protocol)
2.1 Introduction to
OPENDRIVE OPENDRIVE is an open source document that stores road image data, which is a good representation of road information:
----Contains real road network information
----Open source
----Widely used
----Since 2006, There are many application cases
Enlarge the intersection of the above demo, the effect picture: [there are many details in the picture]
2.2 How does opendrive represent the map
1) Two coordinate systems:
----Track co-ordinates (Frenet co-codinate): Lane coordinate system, as shown in the figure above, the st coordinate system, its starting point is the origin of a certain section of road
----Inertial co-ordinates: Inertial coordinate system (usually called the world coordinate system)
2) The inertial coordinate system
Yaw, pitch and roll conform to the right-hand criterion
3) Lane coordinate system
S: along the center line of the lane; T: indicates the direction on the left and right sides of the road; H: indicates the height, if it is high above the origin of the coordinate, take a positive value.
The sign of S/T depends on the position of the coordinate system.
4) Road
Opendrive divides the road information into the following three groups: features (extra information on the road), road line information (where the road boundary is, the type of boundary, where is the road middle line, how much is the distance from the road middle line to the boundary, etc.), Reference line (such as the line provided by the lane coordinate system)
2.3 Introduction to the contents of the opendrive protocol file
1)
[Individual: file format, similar to urdf file format xml, fill in various information in the form of tags]
Basic concepts:
opendrive format xodr: a data structure based on xml.
( https://blog.csdn.net/weixin_44108388/article/details/111303985)
[This section introduces the meaning of the labels in the file, and see how to describe the road network information]
2) Dependent libraries needed for xodr files:
----matplotlib, numpy
----lxml: python tool library for parsing xml
----opendriveparser: opendrive parsing library, convert xodr into python data structure [written by the teacher Yes, this part can only be used for reference, because the document is not available at present, but you can know that there is such a way of thinking for reference]
3) When writing the xodr file, use the jupyter tool [use python to write the function, read the xodr file you have written, and the corresponding effect will be displayed] to check the actual effect of the file
4) How to draw reference lines
4-1)
-Line: no curvature
---- Spiral: also called the Euler spiral curvature linear change
---- Arc: arc, a non-constant curvature 0
- ---Cubic polynom: Cubic polynomial, curvature uncertainty
4-2) The
reference line is essentially a coordinate transformation, from the lane coordinate system to the world coordinate system
5)
Calculate the central angle and radius The radius of the
circle is the reciprocal of the curvature ：R=1/c (c is the curvature)
s is the arc length, s=2ΠrQ, Q is the angle
The new coordinate point position is calculated by the trigonometric formula.
Lane line angle: h'=h+sc
[For a very curved road, a third-degree polynomial cannot be expressed, and it can be divided into multiple segments, which can be expressed by multiple third-degree polynomials]
6) Euler spiral drawing
The calculation is very complicated. Reasonable formulas have been given in computational graphics. You can refer to:
https://en.wikipedia.org/wiki/Euler_spiral[Introduction to Euler's Spiral]
http://www.opendrive.org/tools/odrSpiral.zip[Implementation of the official drawing code c; I did not find it]
[This section is how to use the opendrive file to draw the road network of the map]
7) More high-precision map formats
LaneLet:
---- Define a more efficient and convenient road description file (ref: P. Bender, J. Ziegler and C. Stiller, "Lanelets: Efficient map representation for autonomous driving," 2014 IEEE Intelligent Vehicles Symposium Proceedings, Dearborn, MI, 2014, pp. 420-425, doi: 10.1109/IVS.2014.6856487.)
----Each x and y points are stored, and the position and curvature of the boundary are also restored Defined
----Using the grid to express the road relationship
[Small episode: The commercialization of unmanned driving may take some time, but the commercialization of high-precision maps will be very fast, because it is suitable for L2 and L3 driving. If needed, Baidu Maps, Google Maps, and NavInfo are all doing it; because high-precision maps involve a lot of geographic details and are under national control, there are only some demos]
8) Example: Apollo OpenDrive
---- more complex road network conditions, optimized for L4 level automatic driving, including more road types: such as bifurcated and merged areas, roundabout areas
---- provide more road signs More detailed information, reducing online calculations, such as: the precise location of traffic lights at intersections, providing the efficiency of detecting and identifying areas,
directly saving the location information of traffic lights to the high-precision map
3、CLOUD POINT MAPPING
3.1 Lidar&Image
1) Sensor data fusion: camera-lidar-radar
----Depth information: XYZ
----Picture information: RGB
----Key object classification and detection
2) iDAR: 3D point cloud of camera RGB+lidar
3) Slam: sensor position estimation + particle filter correction
4) Mapping: Reconstruct the global map
See information:
https://www.aeye.ai/idar/
F. Zhang, D. Clarke and A. Knoll, “Vehicle detection based on LiDAR and camera fusion,” 17th International IEEE Conference on Intelligent Transportation Systems (ITSC), Qingdao, 2014, pp. 1620-1625, doi: 10.1109/ITSC.2014.6957925.
3.2点云匹配
1) Dependent library description
----OpenCV3: currently the most commonly used and most widely used image processing dependent library, familiar with basic operations, such as cv::Mat, imread
----Eigen: the most commonly used matrix operation library under C++, Similar to Matlab, the description and calculation of the matrix are very convenient, and any calculation is a matrix
----PCL: Point Cloud Library, point cloud computing library, convenient for operation and visualization of point clouds
2) Installation method:
---- Eigen library use source copy method or apt-get installation
---- Opencv installation can use ppi source to install, you can also use source code to compile and install
---- use apt- under Ubuntu get install PCL
3) Instructions for use
代码库：JointMapping
–cmake .
–make
–./joinMap
–pcl_viewer ./map.pcd
3.3 Basic concepts
1) Camera internal parameters: describe how the camera images external objects as internal parameters of the picture (there is also a camera internal parameter: distortion parameter)
2) Camera external parameters: describes how to convert a point from the world coordinate system to the camera coordinate system (there are many coordinate systems, such as world coordinate system, lane coordinate system, camera coordinate system, vehicle coordinate system, etc.)
3) Pixel coordinate system: also called image coordinate system, gray scale image: 0-255 (8 bits), depth image: 0-65535 (16 bits), color image: multi-channel (RGB, BGR, RGBA, etc.)
4 ) Rigid body motion mode: rigid body motion is a six-degree-of-freedom motion, XYZ coordinate translation, rpy optional (roll, pitch, yaw); camera motion is a rigid body motion, and its coordinate system is a mobile coordinate system
5) How to convert the object coordinate pc in the camera coordinate system and the world coordinate system pw? Assume a T to represent the coordinate system conversion:
pc=Tpw
6) Rotating coordinates:
-The rotation of the two sets of coordinate systems is determined by the inner product space of the two sets of coordinate bases, which we define as R
---- where e and t are the base coordinates of the two space coordinate systems
--- -The rotation matrix actually describes the rotation of the camera
7) Translation coordinate system:
----The translation vector is defined as t, so the conversion between the coordinate systems can be expressed as:
8) Homogeneous coordinate system, transformation matrix
---- Although the above-mentioned linear expression form Coordinate transformation is achieved, but this form is very unfavorable for multiple transformations and cannot achieve matrix calculation, so a homogeneous coordinate system is introduced:
So far, the T here is the same as the coordinate transformation T we said, also called the transformation matrix; and the coordinate system [a 1]T (representing the column matrix) with one dimension added is also simplified to b, also called Homogeneous coordinate system
9) Euler angle
----yaw: rotate around the Z axis
----pitch: rotate around the Y axis
----roll: rotate around the X axis
10) Quaternion (Quaternion)
is an extended complex number, used to solve compactness (no redundancy) and singularity (continuous in any case in space, such as longitude and latitude in 90 dimensions, longitude is singular) The balance of is expressed as follows:
----s means real part; v means imaginary part; therefore, there is a distinction between real quaternion and imaginary quaternion
----Intuitive understanding: All rotations in space can be represented by quaternions. The geometric meaning of I, j, and k itself can be understood as a kind of rotation, but it is not simply a rotation around the XYZ axis. The real part is mainly Is used to eliminate singularity
Example:
----A certain rotation unit vector 
The quaternion expression is:
----The rotated point P'can be expressed as:
----The rotation matrix is expressed as:
----Therefore, the rotation matrix And quaternion are mutually convertible
The above-mentioned basic concepts are a spatial foundation of robotics
3.4 Camera model
1) The principle of camera imaging is the pinhole camera model:
----P represents a point in the physical world; P'represents the position in the model during imaging; the coordinates are XYZ and X'Y'Z' respectively
---- Similar triangle relationship:
2) Camera internal parameters To
switch the imaging coordinate P'to the pixel coordinate system, two steps are required: scaling and origin translation.
Assuming that the pixel coordinate of P'is [uv]T (representing the column matrix), the transformed coordinate is :
Switch this form to the homogeneous coordinate system:
here K is the internal parameter matrix we are talking about, which is the internal parameter that each camera needs to be calibrated
3) Camera external parameters
Note that P here is the position in the camera coordinate system, and the coordinate Pw in the world coordinate system can be obtained according to the rotation and translation:
---- Here is a homogeneous transformation to non-homogeneous coordinates The transformation of the
camera ---- therefore, the camera attitude R, T (does t? refer to the above formula) is also called the camera external parameter
---- the external parameter will move with the movement of the vehicle, so our high-precision map During the construction process, it is necessary to estimate this external parameter, which is also the trajectory of the vehicle.
4) Data structure
-PGM data format: Portable Gray Map, grayscale data of the most simple format, where we bring the point cloud represents a planar pressing depth information (0-65535), where 0 indicates no detected Depth
- png: Common format of RGB information-
pose: 5 groups of 7 states, txt file, [xyz qx qy qz qw], translation vector and rotation quaternion
5) Code example:
– Calculate the corresponding relationship between RGB and point cloud
according to the camera internal parameters – fuse the point cloud and RGB according to the external camera parameters of each image to form a global map
[The source code is from the teacher, not put in the cloud, the following are some screenshots, for reference only]
Through this function, you can get the point cloud image
[Steal a little lazy below, cut a picture and forget it, hahaha]
4、V2X
4.1 Comparison
High-precision map provides static perception; V2X provides dynamic perception.
V2X: low latency, strong robustness, dynamic network, ultra-long line of sight
4.2 Field-related knowledge
C-V2X is designated by China, and outstanding representatives are Datang and Huawei; DSRC is formulated by the United States and Japan, and outstanding representatives are Sony and AT&T
OBU refers to the on-board network signal; road network information and road information are remotely transmitted to unmanned vehicles through the LET-V architecture; V2X local area network is relatively closed, not an open network, so the security is relatively high
Personal summary: This lesson mainly introduces the high-precision map, introduces its function and the functions of each module, and then introduces how to make it. To make a high-precision map, you need to use the tools of the opendrive protocol, and one of the more difficult knowledge points is coordinate conversion. The following briefly introduces V2X that can provide dynamic information. Unfortunately, the code is not available, only partial screenshots.
#################### If you
don’t accumulate silicon steps, you can’t even get thousands of miles away. A
good memory is not as good as a bad pen
. The copyright belongs to the original author. I am
grateful for the teacher’s dedication
. Trouble, please like and collect it