Transformation between LiDAR coordinate system, camera coordinate system and image coordinate system
The problem solved in this paper is the mutual conversion of coordinates between point cloud and image after joint calibration. Required prior knowledge and conditions: camera calibration; lidar and camera joint calibration; camera internal reference, camera and radar external reference.
Considering the radar coordinate system as the world coordinate system, the world coordinates of any point W in the world coordinate system are: its world
homogeneous coordinates are:
and assuming that the transformation matrix from the radar coordinate system to the camera coordinate system is E, (the matrix is is the extrinsic matrix, take its English Extrinsic initials E), assuming its inverse matrix is:
Set the coordinates of point W in the camera coordinate system as:
its homogeneous coordinates in the camera coordinate system are:
Then there are:
namely:
In order to eliminate the homogeneous coordinates and simplify the calculation, the special order: the formula
becomes:
Set the conversion matrix K from the camera coordinate system to the image coordinate system (the K matrix is the camera internal parameter matrix) and its inverse matrix K^(-1) is:
Let the coordinates of point W in the image coordinate system be:
its homogeneous coordinates in the image coordinate system are:
Then there is:
That is to say:
Putting it into the above formula can get the mutual conversion formula of the camera coordinate Ximg_h and the radar coordinate:
Among them, 
Represents taking the first three rows and the first three columns of the matrix M.
It can be seen from the above formula that the world (radar) coordinates are known and can be accurately projected onto the image coordinates. But to make the camera coordinates can be accurately converted to radar coordinates, you must know Zcam, which is the distance from the point to the camera plane. Since Zcam cannot be known from the image only, the point on the image cannot be projected to the world (radar) )Coordinate System.