In computer vision, reprojection errors are often used. For example, when calculating the planar homography matrix and projection matrix, the reprojection error is often used to construct the cost function, and then the cost function is minimized to optimize the homography matrix or projection matrix. The reason why the reprojection error is used is that it not only considers the calculation error of the homography matrix, but also considers the measurement error of the image point, so its accuracy will be higher.
Minimizing the Reprojection Error Problem (Bundle Adjustment Problem)
-
The pose and 3D feature points P can be optimized simultaneously
When optimizing and solving the camera motion pose (3D-2D) in SLAM, the reprojection error is used: it is the pixel coordinate (observed projection position) compared with the position obtained by projecting the 3D point according to the current estimated pose. error.
As shown in the figure below, we know through feature matching that the observation value 
sum 
is the projection of the same spatial point 
, 
the projection of
There is a certain distance from the observed value , which is the reprojection error. So we adjust the pose of the camera to make this distance smaller. Since this adjustment needs to consider many points, the error of each point in the end is usually not accurate to 0.
Considering n three-dimensional space points P and their projections 
, we wish to calculate R, t, 
represented by Lie algebra. Assuming a point in space 
, the projected pixel coordinates are
The relationship between pixel position and spatial point position is as follows:
Written in matrix form is:
There is an error in this equation due to the unknown camera pose and the noise of the observation point. We sum the errors, construct a least squares problem, and then find a good camera pose that minimizes it:
Solved using the Gauss-Newton/Levenberg-Marquardt method.