1. The purpose of camera calibration:
(1) Calculate the internal parameters and external parameters of the left and right cameras through monocular camera calibration.
(2) Correct the deformation of the picture caused by lens distortion. For example, straight lines in reality will be convex or concave after being captured as an image. This situation can be corrected after camera calibration;
(3) Use the parameters obtained from the calibration of the left and right cameras to perform dual-target calibration, and obtain depth and position information through calculation, so as to perform 3D reconstruction and ranging.
2. Four coordinate systems
One of the purposes of camera calibration is to establish the correspondence between the object from the three-dimensional world to the coordinate points on the imaging plane, so first understand the following four coordinate systems:
World coordinate system : The coordinate system of the three-dimensional world defined by the user is introduced to describe the position of the target object in the real world and the position of the camera.
Camera coordinate system : The coordinate system established on the camera is defined to describe the position of the object from the perspective of the camera, as an intermediate link between the world coordinate system and the image / pixel coordinate system.
Image coordinate system : Introduced to describe the projection and transmission relationship of the object from the camera coordinate system to the image coordinate system during the imaging process, so as to facilitate further obtaining the coordinates in the pixel coordinate system.
Pixel coordinate system : Introduced to describe the coordinates of the image point on the digital image (photo) after the object is imaged, it is the coordinate system where the information we actually read from the camera is located, and the unit is a (number of pixels).
3. Conversion between coordinates
World coordinate system: Xw , Yw , Zw .
Camera coordinate system: Xc , Yc , Zc .
Image coordinate system: x , y .
Pixel coordinate system: u , v .
Among them, the Z axis of the camera coordinate system coincides with the optical axis, is perpendicular to the plane of the image coordinate system and passes through the origin of the image coordinate system, and the distance between the camera coordinate system and the image coordinate system is the focal length f . The plane uv of the pixel coordinate system coincides with the plane xy of the image coordinate system , but the origin of the pixel coordinate system is located in the upper left corner of the figure.
3.1 World coordinate system to camera coordinate system
Assuming a rotation around the x-axis (counterclockwise)
And so on, rotate around the other axis (clockwise)
There are 6 degrees of freedom from the world coordinate system to the camera coordinate system, and translation is required in addition to rotation
3.2 Camera coordinate system to image coordinate system
3.3 Image coordinate system to pixel coordinate system
The origin of the image coordinate system is at the center of the image, and the unit is mm .
The origin of the pixel coordinate system is at the upper left corner of the image, and the unit is Pixel (unit).
dx, dy : are inherent parameters of the sensor, representing the number of millimeters per pixel.
u 0 , v 0 : represent the offset of the origin of the image coordinate system relative to the pixel coordinate system, and the unit is pixel.
3.4 The whole process of conversion from world coordinate system to pixel coordinate system
So far, if we want to reconstruct a 3D scene from the captured 2D images, we need to obtain the internal parameter M 1 and the external parameter M 2 .
3.5 Zhang Zhengyou Calibration Method
Homography transformation . It can be simply understood that it is used to describe the positional mapping relationship between the object in the world coordinate system and the pixel coordinate system. The corresponding transformation matrix is called the homography matrix.
How to get the homography matrix according to the calibration map?
After the previous series of introductions, we should have a general understanding of how to calculate the homography matrix H based on the printed checkerboard calibration map and the photos taken . Let's summarize the general process.
1. Print a checkerboard calibration drawing and stick it on the surface of a flat object.
2. Take a set of checkerboard pictures in different directions, which can be realized by moving the camera or moving the calibration picture.
3. For each chessboard picture taken, detect the feature points of all the checkerboards in the picture ( corner points, which are the intersection points of black and white chessboards in the figure below , a corner point is inside the magenta circle in the middle, and four diagonal points (red Yellow blue green) is the most special corner). We define that the printed checkerboard drawing is located on the plane of the world coordinate system Zw =0 , and the origin of the world coordinate system is located at a fixed corner of the left checkerboard drawing (such as the yellow point in the figure below). The origin of the pixel coordinate system is located at the upper left corner of the picture on the right.
Because the spatial coordinates of all corner points in the chessboard calibration drawing are known, the pixel coordinates of these corner points corresponding to the corner points in the captured calibration picture are also known , if we get such N>=4 matching point pairs (The more calculations, the more robust the results), the homography matrix H in this perspective can be obtained according to optimization methods such as LM . Of course, the calculation of the homography matrix generally does not need to be implemented by writing a function, and there are ready-made functions in OpenCV that can be called.
However, the above is only a theoretical derivation. In real application scenarios, the point pairs we calculate will contain noise. For example, the position of the point deviates by a few pixels, and even the phenomenon of feature point pair mismatch occurs. If only 4 point pairs are used to calculate the homography matrix, there will be a large error. Therefore, in order to make the calculation more accurate, generally more than 4 point pairs are used to calculate the homography matrix. In addition, it is usually difficult to obtain the optimal solution by using the direct linear solution method for the above equations, so other optimization methods are generally used in actual use, such as singular value decomposition, Levenberg- Marquarat ( LM ) algorithm
After the homography matrix H obtained by the above method , the following is the reverse deduction of the internal and external parameters of the camera through H
Through the above 2.28 and 2.29:
You can calculate B as a diagonal matrix by yourself, so B has only 6 unknowns, so you can set 6 parameters for vector b
The second part of stereo calibration, stereo correction and stereo matching
Stereo (binocular) calibration
The extrinsic parameters obtained from the calibration of each binocular camera are deduced by substitution to obtain the relative positional relationship between the two cameras in the binocular camera. Finally, the rotation matrix R and translation matrix T of the two relative to the same coordinate system can be obtained to make the two cameras overlap.
Stereo correction (polar line correction)
The purpose of stereo correction is to use the parameters obtained by stereo calibration to correct the two images of the actual non-coplanar line alignment into coplanar line alignment . Stereo matching is more efficient and computing stereo disparity is easiest when the two image planes are perfectly coplanar and aligned. Because when two image planes are perfectly coplanar and aligned, stereo matching is reduced from 2D search to 1D search, and no-matching points can be filtered out. However, in the actual binocular stereo vision system, there is no two camera image planes that are completely coplanar and line aligned, so we need to perform stereo correction (coplanar line alignment refers to: the two camera image planes are in the On the same plane, and when the same point is projected onto two camera image planes, it should be in the same row of the two pixel coordinate systems )
Correction: W is the object point, M1, M2 are the projection points. Finally , R1 and R2 are coplanar and the two pixel coordinate systems of M1 and M2 are in the same row.
Before stereo correction
After stereo correction
Stereo matching (feature matching, disparity calculation)
Using the principle of similar triangles
After simplification
Let the parallax d= x l - x r
Ensure that the target for depth perception of the left and right cameras is the pixel point of the same object, for example, 
the optimal matching point of 
. The left and right cameras can accurately calculate the depth information of the target by using the parallax for the pixels of the same object. There are many difficulties, such as the feature points that may be captured are blurred, and a single camera may not capture the same object, so the matching algorithm is very important.
Use SGM to do the matching step. The matching method is as described above, and the one with the highest correlation on the same horizontal line as the pixel to be tested in the left view in the right view is the optimal match. After the matching is completed, we need to record its parallax d , 
which is known by the depth calculation formula, and the depth can be calculated 
.
Given a disparity map, baseline, and focal length, triangulation computes the corresponding position in 3D space
Finally, the 3D reconstruction effect shows:
(a) Ordinary binocular camera
(b) Shoot the checkerboard
(c) 3D reconstruction effect
Some of the pictures are from the video screenshots of station B, welcome to exchange and learn.
The complete content of this blog can be downloaded: Calibration principles and diagrams of monocular and binocular cameras_Principles of binocular vision calibration-Industry Report Document Resources-CSDN Download
It is strongly recommended to read this article: Detailed Explanation of the Mathematical Principles of Zhang Zhengyou’s Calibration Method for Camera Calibration (including python source code) bazyd