The camera imaging principle and the conversion relationship between each coordinate system are introduced above , and an idealized model from 3D to 2D is established, and the internal and external parameter matrices of the camera are expressed. And for the real situation, the distortion parameters of the camera lens that may be distorted are introduced. This article is mainly to solve the problem of camera calibration, the method used is: Zhang Youzheng calibration method.

1. Purpose of calibration

There are two purposes for camera calibration: 1. To establish a camera imaging model; 2. To correct camera distortion.

1. Establishing a camera imaging model: We have introduced the process of establishing an idealized model above. The established model makes the three-dimensional point in space become a two-dimensional point in the image. In theory, if we know the internal parameter matrix of the camera and The external parameter matrix can help us realize the inversion of the model, so as to achieve the purpose of recovering the three-dimensional information from the two-dimensional information of the image. This purpose is to obtain the unknown internal and external parameter matrix of the camera through camera calibration.

2. Correct camera distortion: For the ideal camera model above, it is a pinhole imaging model, but the pinhole imaging model has many limitations, such as less light passing through the pinhole, resulting in lower brightness. Later, people invented the lens, but due to the manufacturing process of the lens and its thickness characteristics, lens distortion will occur. $k_{1},k_{2},k_{3}$ The distortion of the lens can be corrected using the lens distortion parameters, namely the sum mentioned above $p_{1},p_{2}$ . These distortion parameters are also what we interrogate.

2. Brief introduction of Zhang Zhengyou's calibration method

The following is the original text and translation of the article.

原文：Flexible camera calibration by viewing a plane from unknown orientations

Translation: http://t.csdn.cn/tPjLF

The Zhang’s calibration method uses a calibration board composed of two-dimensional squares for calibration, collects pictures of different poses of the calibration board, extracts the pixel coordinates of the corner points in the picture, and calculates the initial value of the internal and external parameters of the camera through the homography matrix. The distortion coefficients are estimated multiplicatively, and finally the parameters are optimized using the maximum likelihood estimation method. This method is simple to operate and has high precision, which can meet most occasions.

The preparations are as follows:

Fixed-focus camera: If your camera is zoomed, do not use its zoom function when acquiring image sequences. For example, if you want to calibrate the camera on your mobile phone, you cannot tap the screen to zoom when taking pictures. The focal length of the man-machine can be controlled manually, so the focal length used in calibration and the focal length used in 3D reconstruction in the later stage should be consistent. If the focal length changes, the internal parameter matrix of the calibrated camera will also change
The resolution is generally known.

Some blog posts say that resolution is needed, but I haven't found where to use it yet.

Checkerboard: The purpose of using checkerboard is that it is simple and easy to prepare, and the corner points can be detected quickly through the corner detection algorithm. We can change the orientation of the chessboard many times to obtain images in order to obtain richer coordinate information.

Examples of pictures taken from multiple angles are as follows:

3. Calibration principle

3.1 Input and output

Input: A sequence of images of a checkerboard pattern captured with a fixed-focus camera.
Output: camera internal and external parameter matrix, distortion parameters, etc.

The internal reference matrix and distortion parameters are mainly used in the following 3D reconstruction procedures, and others will be discussed later.

Internal reference matrix K:

The external parameter matrix given above is: it is a 3×4 matrix, but the last column is 0, and the required parameters are located in the first three columns, so for the convenience of calculation and expression, the internal parameter matrix is written as follows:

but:

$[I|0]$ Represented as a matrix is a 3×3 matrix and a column of 0 vectors. The internal reference matrix discussed below is 3×3 K. $f_{x},f_{y},\gamma,u_{0},v_{0}$ Internal reference for 5 cameras.

Here the original text is expressed as A for the internal reference matrix, and this text is expressed as K, so do not confuse it.

External parameter matrix:

This matrix is a 4×4 matrix, and the column vectors of the rotation matrix R can be named respectively $r_{1},r_{2},r_{3}$ , and written in the following form:

Here, $(X_{C},Y_{C},Z_{C})$ is the camera coordinate, $(X_{W},Y_{W},Z_{W})$ is the world coordinate, R is the rotation matrix, and t is the translation vector.

Why is Zw=0 on the left side of the arrow, and the right side of the arrow $r_{3}$ disappears?

Assuming that in the world coordinate system, the coordinate origin is selected at the upper left corner of the chessboard, the plane of the chessboard coincides with the Xw-Yw plane, and the Zw axis crosses the plane vertically upwards. At this time, Zw=0 of all the characteristic points of the chessboard is particularly convenient later calculations. It can be directly converted into the form of the above formula, that is, the amount of rotation to the Zw axis is omitted $r_{3}$ .

When reading other big blog posts, including the original text of Zhang’s calibration method, the position of the arrow is an equal sign, but because the dimensions are not equal, I feel that the arrow is more accurate here.

Distortion parameters:

$k_{1},k_{2},k_{3}$ The $p_{1},p_{2}$ five distortion parameters introduced above , but the Zhang Zhengyou calibration method does not require tangential distortion . So only $k_{1},k_{2},k_{3}$ 3 parameters.

So the final mathematical model can be simplified to the following formula:

$(u,v)$ is the pixel coordinate, $s=\frac{1}{Z_{C}}$ is the scale factor, $f_{x},f_{y},\gamma,u_{0},v_{0}$ is 5 camera internal parameters, R is the rotation matrix, t is the translation vector, is the camera external parameter, and $(X_{W},Y_{W},Z_{W})$ is the world coordinate.

For previous formula comparison and symbol details, please refer to the previous article.

3.2 Homography matrix

3.2.1 Homography transformation

Homography transformation is a linear transformation between homogeneous coordinates. In 3D reconstruction, it mainly describes the transformation of an object between world coordinates and pixel coordinates. The corresponding transformation matrix is the homography matrix H. In summary, here H is as follows:

Homography is mainly used in image correction, image stitching, camera pose estimation, SLAM, etc.

3.2.2 Solving the homography matrix (“4-point method”)

Now that we have a homography matrix H that converts one coordinate $(x, y)$ to another , the question arises: how to solve it? $(x',y')$

Express relationship between two coordinates:

According to matrix theory, in order to understand all the elements in the homography matrix, it is necessary to know how many degrees of freedom H has, so as to find the corresponding number of corresponding points and solve H.

Degree of freedom: The parameter in the formula is n. Under the constraint of this formula, the number (maximum) of variables that can be freely transformed is the degree of freedom under the constraint.

Here, because H has 9 parameters, many people think that it has 9 degrees of freedom. In fact, it is not. Here, we use homogeneous coordinates, which can be scaled at any scale. Another way to explain is that in the above On the right side of the arrow in the formula, we multiply the numerator and denominator by the same number at the same time, and the equation remains unchanged. So H and kH are equivalent (k is a non-zero parameter), so:

At this time, set k as: $\frac{1}{h_{33}}$ , then $k{h_{33}=1$ , so H is equivalent to:

Therefore, the parameter of H that can be changed freely under this constraint is reduced by one, so the degree of freedom of H is actually 8. When its degree of freedom is 8, there are the following two processing methods:

1. Order $h_{33} = 1$ ;

2. Add constraints: let its modulus be 1

Taking the second method as an example, expand the equation:

Assuming that the number of corresponding points is known to be n, the following equation can be obtained:

Theoretically, since its degrees of freedom are 8, and a set of corresponding points can establish a set of (two, first row and second row) equations, we need 8 equations, ie, 4 sets of points to calculate H.

But in real application scenarios, the point pairs we calculate will contain noise. For example, the position of the point deviates by a few pixels, and even the phenomenon of feature point pair mismatch occurs. If only 4 point pairs are used to calculate the homography matrix, there will be a large error. Therefore, in order to make the calculation more accurate, generally more than 4 point pairs are used to calculate the homography matrix. In addition, the above equations are usually difficult to obtain the optimal solution by using the direct linear solution method, so other optimization methods, such as singular value decomposition, Levenberg-Marquarat (LM) algorithm, etc. are usually used to solve them in practice.

3.2.3 From checkerboard to homography matrix

Overall process:

1. Print a checkerboard calibration drawing and stick it on the surface of the plane object.

2. Take a set of checkerboard pictures in different directions, which can be realized by moving the camera or moving the calibration picture.

3. For each chessboard picture taken, detect the feature points (corner points, which are the intersection points of the black and white chessboard in the figure below) of all the checkerboard grids in the picture. We define that the printed checkerboard drawing is located on the plane of the world coordinate system Zw=0, and the origin of the world coordinate system is located at a fixed corner of the checkerboard drawing. The origin of the pixel coordinate system is located at the upper left corner of the image.

4. Because the spatial coordinates of all corner points in the chessboard calibration drawing are known, the pixel coordinates of these corner points corresponding to the corner points in the captured calibration picture are also known. If we get such N>=4 matches Point pairs (the more calculation results are more robust), the homography matrix H can be obtained according to optimization methods such as LM. Of course, the calculation of the homography matrix generally does not need to be implemented by writing a function. There are ready-made functions in MATLAB and OpenCV that can be called.

4. Mathematical derivation

First of all, we already know the form of the homography matrix in the camera imaging model, and we can use the 4-point method to solve H above, but in the middle, the internal and external parameter matrices are mixed together, because the internal parameters of each picture The matrix is fixed, so we solve the internal parameter matrix there, and then solve the external parameter matrix.

Among them, s is a non-zero constant, which can be known in the previous section, because H uses a homogeneous coordinate system here, that is to say, it can be scaled at any scale, and s can be simply understood as a multiple of scaling.

The properties of the rotation matrix are mainly used in the solution. Why is it said that the properties of the external parameters are used for the solution of the internal parameters? This will be introduced when the solution is performed below. Here, some properties of the rotation matrix will be introduced first.

4.1 Properties of the rotation matrix

First of all, we know that the rotation matrix R in the external parameter matrix can be written $R=R_{1} \times R_{2} \times R_{3}$ as $R_{1} , R_{2}, R_{3}$ the rotation coordinates generated by the rotation of the world coordinate system around the X, Y, and Z axes (detailed above). We discuss it separately. for $R_{1}$ ,

$R_{1}$ is an orthogonal matrix, according to the nature of an orthogonal matrix: $A^{T}A=E$ , it is not difficult for us to prove, and $R_{2}, R_{3}$ both are orthogonal matrices.

        It is not difficult to find from some properties of orthogonal matrices:

        The determinant of an orthogonal matrix must be +1 or -1.

        And an orthogonal matrix whose determinant value is +1 is called a special orthogonal matrix, which is a rotation matrix.

        An orthogonal matrix whose determinant value is -1 is called a rotation matrix. Flawed rotation is rotation plus mirroring. Mirroring is also a form of rotation.

        From this we can also conclude that the rotation matrix is an orthogonal matrix. And all special orthogonal matrices form a subgroup, called the special orthogonal group. Therefore, the product of the rotation matrix and the rotation matrix is also a rotation matrix, that is, R is also an orthogonal matrix.

So for the rotation matrix R, it has the properties of an orthogonal matrix. However, the properties we mainly use are the following two points:

All column vectors in R are unit vectors and are mutually orthogonal.
The inverse of R is equal to its transpose.

4.2 Internal parameter solution

Through the previous section, we know that the column vectors of R are orthogonal to each other and are unit vectors, so here, the $r_{1},r_{2}$ above properties are satisfied. In order to take advantage of this relationship, convert H into a column vector form to obtain the following relationship:

Next, the properties are used: 1. They $r_{1},r_{2}$ are mutually orthogonal; 2. $r_{1},r_{2}$ They are unit vectors with equal modulus lengths and 1. Get the following formula:

make:

It is not difficult to see that B is symmetrical along the diagonal.

The above formula can be written as follows: , $h_{i}$ which is the ith column vector of H, then:

Bring in and sort out:

It is not in the original text $\lambda$ , but it is added here for the rigor of later calculations $\lambda$ . Although this parameter is added, because it is a scaling factor and can be subtracted in the equation, it does not affect other formulas.

make:

Then simplify as follows:

Right now:

According to the calibration pictures of n different angles, we finally get a matrix set Vb=0, where V is a (2n x 6) matrix.

A picture can get a set (2) of the above equations. Taking n calibration pictures from different angles, we can get 2n equations. Among them, $v_{12},v_{11},v_{22},$ it is obtained through the previously calculated homography matrix, so it is known, and the 6 elements in b are unknowns to be found. Therefore, at least we need to ensure that the number of pictures n>=3, we can solve b. But this time with a scaling factor $\lambda$ . If b is known, then B is known.

The n here is the number of pictures, and the n used to solve the homography matrix above is the number of corresponding points in the two pictures.

Then calculate the internal parameter matrix according to the following formula:

4.3 Solving external parameters

Now that H is known and K is also known, we can solve the extrinsic parameters, but we have never used it before $r_{3}$ , but it can be obtained by following the orthogonality between the coordinate axes $r_{3}=r_{1} \times r_{2}$ , then:

and according $\left \| r_{1}\right \|=\left \| r_{2}\right \|=1$ to

I really don't know why I asked again. In reality, there is noise in the data, so the calculated rotation matrix R does not necessarily satisfy the properties of the rotation matrix. Therefore, the rotation matrix R is usually obtained according to the singular value decomposition.

5. Maximum Likelihood Optimization

The above is a theoretical derivation, but in the actual application process, the maximum likelihood estimation method is often used for optimization. (Minimize the position of x, x' to solve).

Calibrate checkerboard pictures n
A checkerboard image has m corner points
Space point coordinate X
The pixel coordinate x corresponding to the spatial point in the image
The space point is transformed by the camera internal parameter K, external parameter R, and t to obtain the coordinate x'
Distortion parameters should be added when considering distortion (only considered here $k_{1},k_{2}$ , other parameters have little effect, and increasing it will make the model responsible and unstable)
Assumption: the noise is independent and identically distributed

Finding the minimum is a nonlinear optimization problem that can be solved by the LM (Levenberg-Marquardt) algorithm. Generally, the initial values of k1 and k2 are set to 0.

Summarize

The principle and solution process of Zhang Zhengyou's calibration method are introduced. The internal and external parameter matrix of the camera can be solved from the checkerboard image sequence for subsequent reconstruction work.

code show as below:

daheizy/Camera-Calibration: Zhang Zhengyou camera calibration, use checkerboard input sequence for camera calibration, output camera internal parameters; distortion parameters; camera internal parameters, etc. (github.com)