(U, V, W) is the world coordinate system, which becomes the camera coordinate system after rigid body transformation (such as: rotation, translation), and then transforms into the image coordinate system through perspective projection , and finally converts into the pixel coordinate system through affine transformation ( u,v). The conversion relationship is as follows (Z is the scaling factor):

(1) Convert the camera coordinate system to the image coordinate system (perspective projection transformation follows the principle of pinhole imaging )

(2) Reasons for introducing homogeneous coordinates: ( Why introduce homogeneous coordinates??? )

Summarized as follows:

a. Upscaling --- express the transformation of the image as a form of matrix multiplication for unified calculation

Most image transformations can use matrix multiplication to express the mapping relationship between pixels before and after transformation, but the translation relationship can only be expressed by matrix addition, and cannot be expressed by matrix multiplication. When representing image transformations (such as translation, rotation, and scaling), matrices involve multiplication and addition, which increases calculations.

Therefore, the form of the matrix is unified by increasing the dimension, thereby reducing the amount of calculation.

b. Dimension-up --- express the coordinates at infinity in a form that can be used for calculation

Example: If the point (3,4) in the Cartesian coordinate system moves to infinity, the coordinates at this time are (oo, oo), which can be expressed as (3,4,0) using homogeneous coordinates.

Discovery: Using homogeneous coordinates, it is possible to express that parallel lines meet at a point at infinity in perspective space. In Euclidean space, this becomes meaningless, so Euclidean coordinates cannot be represented.

Homogeneous coordinates are converted to Euclidean space (3,4,5) => (3/5,4/5) ---(6,8,10) => (3/5,4/5)---( 24,32,40) => (3/5,4/5)

It can be found that homogeneous coordinates have scale invariance

2. Composition of internal and external parameter matrix

Internal reference matrix: It is only related to the camera itself and depends on the internal parameters of the camera.

External parameter matrix: The corresponding parameters will change depending on the pictures taken by the camera, that is, with the relative position of the world coordinate system and the camera coordinate system.

In the camera calibration parameters, there are 11 parameter variables. Among them, there are 5 internal parameters of the camera: focal length, principal point coordinates, and distortion parameters; there are 6 external parameters of the camera: rotation and translation.

The internal parameter matrix is expressed as

Among them, f is the image distance, dX and dY represent the physical length of a pixel on the photosensitive plate of the camera in the X and Y directions, that is, how many millimeters a pixel is on the photosensitive plate, and u0 and v0 represent the center of the photosensitive plate of the camera respectively. Coordinates in the pixel coordinate system, θ represents the angle between the horizontal and vertical sides of the photosensitive plate (90° means no error).

The extrinsic parameter matrix is expressed as

Among them, R represents the rotation matrix, and T represents the translation vector.

3. Distortion parameters

Distortion is mainly divided into radial distortion and tangential distortion.

Radial Distortion : Distortion distributed along the radius of the lens;

Cause: It is caused by the quality of the lens, and the light is more curved away from the center of the lens than near the center.

Radial distortion mainly includes barrel distortion and pincushion distortion.

Tangential Distortion: Tangential distortion occurs when the lens itself is not parallel to the camera sensor plane (image plane) or image plane.

Tangential distortion mainly occurs when the camera sensor and the lens are not parallel; because of the angle, when the light passes through the lens to the image sensor, the imaging position changes.

Distortion model:

Radial distortion model:

Tangential Distortion Model:

Radial distortion + tangential distortion model:

In the distortion model, (x', y') are the normalized image coordinates after distortion, and (x, y) are ideal undistorted normalized image coordinates.

r is the distance from the image pixel to the center of the image, ie $r^{^{2}}=x^{2}+y^{^{2}}$ .

Among them, k1, k2, k3 are radial distortion parameters; p1, p2 are tangential distortion parameters.

In the undistorted calibration, assuming that the internal reference matrix is K, a distortion parameter will be added to the matrix after distortion.

——After Radial Distortion——

2. Camera Calibration

Synchronous calibration of internal parameters and external parameters generally includes two strategies:

a. Optical calibration: use known geometric information (such as fixed-length checkerboard) to achieve parameter solution;

b. Self-calibration: Use structure from motion to estimate parameters in static scenes.

From the (feature) points (Xi, Yi, Zi) with known coordinates in space and their corresponding coordinates (ui, vi) in the image, the 11 internal and external parameters to be solved are directly estimated.

1. Zhang Zhengyou's calibration method (optical calibration) and its solution

Reference: Zhang Zhengyou Camera Calibration Method

Zhang Zhengyou’s calibration method uses the checkerboard to calibrate the board image, and uses the corresponding image detection algorithm to obtain the pixel coordinates (u, v) of each corner point.

Zhang Zhengyou’s calibration method fixes the world coordinate system on the checkerboard, so the physical coordinates of any point on the checkerboard are W=0. Since the world coordinate system of the calibration board is artificially defined in advance, the size of each grid on the calibration board is known. Yes, we can calculate the physical coordinates (U, V, W=0) of each corner point in the world coordinate system.

We will use this information: the pixel coordinates (u, v) of each corner point, and the physical coordinates (U, V, W=0) of each corner point in the world coordinate system to calibrate the camera and obtain the camera's Internal and external parameter matrix, distortion parameters.

2. The idea of solving the internal parameter matrix and external parameter matrix:

If the world coordinate system is fixed on the checkerboard, the physical coordinate of any point on the checkerboard is W=0. Therefore, the imaging model of the original single point without distortion can be transformed into the following formula. Among them, (R1 R2 T) is the first two columns of the rotation matrix R.

1. Calculate the product M of the internal and external parameter matrix--> 2. Solve the internal parameter matrix A--> 3. Solve the external parameter matrix (R1 R2 T)

(1) Calculate the product M of the internal and external parameter matrix

Among them $x=(u,v,1)^{T}$ , $X=(X,Y,Z,1)^{T}$ , the M matrix is a homogeneous matrix with 8 unknown elements, and each corner point on the calibration plate will provide two constraint equations. The matrix M can be obtained with only 4 corner points on a picture . When there are more than 4 corner points on a picture, the optimal M can be obtained by using the least squares method.

(2) Solve the internal reference matrix A

Use M1=A(R1, R2, T) to solve the internal friction matrix A.

R1, R2, as the two columns of the rotation matrix, has unit orthogonality and satisfies the following formula:

$R_{1}^{T}R_{2}=0, R_{1}^{T}R_{1}=R_{2}^{T}R_{2}=1$ 。

by $R_{1}=A^{-1}M_{1} , R_{2}=A^{-1}M_{2} ,$ , know

$M_{1}^{T}A^{-T}A^{-1}M_{2}^{T}=0,$ ，

$M_{1}^{T}A^{-T}A^{-1}M_{1}^{T}=M_{2}^{T}A^{-T}A^{-1}M_{2}^{T}=1$ 。

remember $A^{-T}A^{-1}=B$ , then

Among them, B is a symmetric matrix.

$M_{1}^{T}BM_{2}^{T}=0$

$M_{1}^{T}BM_{1}^{T}=M_{2}^{T}BM_{2}^{T}=1$

It can be recorded as $M_{i}^{T}BM_{j}^{T}=v_{ij}b$ , among them $b=[B_{_{11}}, B_{_{12}},B_{_{22}},B_{_{13}},B_{_{23}},B_{_{33}} ]$ .

Each calibration board picture can provide a constraint relationship of vb=0, which contains two constraint equations. However, vector b has 6 unknown elements. The vector b can be solved by taking 3 photos of the calibration board . When the number of pictures on the calibration board is greater than 3, use the least squares method to fit the best vector b, and obtain the matrix B; then find the internal parameter matrix A.

$v_{0}=\frac{B_{12}B_{13}-B_{11}B_{23}}{B_{11}B_{22}-B_{12}^{2}}$

$\alpha =\sqrt{\frac{1}{B_{11}}$

$\beta =\sqrt{\frac{B_{11}}{B_{11}B_{22}-B_{12}^{2}}}$

$\gamma =-{B_{12}\alpha ^{2}}\beta$

$u_{0}=\frac{\gamma v_{0}}{\beta }-B_{13}\alpha ^{2}$

(3) Solve the external parameter matrix (R1 R2 T)

For different pictures, the positional relationship between the calibration board and the camera has changed, and at this time the extrinsic parameter matrix corresponding to each picture is different.

By $H=(R_{1},R_{2},T)$ knowing the extrinsic parameter matrix can be expressed as:

$(R_{1},R_{2},T)=A^{-1}H$

The extrinsic parameter matrix of each picture can be obtained $(R_{1},R_{2},T)$ .

The rotation matrix R in the complete external parameter matrix has three columns, and the complete external parameter matrix can be obtained by calculating R3=R1×R2.

3. Parameter optimization

Our general logic is that when solving the internal and external reference matrices, we assume that there is no distortion; when solving the distortion coefficients, we assume that the obtained internal and external reference matrices are error-free. Finally, we iteratively optimize the parameters through the LM algorithm. (Zhang Zhengyou’s camera calibration method only considers the radial distortion that has a large influence in the distortion model)

1. LM algorithm

The full name of the LM method is the Levenberg-Marquardt method, which is an estimation method for least squares estimation of regression parameters in nonlinear regression. This method is a method that combines the steepest descent method and the linearization method (Taylor series). Because the steepest descent method is suitable for the situation where the parameter estimate is far from the optimal value at the beginning of the iteration, and the linearization method, that is, the Gauss-Newton method is suitable for the later stage of the iteration, and the parameter estimate is close to the optimal value. The combination of the two methods can find the optimal value quickly .

Iterative formula of LM algorithm:

It is known from the above formula that the LM algorithm is a combination of the Gauss-Newton method and the gradient descent method: when u is small, the matrix J is close to the matrix G, which is equivalent to the Gauss-Newton method. At this time, the iterative convergence speed is fast. When u is large, It is equivalent to the gradient descent method, and the iterative convergence speed is slow at this time. Therefore, the LM algorithm not only has the advantages of fast convergence speed of the Gauss-Newton method and is not easy to fall into the local extremum, but also has the characteristics of the gradient descent method stably approaching the optimal solution.

In the iterative process of the LM algorithm, it is necessary to change the size of u according to the actual situation to adjust the step size:

(1) If the objective function value of the current round of iteration is greater than that of the last round of iteration, that is, $f_{k+1}>f_{k}$ it indicates that there is a deviation in the current approach direction, resulting in skipping the optimal point, and the step size needs to be reduced by increasing the value of u.

(2) If the objective function value of the current iteration is smaller than the objective function value of the last iteration, that is, the current $f_{k+1}<f_{k}$ step size is appropriate, and the step size can be increased by reducing the u value to speed up the convergence speed.

2. Camera calibration steps

1. Print a piece of checkerboard A4 paper (the black and white spacing is known), and paste it on a flat plate;

2. Take several pictures of the checkerboard;

3. Detect feature points (Harris corners) in the picture;

4. Solve the product M of the internal parameter matrix and the external parameter matrix according to the corner position information and the coordinates in the image;

5. Use matrix B to calculate internal matrix A with 5 internal parameters and 6 external parameters;

6. Solve the distortion parameters;

7. Design the optimization objective and realize the optimization of the parameters.

4. Relevant codes and experimental results

Reference code: Zhang Zhengyou camera calibration method

import os
import numpy as np
import cv2
import glob


def calib(inter_corner_shape, size_per_grid, img_dir, img_type):
    # criteria: only for subpix calibration, which is not used here.
    # criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)
    w, h = inter_corner_shape
    # cp_int: corner point in int form, save the coordinate of corner points in world space in 'int' form
    # like (0,0,0), (1,0,0), (2,0,0) ....,(10,7,0).
    cp_int = np.zeros((w * h, 3), np.float32) #返回来一个给定形状和类型的用0填充的数组  54行3列
    cp_int[:, :2] = np.mgrid[0:w, 0:h].T.reshape(-1, 2) #  三维网格坐标划分  行2列

    # cp_world: corner point in world space, save the coordinate of corner points in world space.
    cp_world = cp_int * size_per_grid

    obj_points = []  # the points in world space
    img_points = []  # the points in image space (relevant to obj_points)
    images = glob.glob(img_dir + os.sep + '**.' + img_type)
    for fname in images:
        img = cv2.imread(fname)
        gray_img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
        # find the corners, cp_img: corner points in pixel space.
        ret, cp_img = cv2.findChessboardCorners(gray_img, (w, h), None)
        # if ret is True, save.
        if ret == True:
            # cv2.cornerSubPix(gray_img,cp_img,(11,11),(-1,-1),criteria)
            obj_points.append(cp_world)
            img_points.append(cp_img)
            # view the corners
            cv2.drawChessboardCorners(img, (w, h), cp_img, ret)
            cv2.namedWindow('FoundCorners', cv2.WINDOW_NORMAL)
            cv2.resizeWindow("FoundCorners", 600, 600)  # 设置窗口大小
            cv2.imshow('FoundCorners', img)
            cv2.waitKey(1)
    cv2.destroyAllWindows()
    # calibrate the camera
    ret, mat_inter, coff_dis, v_rot, v_trans = cv2.calibrateCamera(obj_points, img_points, gray_img.shape[::-1], None,
                                                                   None)
#    print(("ret:"), ret)
    print(("internal matrix:\n"), mat_inter)
    # in the form of (k_1,k_2,p_1,p_2,k_3)
    print(("------------------------------------------------------------------"))
    print(("distortion cofficients:\n"), coff_dis) #畸变系数k1,k2,k3径向畸变系数, p1,p2是切向畸变系数。
    print(("------------------------------------------------------------------"))
    print(("rotation vectors:\n"), v_rot)
    print(("------------------------------------------------------------------"))
    print(("translation vectors:\n"), v_trans)
    print(("------------------------------------------------------------------"))
    # calculate the error of reproject
    # 反投影误差
    # 通过反投影误差，我们可以来评估结果的好坏。越接近0，说明结果越理想。
    # 通过之前计算的内参数矩阵、畸变系数、旋转矩阵和平移向量，使用cv2.projectPoints()计算三维点到二维图像的投影，
    # 然后计算反投影得到的点与图像上检测到的点的误差，最后计算一个对于所有标定图像的平均误差，这个值就是反投影误差
    total_error = 0
    for i in range(len(obj_points)):
        img_points_repro, _ = cv2.projectPoints(obj_points[i], v_rot[i], v_trans[i], mat_inter, coff_dis)
        error = cv2.norm(img_points[i], img_points_repro, cv2.NORM_L2) / len(img_points_repro)
        total_error += error
    print(("Average Error of Reproject: "), total_error / len(obj_points))
    return mat_inter, coff_dis

def dedistortion(inter_corner_shape, img_dir, img_type, save_dir, mat_inter, coff_dis):
    w, h = inter_corner_shape
    images = glob.glob(img_dir + os.sep + '**.' + img_type)
    for fname in images:
        img_name = fname.split(os.sep)[-1]
        img = cv2.imread(fname)
        newcameramtx, roi = cv2.getOptimalNewCameraMatrix(mat_inter, coff_dis, (w, h), 0, (w, h))  # 自由比例参数
        dst = cv2.undistort(img, mat_inter, coff_dis, None, newcameramtx)
        # clip the image
        # x,y,w,h = roi
        # dst = dst[y:y+h, x:x+w]
        cv2.imwrite(save_dir + os.sep + img_name, dst)
    print('Dedistorted images have been saved to: %s successfully.' % save_dir)

if __name__ == '__main__':
    inter_corner_shape = (9, 6)
    size_per_grid = 0.026
    img_dir = ".\\pic\\photo5"
    img_type = "jpg"
    # calibrate the camera
    mat_inter, coff_dis = calib(inter_corner_shape, size_per_grid, img_dir, img_type)
    # dedistort and save the dedistortion result.
    save_dir = ".\\pic\\photo5-ch"
    if (not os.path.exists(save_dir)):
        os.makedirs(save_dir)
    dedistortion(inter_corner_shape, img_dir, img_type, save_dir, mat_inter, coff_dis)

1. Camera calibration at the same angle

(1) front

Among them: 5 pictures (1-5.jpg) 10 pictures (1-10.jpg) 15 pictures (1-15.jpg)

Internal reference matrix:

5 pictures

10 pictures

15 pictures

Distortion parameters

5 pictures

10 pictures

15 pictures

rotation matrix

5 pictures

10 pictures

15 pictures

translation vector

5 pictures

10 pictures

15 pictures

back projection error

5 pictures

10 pictures

15 pictures

(2) side

Among them: 5 pictures (1-5.jpg) 10 pictures (1-10.jpg) 15 pictures (1-15.jpg)

internal reference matrix

5 pictures

10 pictures

15 pictures

Distortion parameters

5 pictures

10 pictures

15 pictures

rotation matrix

5 pictures

10 pictures

15 pictures

translation vector

5 pictures

10 pictures

15 pictures

back projection error

5 pictures

10 pictures

15 pictures

2. Camera calibration at different angles

Among them: 5 pictures (1-5.jpg) 10 pictures (1-10.jpg) 15 pictures (1-15.jpg) 20 pictures (1-20.jpg)

internal reference matrix

5 pictures

10 pictures

15 pictures

20 pictures

Distortion parameters

5 pictures

10 pictures

15 pictures

20 pictures

rotation matrix

5 pictures

10 pictures

15 pictures

20 pictures

translation matrix

5 pictures

10 pictures

15 pictures

20 pictures

back projection error

5 pictures

10 pictures

15 pictures

20 pictures

6. Summary

For different pictures, the internal reference matrix A is a constant value; for the same picture, the internal reference matrix A and the external parameter matrix (R1, R2, T) are constant values; for a single point on the same picture, the internal reference matrix A and the external reference matrix Matrix (R1, R2, T), scale factor Z is a fixed value.

For the same camera, the internal reference matrix of the camera depends on the internal parameters of the camera. No matter what the positional relationship between the calibration plate and the camera is, the internal reference matrix of the camera remains unchanged.

The external parameter matrix reflects the positional relationship between the calibration board and the camera. For different pictures, the relationship between the calibration board and the camera position changes, so the extrinsic parameter matrix corresponding to each picture is different.

Through the backprojection error, we can evaluate the quality of the result. The closer to 0, the better the result.

The corner shape I used is (9,6) and the distance between the corners is 0.026cm.

After comparing different numbers of experiments at different shooting angles, we know:

1. After shooting from the same angle:

(1) For different numbers of checkerboards, the internal parameter matrix changes slightly.

(2) The distortion coefficient includes radial distortion and tangential distortion, and the coefficient will change with the number of checkerboards.

(3) The rotation matrix and translation vector have slightly different values in different numbers of experiments for different pictures.

The changes in the 10 pictures on the front are relatively large, possibly due to the shaking of the shooting angle.

(4) The backprojection error does not vary much with the number of our checkerboards.

2. After shooting from different angles:

(1) For different numbers of checkerboards, the internal parameter matrix changes significantly.

(2) The distortion coefficient includes radial distortion and tangential distortion, and the coefficient will change with different numbers of experiments on the checkerboard.

(3) The rotation matrix and translation vector have different values for different pictures, and the same picture has a large difference in values in different numbers of experiments.

(4) The back-projection error will gradually increase with the number of our checkerboards.

Through the backprojection error, we can evaluate the quality of the result. The closer to 0, the better the result.

3. According to the difference in shooting angle:

Regardless of the shooting angle or the number of checkerboards used in the experiment, the internal parameter matrix, external parameter matrix, distortion parameters, and back-projection errors all change. That is: the result of each calibration will change.

The results of each calibration are different, and there are several possible sources:
a. The initial values of the calculated internal and external parameters are different, which may be because the camera calibration process does not converge to the minimum value when using iterative optimization, but only converges to a local minimum value.
b. The data obtained by calibration is unstable, and the size of the calibration plate is not suitable, which has a great impact.

Computer vision (camera calibration; internal reference; external reference; distortion coefficient)

1. Preliminary knowledge

1. Coordinate system transformation process (camera imaging process)

(1) Convert the camera coordinate system to the image coordinate system (perspective projection transformation follows the principle of pinhole imaging )

(2) Reasons for introducing homogeneous coordinates: ( Why introduce homogeneous coordinates??? )

2. Composition of internal and external parameter matrix

3. Distortion parameters

2. Camera Calibration

1. Zhang Zhengyou's calibration method (optical calibration) and its solution

2. The idea of solving the internal parameter matrix and external parameter matrix:

(1) Calculate the product M of the internal and external parameter matrix

(2) Solve the internal reference matrix A

(3) Solve the external parameter matrix (R1 R2 T)

3. Parameter optimization

1. LM algorithm

2. Camera calibration steps

4. Relevant codes and experimental results