1. Principles of Camera Calibration

The purpose of camera calibration is to obtain the parameters of the camera itself. Appropriate calibration parameters are the prerequisites for the camera to accurately obtain target information. This article introduces Zhang Zhengyou's camera calibration method. The process of camera calibration is to convert the world coordinate system to the camera coordinate system through rigid body transformation, then convert the camera coordinate system to the image coordinate system through projection transformation, and finally convert the image coordinate system to the pixel coordinate system through translation transformation. The coordinate system conversion relationship during camera shooting is shown in the figure below.

**Coordinate system conversion diagram**

2. Zhang Zhengyou camera calibration method

Zhang Zhengyou calibration refers to the single-plane checkerboard camera calibration method proposed by Professor Zhang Zhengyou in 1998. This method is between the traditional calibration method and the self-calibration method. It not only overcomes the shortcomings of the high-precision calibration object required by the traditional calibration method, but also improves the accuracy and is easy to operate compared with the self-calibration method. Therefore, the Zhang Zhengyou calibration method is widely used in computer vision.

General steps of Zhang Zhengyou calibration method:

1. Print a piece of checkerboard A4 paper (the black and white spacing is known), and stick it on a flat plate

2. Take several pictures for the checkerboard (usually 10-20 pictures)

3. Detect feature points (Harris corners) in the picture

4. Solve the Homographic matrix according to the corner position information and the coordinates in the image

5. Using the analytical solution estimation method to calculate 5 internal parameters and 6 external parameters

6. According to the maximum likelihood estimation strategy, design the optimization goal and realize the refinement of the parameters

3. Mathematics principle of Zhang Zhengyou calibration method

3.1 Parameter introduction

2D image points: $m=\begin{pmatrix} u & v \end{pmatrix}^{T}$

3D space point: $M=\bigl(\begin{smallmatrix} X & Y& Z \end{smallmatrix}\bigr)^{T}$

Homogeneous coordinates: $\tilde{m}=\begin{pmatrix} u &v &1 \end{pmatrix}^{T}$ , $\tilde{M}=\begin{pmatrix} X &Y &Z &1 \end{pmatrix}^{T}$

Describe the mapping from space coordinates to image coordinates:

s: scale factor from world coordinate system to image coordinate system

K: camera internal parameter matrix

$\begin{pmatrix} R & t \end{pmatrix}$ : extrinsic matrix

$(u_{0},v_{0})$ : Like principal point coordinates

α, β: fusion of focal length and pixel aspect ratio

γ: radial distortion parameter

3.2 Internal parameter solution

Assume that the checkerboard is located at Z = 0,

Define the i-th column of the rotation matrix R as $r_{i}$ , then:

Then the space-to-image mapping can be changed to:

$s\tilde{m}=H\tilde{M}$

$H=K\bigl(\begin{smallmatrix} r_{1} & r_{2}& t \end{smallmatrix}\bigr)$

Where H is a description of the Homographic matrix, which can be solved by least squares from the world coordinates of the corners to the image coordinates

Let H be H = [h1 h2 h3]

Homography has 8 degrees of freedom. Through the matrix operation of the above equation, according to the orthogonality of r1 and r2, and the constraints of normalization, the following equation can be obtained:

definition $B=K^{-T}K^{-1}$ == $\begin{pmatrix} B_{11} & B_{21} & B_{31}\\ B_{12}& B_{22} & B_{32}\\ B_{13}& B_{23} & B_{33} \end{pmatrix}$ _

B is a symmetric matrix, and its unknown can be expressed as a 6D vector $b=\begin{bmatrix} B_{11} & B_{12} &B_{22} &B_{13} &B_{23} & B_{33} \end{bmatrix}^{T}$

Let the i-th column in H be $h_{i}=\bigl(\begin{smallmatrix} h_{i1} & h_{i2} & h_{i3} \end{smallmatrix}\bigr)^{T}$ , according to the definition of b, deduce: $h_{i}^{T}Bh_{j}=v_{ij}^{T}b$

It can be deduced that: $\begin{pmatrix} v_{12}^{T}\\ (v_{11}-v_{22})^{T} \end{pmatrix}b=0$

If there are n groups of observed images, then V is a 2n x 6 matrix Vb=0,

According to the definition of least squares, the solution of V b = 0 is $V^{T}V$ the eigenvector corresponding to the smallest eigenvalue,

Therefore, b can be directly estimated, and the internal parameters can be solved through b later.

When the observed image n ≥ 3, the unique solution of b can be obtained; when n = 2, the distortion parameter γ = 0 can be generally set; when n = 1, only α and β can be estimated, and at this time it can generally be assumed that The principal point coordinates sum $u_{0}$ to $v_{0}$ 0.

$B=K^{-T}K^{-1}$ , B is a symmetric matrix constructed by b,

The internal parameters can be calculated by the following formula (cholesky decomposition):

3.3 Solving external parameters

The external parameters can be solved by Homography, $H=\bigl(\begin{smallmatrix} h_{1} &h_{2} & h_{3} \end{smallmatrix}\bigr)=\lambda K\begin{pmatrix} r_{1} & r_{2} & t \end{pmatrix}$ , can be deduced:

Generally speaking, the solution $R=\begin{pmatrix} r_{1} &r_{2} & r_{3} \end{pmatrix}$ will not meet the standards of orthogonality and normalization. In actual operation, R can be normalized through SVD decomposition.

4. Experimental procedure

1. Prepare a 5X5 black and white checkerboard as shown in the figure below

2. Adjust the camera angle to take checkerboard photos in different directions. The image set is shown in the figure below

3. Extract the checkerboard corner points in the image, and the corner point image set is shown in the figure below

An example is shown in the figure below:

4. Code

import cv2
import numpy as np
import glob

# 设置寻找亚像素角点的参数，采用的停止准则是最大循环次数30和最大误差阈值0.001
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 35, 0.001)

# 获取标定板4*4角点的位置
objp = np.zeros((4*4,3), np.float32)
objp[:,:2] = np.mgrid[0:4,0:4].T.reshape(-1,2) # 将世界坐标系建在标定板上，所有点的Z坐标全部为0，所以只需要赋值x和y

obj_points = [] # 存储3D点
img_points = [] # 存储2D点

# 获取指定目录下.jpg图像的路径
images = glob.glob(r"/Users/xionglulu/Downloads/project1/BW/*.jpg")
# print(images)

i=0
for fname in images:
    # print(fname)
    img = cv2.imread(fname)
    # 图像灰度化
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    size = gray.shape[::-1]
    # 获取图像角点
    ret, corners = cv2.findChessboardCorners(gray, (4, 4), None)
    # print(corners)

    if ret:
        # 存储三维角点坐标
        obj_points.append(objp)
        # 在原角点的基础上寻找亚像素角点，并存储二维角点坐标
        corners2 = cv2.cornerSubPix(gray, corners, (1, 1), (-1, -1), criteria)
        #print(corners2)
        if [corners2]:
            img_points.append(corners2)
        else:
            img_points.append(corners)
        # 在黑白棋盘格图像上绘制检测到的角点
        cv2.drawChessboardCorners(img, (4, 4),corners, ret)
        i+=1
        cv2.imwrite('conimg'+str(i)+'.jpg', img)
        cv2.waitKey(10)
print(len(img_points))
cv2.destroyAllWindows()

# 标定
ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(obj_points, img_points, size, None, None)

print('------照相机内参与外参------')
print("ret:", ret) # 标定误差
print("mtx:\n", mtx) # 内参矩阵
print("dist:\n", dist) # 畸变参数 distortion coefficients = (k_1,k_2,p_1,p_2,k_3)
print("rvecs:\n", rvecs) # 旋转向量 # 外参数
print("tvecs:\n", tvecs ) # 平移向量 # 外参数


img = cv2.imread(images[10])
print(images[10])
h, w = img.shape[:2]
# 计算一个新的相机内参矩阵和感兴趣区域（ROI）
newcameramtx, roi = cv2.getOptimalNewCameraMatrix(mtx,dist,(w,h),1,(w,h))
print('------新内参------')
print (newcameramtx)
print('------畸变矫正后的图像newimg.jpg------')
newimg = cv2.undistort(img,mtx,dist,None,newcameramtx)
x,y,w,h = roi
newimg = newimg[y:y+h,x:x+w]
cv2.imwrite('newimg.jpg', newimg)
print ("newimg的大小为:", newimg.shape)

5. Experimental results

6. Analysis of experimental results

（1）ret: The calibrated error, representing the average or sum of the reprojection errors. Reprojection error refers to the error between the predicted pixel coordinates and the actual pixel coordinates calculated by projecting the 3D coordinates onto a 2D plane when the calibration result is applied to the checkerboard image. The smaller the reprojection error, the more accurate the calibration result. The calibration error in this experiment is 3.3357. This error is still quite large, which may be caused by the small number of images or the large tilt of the shooting angle.

(2) It can be seen from the experimental results that the value of the parameter pixel spacing in the internal reference matrix mtx is 0, indicating that the pixel spacing of the camera is uniform. In this case, the spacing of adjacent pixels on the image plane is equal.

(3) There is almost no difference between the original image and the image after distortion correction. It may be that the calibration data is insufficient or the accuracy of the calibration plate is low: if the calibration data is insufficient or the accuracy of the calibration plate is low, it will affect the accuracy and accuracy of the calibration. As a result, the distortion correction effect is not good; or the camera imaging distortion is small: if the distortion of the camera imaging is small, the image after distortion correction is not much different from the original image.

Python computer vision (4) - camera calibration